Olney was born in Oxford, Massachusetts, and studied at Brown University (Class of 1856), and Harvard Law School (Class of 1858). In 1859 he began practicing law in Boston, and attained a high position at the bar. He served in the Massachusetts House of Representatives in 1874.

In March 1893, Olney became U.S. Attorney General. During the Pullman strike in 1894, he instructed the district attorneys to secure from the Federal Courts writs of injunction restraining the strikers from acts of violence; thus setting a precedent for "government by injunction." He also advised the use of Federal troops to quell the disturbances in the city, on the ground that the government must prevent interference with its mails and with the general railway transportation between the states.

Upon the death of Secretary of State Walter Q. Gresham, Olney succeeded him on June 10, 1895. He quickly elevated U.S. foreign diplomatic posts to the title of Embassy, thus making it official that the U.S. would be regarded as an equal of the world's greater nations (up until that time, the United States had had only Legations, which diplomatic protocol dictated be treated as inferior to Embassies). He became specially prominent in the controversy with United Kingdom concerning the boundary dispute between the British and Venezuelan governments, and in his correspondence with Lord Salisbury gave an extended interpretation to the Monroe Doctrine which went considerably beyond previous statements on the subject.

In 1897, at the expiration of Cleveland's term, Olney returned to the practice of the law.