Born in Middlebury, Vermont, Stewart graduated from the Middlebury Academy in 1846. He studied law, was admitted to the bar in 1850, and commenced practice there. He became the prosecuting attorney of Addison County from 1852 to 1854.

Stewart served as a member of the Vermont House of Representatives in 1856, and then was a member of the Vermont Senate from 1861 to 1862. He returned to the state house from 1865 to 1867, serving as the speaker, and then became the Governor of Vermont from 1870 to 1872. Later he returned to the state house for a single year, in 1876, again serving as speaker.

Stewart was elected as a Republican to the United States House of Representatives, serving from March 4, 1883 to March 3, 1891. He declined to be a candidate for renomination in 1890, but instead engaged in the banking business at Middlebury. He was appointed to the United States Senate on March 24, 1908, to fill the vacancy caused by the death of Redfield Proctor, and served until October 21, 1908, when a successor was elected. After this he retired from political life and active business pursuits, and resided in Middlebury until his death. He was intered in West Cemetery.