Born in Gilmerton, Virginia, Gilmer studied law, practiced in Charlottesville, Virginia, and served for many years in the Virginia House of Delegates (1829-1836, 1839-1840), for two sessions as speaker. From 1840 until 1841 he served as Governor of Virginia. In 1841 he entered the 27th U.S. Congress, and although he had been elected as a Whig, sustained President John Tyler's vetoes. He was re-elected to the 28th Congress as a Democrat in 1842 by a close vote. His competitor, William L. Goggin, contested the result without success.

He was married to Anne E. Baker, daughter of John Baker of West Virginia.

As one of President Tyler's close Virginia allies in Washington, Gilmer was involved in the effort by the Tyler Administration to make the annexation of Texas the basis for his failed bid for reelection in 1844. On February 15, 1844, he was appointed by Tyler to be U.S. Secretary of the Navy, and resigned his seat in congress on February 18 to enter on the duties of the office, but 10 days later was killed by the bursting of a gun on board USS Princeton while on a tour of the Potomac River. His death meant the loss of a valuable ally for Tyler and some historians suggest that it may have delayed the Texas annexation effort.

He is buried at Mount Air Cemetery in Albermarle County, Virginia. Two ships of the United States Navy have been named USS Gilmer in his honor. Also, Gilmer County, West Virginia and an academic building at the University of Virginia, Gilmer Hall, and the city of Gilmer, Texas are also named in his honor. (Gilmer is the county seat of Upshur County, Texas, named after Abel Parker Upshur, another victim of the USS Princeton explosion.)