Mathias was born in Frederick, Maryland and attended Frederick County Public Schools. In 1944, he graduated from Haverford College in Pennsylvania. He later attended Yale University, and received a law degree from the University of Maryland School of Law in 1949.

In 1942, during the Second World War, Mathias enlisted in the United States Navy as an apprentice seaman. He was promoted to an ensign in 1944 and served sea duty in the Pacific Ocean area from 1944 until he was released from active duty in 1946.

Following the War, Mathias rose to the rank of Captain in the United States Naval Reserve and became assistant Attorney General of Maryland from 1953-1954. From 1954-1959, he was the City Attorney of Frederick. In 1959, he was elected to the Maryland House of Delegates.

After only serving a year in the House of Delegates, Mathias was elected to the Eighty-seventh and to the three succeeding Congresses, serving from 1961 to 1969. Mathias was elected to the United States Senate in 1968, and reelected in 1974 and 1980, serving from 1969 to 1987. He chose not to run for re-election in 1986.

While senator, he served as chairman of the U.S. Senate Special Committee on Termination of the National Emergency (92nd to 94th Congresses), co-chairman of the Joint Committee on Printing (97th and 99th Congresses), member of the Joint Committee on the Library (98th and 99th Congresses) and chairman of the U.S. Senate Committee on Rules and Administration (97th to 99th Congresses).

Senator Mathias' greatest legacy is his contribution to the Vietnam Veterans Memorial. * On November 8, 1979, Senator Mathias, and senator John W. Warner of Virginia introduced legislation to authorize a site of national parkland for the Memorial. .

Today, Mathias practices law in Washington, D.C. and is a resident of Chevy Chase, Maryland.