Born in Columbus Grove, Ohio, he dropped out of high school twice before graduating from Wabash College. Working first as a school teacher, he soon became interested in acting and enrolled at Chicago's Lyceum Art Conservatory, beginning his acting career by appearing in vaudeville and with touring stock companies in the 1920s.

Jagger made his film debut in The Woman from Hell (1929) with Mary Astor. He became a successful character actor, without becoming a major star, and appeared in almost 100 films in a career that lasted until shortly before his death.

He received an Academy Award for Best Supporting Actor for his role in Twelve O'Clock High (1949). Other notable film roles included the 1956 British science-fiction film X the Unknown, although there was controversy when he refused to work with director Joseph Losey on this film because Losey was on the Hollywood blacklist. Losey was removed from the project after a few days shooting and replaced with Les Norman.

Jagger also achieved success in the television series Mr. Novak, winning Emmy Award nominations for his role, in 1964 and 1965.

Jagger actually spoke with a pronounced lisp in real life. This speech characteristic however disappeared when he was in front of the camera.

He died from heart disease in Santa Monica, California.