Elliott was born in London to Myles Layman Elliott and Nina. He served in the RAF during World War Two. In 1942 he was shot down over Germany and spent the rest of the war in a P.O.W. camp. After the war, he made his film debut in Dear Mr. Prohack (1949). He went on to play a large range of parts, often playing ineffectual and occasionally seedy characters, such as the journalist Bayliss in Defence of the Realm, the abortionist in Alfie, and the washed-up film director in The Apprenticeship of Duddy Kravitz.

In the 1980s he won three BAFTA awards as best supporting actor for Trading Places, A Private Function and Defence of the Realm, as well as an Academy Award nomination for A Room with a View. He also became familiar to a wider audience as Dr. Marcus Brody in Raiders of the Lost Ark and Indiana Jones and the Last Crusade. In 1988, Elliott was awarded the CBE for his services to acting.

Elliott was married twice, the first time to the British actress, Virginia McKenna, and the second to Susan Robinson, with whom he had two children.

He died of AIDS at the age of 70 after a transfusion of infected blood. His widow Susan set up a charity, European and UK Coalition of People Living with HIV and Aids, in his memory.