Klugman began acting after serving in the United States Army during World War II. A struggling actor in New York City, Klugman was a roommate of another starving actor, Charles Bronson, before the two went onto bigger and better things. He starred in several classic films including 12 Angry Men (which he says is his favorite), Goodbye, Columbus, and Days of Wine and Roses. He also won an Emmy Award for his work on the television series The Defenders and appeared in four episodes of the acclaimed series The Twilight Zone. Klugman says his greatest thrill was appearing with Humphrey Bogart and Henry Fonda in a 1954 live television broadcast of The Petrified Forest.

He is best known for his starring roles in two popular television series of the 1970s and early 1980s: The Odd Couple (1970-1975) and Quincy, M.E. (1976-1983). He won two Emmy Awards for The Odd Couple.

In the early 1990s, Klugman lost a vocal cord to cancer, but has continued acting on stage and on television. He survived, though the effect of the cancer on his voice is significant, and he now has a very quiet and rasping voice. Klugman admits that for many years he was a heavy smoker. During the 1990s his television credits included a guest staring role in Diagnosis Murder. Interestingly, his cancer was written into his appearance in Diagnosis Murder, where Klugman played a detective who had terminal cancer and had to solve his last case before he died.

He has lived with Peggy Crosby (the ex-wife of Bing Crosby's son, Phillip Crosby) since 1988. He is the father of three children: Adam, David, and daughter Leslie, all of whom are from his marriage to Match Game regular Brett Somers. Klugman and Somers were married from 1953 until 1974.

In 2005, Klugman published a book called Tony And Me: A Story of Friendship, about his long friendship with his Odd Couple co-star Tony Randall. Klugman said Randall was the best friend he ever had. He talked of their long working relationship and how good Randall had been to him after his cancer operation.