Chancellor was born in Chicago, Illinois, and graduated from the University of Illinois at Urbana-Champaign in 1949. Originally a reporter for the Chicago Sun-Times, he first started his career in national television news as a correspondent on the Huntley-Brinkley Report. In 1961, he replaced Dave Garroway as host of The Today Show, a role he filled for one year.

Chancellor then became an NBC News correspondent. At the 1964 Republican National Convention, he was arrested for refusing to cede his spot on the floor to "Goldwater Girls," supporters of the Republican presidential candidate, Barry Goldwater. When security came to get him, he was forced to sign off: "I've been promised bail, ladies and gentlemen, by my office. This is John Chancellor, somewhere in custody." Due to the political nature of convention coverage, the bad press resulting from the arrest caused him to resign from NBC News. He then became the director of the Voice of America in 1965, a spot he held until 1967.

He returned to NBC in 1968 and eventually became an anchor on the NBC Nightly News, a spot he held from 1970 to 1982; this job became the defining point of his career. Inaugurating the name and setting the pace of the format of "NBC Nightly News," from 1970 to 1971, Chancellor, along with David Brinkley and Frank McGee, was one of three anchors who rotated in a co-anchor duo format. From 1971 to 1976, Chancellor was solo anchor, with contributing commentary from David Brinkley. Facing renewed competition from ABC News, and continued popularity of top rated CBS Evening News with Walter Cronkite, NBC Nightly News went back to co-anchor format from June 1976 until October 1979 with David Brinkley

Although Chancellor was a well spoken journalist and noted author in his own right, his broadcast ratings were overshadowed by Walter Cronkite in the '70s when CBS Evening News was the most popular.

He retired from his head anchor duties on April 2, 1982 and was succeeded by a co-anchor duo team of Roger Mudd and Tom Brokaw for two years, before Brokaw became solo anchor and Mudd went on to host Meet the Press and NBC Almanac (news magazine) Chancellor continued to write most notably "Peril or Prosper" and read editorial commentary for the NBC Nightly News program before moving to the New England area in 1993, where he died of stomach cancer in 1996.

Chancellor was also the voice of Baseball, an award winning documentary by Ken Burns.