Born into a poor family in Waco, North Carolina, Patterson was one of eleven children and experienced an insular and troubled childhood. A persistent truant and petty thief he was sent to the Wiltwyck reform school at aged ten, which he credited with turning his life around. At age fourteen he started to box, trained by Cus D'Amato at his now-legendary Gramercy Gym. Aged just 17, Patterson, won the Gold medal in the 1952 Helsinki Olympics as a Middleweight. Patterson turned pro and steadily rose through the ranks, his only early defeat, an eight-round decision to former light-heavyweight champion Joey Maxim. Although a natural at middleweight, he fought Archie Moore in 1956 for the world heavyweight championship left vacant by Rocky Marciano. He beat Moore by a knockout in five, and became at the time boxing's youngest world heavyweight champion in history, at the age of 21.

After a series of defenses, Patterson met Ingemar Johansson of Sweden, in the beginning of what many consider one of boxing's most interesting trilogies of fights. Johansson triumphed over Patterson in 1959, with the referee stopping the fight in the third round after the Swede had knocked Patterson down seven times. Johansson became that country's first world heavyweight champion, thus becoming a national hero in Sweden immediately. Patterson came back and knocked Johannson out in the fifth round of their rematch, with what many boxing historians have called the best punch ever in boxing, to become the first man ever to recover the world's undisputed heavyweight title. After the count, Patterson showed his concern for Johansson by cradling his motionless opponent, and promising him a second rematch. Patterson further endeared himself with the people who had made Johansson their national hero, and when he went on a European exhibition tour after that rematch, he was greeted by Swedish fans, who were eager to shake hands, ask for autographs and take photos with Patterson everywhere he went during his stay there.

A third fight between them was held in 1961, and while Johansson put Patterson on the floor, Patterson retained his title by a knockout in six to win a wild rubber match in which Patterson was decked twice and Johannson, once in the first round. After one more defense, Patterson lost his title in September of 1962 by a knockout in the first to Sonny Liston. The two fighters were a marked contrast. In the ring, Liston's size and power proved too much for Patterson's guile and agility. Ten months later on July 22, 1963, Patterson again attempted to become the first boxer ever to win the world's Heavyweight title three times, but Liston once again knocked him out in the first round.

Following these defeats, Patterson went through a depression, often donning sunglasses and hats to disguise himself in public. However, he eventually recovered and began winning fights again, until he became the number one challenger of the man who twice beat Liston, Muhammad Ali. On November 22, 1965, in yet another attempt to be the first to win the world's Heavyweight title three times, Patterson lost by technical knockout in 12 rounds.

In 1967, Ali was stripped of the heavyweight title for refusing military service after being drafted into the United States Army. The World Boxing Association staged an 8-man tournament to determine Ali's successor. Patterson, in a third and final attempt at winning the title a third time, lost a 15 round split decision to Jimmy Ellis in Sweden despite breaking Ellis' nose and scoring a knockdown.

Patterson went on, but after losing a rematch to Ali for the North American Boxing Federation heavyweight title by technical knockout in seven, he retired for good in 1972.

In retirement, he and Johansson became good friends who flew across the Atlantic to visit each other every year, and he became chairman of the New York state Athletic commission, a job that he held until recently. He also became a member of the International Boxing Hall Of Fame.

Patterson lives in New Paltz, New York, and is a convert to Roman Catholicism.

He had a record of 55 wins, 8 losses and 1 draw, with 40 wins by knockout. He once said that a champion should conduct himself as one in real life as well as in the ring.