He started his musical career in the city of his birth Sheffield, England in some minor bands at the age of 15: the first band was the Avengers (under the stage name Vance Arnold), then Big Blues (1963), and then The Grease Band (1966). His first single was the first of several Beatles covers, "I'll Cry Instead".

After minor success in the UK with the single "Marjorine", he entered the big time with a groundbreaking rearrangement of "With a Little Help from My Friends," another Beatles cover, this time from the Sgt. Pepper's album, featuring lead guitar from Jimmy Page, which topped the UK chart in November 1968 for one week.

In 1969 he appeared at the Woodstock Music Festival, and had a second UK hit with the Leon Russell song "Delta Lady". He had further success covering Beatles tunes in 1970 with his version of "She Came In Through the Bathroom Window" off their Abbey Road album. Though his British success proved difficult to sustain, he enjoyed several chart entries in the US with "Cry Me a River" and "Feelin' Alright". In 1970, his live cover of the Box Tops' hit "The Letter", which appeared on the live album Mad Dogs & Englishmen, became his first US Top Ten hit.

In 1969 he was featured on the American TV program The Ed Sullivan Show. Onstage, he often exhibited a physical intensity as he sang, and his unique stage presence was often spoofed by comedian John Belushi (including an impromptu duet when Joe was the guest musician on Saturday Night Live).

In the beginning of 1970s he had problems with drugs including alcohol that had a negative influence on his musical career. He managed to make a comeback in the 1980s and 1990s with several massive chart hits like: