Shemp, along with brothers Moe and Curly, was born in the Bensonhurst section of Brooklyn, New York.

Shemp played his somewhat homely appearance to comic effect, often allowing his hair to fall in disarray or mugging grotesquely on film. Notoriously phobic, his fears including dogs, airplanes, and automobiles.

The first major film he was in was Soup to Nuts (1930) as a bumbling fireman, the only film in which he plays one of Ted Healy's stooges. Shemp left Healy's act in 1932 to pursue a solo career in film. This turned out to be very successful, performing with such comic greats as Fatty Arbuckle, W.C. Fields, and the comedy team Abbott and Costello. He also attempted, unsuccessfully, to create his own group of "stooges." In 1947, he reluctantly replaced his brother Curly as the third stooge in the film shorts. Initially, Shemp rejoined the Stooges on a temporary basis until Curly recovered from a debilitating stroke. However, as Curly's condition worsened, it became apparent that Shemp's association with the Stooges would be permanent.

Shemp is often compared to Curly, but his comedic stylings were also unique, and some fans even prefer his comic style over Curly's. He starred in seventy-seven Three Stooges shorts and appeared with Moe and Larry in the film Gold Raiders (1951). Shemp suffered a minor stroke in 1952, which appeared to have only a minor effect in some of his last films with the Stooges (largely remakes of earlier films that reused footage to save costs). Many fans point out that in his later remakes, or "cheapies", Shemp looks weak, pale and often disoriented.

Shemp died of a massive heart attack while traveling in the back seat of a car returning home from watching a boxing match on November 22, 1955. Legend has it that Shemp was lighting a cigar, and then suddenly slumped over in his friend's lap. Several Three Stooges films were in production at the time of his death; in order to complete them, longtime Stooge supporting actor Joe Palma stood in for Shemp and was filmed mostly from the back. This form of cinematic trickery came to be known as "shemping". The term "fake shemp" has been adopted later in cinematography to refer to or credit someone who appears in a film under heavy make-up, filmed from the back, or perhaps only showing an arm or a foot.

In a 2000 TV movie, Shemp was played by John Kassir.