Drafted by the Cincinnati Reds in the 9th round of the 1983 amateur draft, Montgomery attended Marshall University. He made his major-league debut with the Reds on August 1, 1987 at age 25 and pitched a total of 14 games, including one start. He was considered a marginal prospect, and the Reds traded Montgomery to the Royals on February 15, 1988 for Van Snider, an outfielder who would go on to only play 19 games in the majors.

Kansas City installed Montgomery in the bullpen, and he appeared in 45 games, posting a record of 7-2 with a 3.45 earned run average. In 1989 he had a breakout season, winning 7 games and losing 3, saving 13, and posting a 1.37 ERA while acting primarily as a set-up man for Steve Farr. The following season, he became the Royals' closer, a role he held for most of his career.

In 1993, Montgomery saved 45 games, tying Dan Quisenberry for the Royals' team record. That year he won the American League Rolaids Relief Man Award. Montgomery was also an All-Star in 1992, 1993, and 1996.

Montgomery lost effectiveness during the 1999 season, which he finished with 12 saves and a 6.84 ERA. He retired following the season, and was inducted into the Royals' Hall of Fame in 2003, his first year of eligibility.

Following his career, Montgomery appeared in an instructional video titled The Fundamentals of Pitching with Jeff Montgomery.

In 2005, his first year of eligibility, Montgomery received less than 5f the vote (he received 2 votes; the threshold was 26) from the Baseball Writers Association of America (BBWAA) for induction into the Baseball Hall of Fame, thus becoming ineligible to appear on future BBWAA ballots. However, he may eventually be considered for induction into the Hall by the Veterans Committee once 20 years have passed from his date of retirement (therefore, in the year 2020), in accordance with current Hall of Fame rules (enacted in 2001).