Koster was introduced to cinema about 1910 when his uncle opened a very early movie theater in Berlin. Koster's mother played the piano to accompany the films, leaving the young boy to occupy himself by watching the films. After working initially as a short story writer, Koster was subsequently hired by a Berlin movie company as scenarist, became assistant to director Kurt Bernhardt. Bernhardt became sick one day and asked Koster to take over as director. In about 1931 or 1932, Koster directed two or three films in Berlin for UFA.

Koster, who was in the midst of directing a film, had already been subject of anti-Semitism, and knew he had to leave. He lost his temper at an SS officer at his bank during lunch hour, and knocked the officer out. He went directly to the railroad station and left Germany for France, where he was rehired by Bernhardt (who had left earlier). Eventually Koster went to Budapest and met and married Kato Kiraly in 1934. In Budapest he met Joe Pasternak, who represented Universal in Europe, and directed three films for him.

In 1936 Koster got a contract to work with Universal in Hollywood, and he travelled to the United States to work with Pasternak, other refugees and his wife. Although Koster didn't speak English, he convinced the studio to let him make Three Smart Girls, for which he personally coached 14-year-old star Deanna Durbin. This picture, a big success, pulled Universal out of bankruptcy. Koster's second Universal film, 100 Men and a Girl, with Durbin and Leopold Stokowski put the studio, Durbin, Pasternak, and Koster on top.

Koster went on to do numerous musicals and family comedies during late 1930's and early 1940's, many with Betty Grable, Durbin and other musical stars of the era. Koster worked at Universal until 1941 or so, and then moved to MGM, and then Fox in 1948. Ironically, despite his escape from Nazi Germany, when the United States entered World War II Koster was considered an enemy alien and had to stay in his house in the evening. Actor Charles Laughton would visit Koster and play chess with him.

Koster discovered Abbott and Costello working at a nightclub in New York. He returned to Hollywood and convinced Universal to hire them. Their first picture, which featured the Who's On First? routine, was One Night in the Tropics. The female lead, Peggy Moran, would become Koster's second wife in 1942. When he married Moran, Koster promised her he would put her in every movie he made from then on. He did, but it was her statue. Usually it is a sculptured head on a mantlepiece or a piano or desk. In "The Robe" he commissioned a Grecian bust which appears prominently in a Roman villa.

Koster was nominated for an Academy Award for The Bishop's Wife. He directed Richard Burton's first U.S. film, My Cousin Rachel, and then was given first CinemaScope film to direct, The Robe in 1952. He did a couple more costume dramas, including Desiree with Marlon Brando, then went back to family comedies and musicals, including Flower Drum Song for Universal in 1961. His last picture was The Singing Nun in 1965. Koster retired to Leisure Village, Camarillo, California, to indulge in his lifelong interest in painting. He did a series of portraits of the movie stars with whom he worked.