He was born in Carman, Manitoba, Canada to Elmer and Elsa Carson (née Brunke). He was the younger brother of actor Robert Carson (1909 – 1979). His father was an executive with an insurance company. In 1914, the family moved to Milwaukee, Wisconsin, which he always thought of as his home town. He attended high school at Hartford School, Milwaukee, and St. John's Military Academy, Delafield, but it was at Carleton College that he acquired a taste for acting. Carson became a U.S. citizen in California in 1949.