In 1951, Ashford took a leave of absence from his Santa Ana, California post office job, where he moonlighted as a Santa Ana municipal league softball and National Night Ball League of Southern California umpire. His colorful style included a personal trademark: when a batter received a base-on-balls, instead of simply calling "Ball Four," Ashford would grandly intone, "Ball Fo-uh, you may proceed to first base." He left Santa Ana to umpire in the Southwestern International League, becoming the first black umpire in the traditionally white professional baseball system. When he was offered a full-season umpiring job, Ashford resigned from the postal service.