Heavily recruited by Pac-8 universities, Graf opted to stay in his native Los Angeles to play for John McKay and the University of Southern California Trojans. Graf was unhappy when line coach Rod Humenuik told him he'd be playing offensive tackle. Graf liked to tackle offensive ball carriers; with his natural aggressiveness, he felt he would play better on defense. Humenuik said: "With your speed you're a great pulling guard. You have a natural tendency to pull with your hips." Graf started three years at tackle for the Trojans. During USC's remarkable undefeated 1972 season, Graf played with Trojan legends Lynn Swann, Pat Haden and Sam Cunningham, helped Anthony Davis to get a record six touchdowns against rival Notre Dame, and won a national championship ring after defeating Ohio State in the 1973 Rose Bowl. The 1972 USC Trojans have sometimes been classed among the best college football teams of all time. Thirty three teammates on the 1972 roster would eventually be drafted by teams in the National Football League, including five first round picks.
Unpicked in the 1973 NFL Draft, Graf attended the Los Angeles Rams' 1973 fall training camp as a free agent. Playing behind All-Pro Tom Mack in camp, Graf asked to be traded or released, but was the last man cut from camp, leaving him without an NFL contract. Graf was one of several Trojan graduates to join the Portland Storm franchise in the new World Football League, but after the league folded in the middle of its second season, Graf thought his football career was behind him.