Milsap was awarded a full college scholarship and briefly attended Young Harris College in Young Harris, Georgia, with plans to become a lawyer. During this time, Milsap joined a popular local R&B band called the Dimensions that played gigs in the Atlanta area, and became a regular attraction at the rough and rowdy Royal Peacock Club. In the fall of 1964, Milsap declined a scholarship to law school and left college to pursue a full-time career in music. He met Joyce Reeves one night at a dinner party during this period, and the two were married in 1965.
Also in 1965, Milsap scored his first hit with the Ashford & Simpson-penned single, "Never Had It So Good", which peaked at No. 19 on the R&B chart in November of that year. It would be his only successful single during his time with Scepter. Milsap cut another Ashford & Simpson tune, "Let's Go Get Stoned", that was relegated to a B-side. A few months later, it became a million-selling single for Ray Charles who heard and liked Milsap's version of the song so much that he decided to record it himself. Milsap's Scepter recording of "Ain't No Soul Left in These Old Shoes" eventually found popularity in England, where northern DJs discovered it and it became a floor filler in Northern soul clubs.