Weathers was born in New Orleans, Louisiana. His father was a day laborer. As an eighth-grade student, he earned an athletic scholarship to St. Augustine High School, a private high school. During high school, he was an all-around athlete, involved in boxing, football, gymnastics, judo, soccer, and wrestling. He graduated in 1966.
Weathers played football as a defensive end in college. He started his college career in 1966 at Long Beach City College, where he did not play in 1966 due to an ankle injury suffered when he tripped over a curb surrounding the running track while warming up for practice with another linebacker, Paul Snow. He then transferred and played for San Diego State University, becoming a letterman for the San Diego State Aztecs in 1968 and 1969, playing for head coach Don Coryell.