Avery was born on November 27, 1945 in Pughsville, Virginia (now part of Suffolk, Virginia), but was raised in Atlantic City, New Jersey. He served in the U.S. Navy in the Vietnam War from 1968 to 1969, and eventually moved to San Diego, California, where he began to write poetry and TV scripts for PBS. He won an Emmy for production during his tenure there, and then received a scholarship to UC San Diego, where he attended Thurgood Marshall College (then Third College), earning a Bachelor of Arts degree in drama and literature in 1976.