In 1978, Carroll published The Basketball Diaries, an autobiographical book concerning his life as a teenager in New York City's hard drug culture. Diaries is an edited collection of the diaries he kept during his high school years; it details his sexual experiences, his high school basketball career, and his addiction to heroin.
In 1978, after he moved to California to get a fresh start since overcoming his heroin addiction, Carroll formed The Jim Carroll Band, a new wave/punk rock group, with encouragement from Patti Smith, with whom he once shared an apartment in New York City, along with Robert Mapplethorpe. He performed a spoken word piece with the Patti Smith Group in San Diego when the support band dropped out at the last moment. The band was originally called Amsterdam, where they originally formed and were based in Bolinas, California. The musicians were Steve Linsley (bass), Wayne Woods (drums), Brian Linsley and Terrell Winn (guitars).
Carroll became sober in the 1970s. After moving to California, he met Rosemary Klemfuss; the couple married in 1978. The marriage ended in divorce, but the two remained friends.