Broadbent was born in Holton cum Beckering, in Lincolnshire, the second son of Doreen "Dee" Broadbent (née Findlay), a sculptor, and Roy Laverick Broadbent, an artist, sculptor, interior designer and furniture maker. Broadbent's parents were both amateur actors who co-founded the Holton Players acting troupe at Holton. The two have been described by the BBC as conscientious objectors who "worked the land" rather than participate in World War II. In Wickenby, a former Methodist Chapel was purchased in 1970 by Holton Players, who converted it into a 100-seat theatre, named Broadbent Theatre in memory of Roy Broadbent, who designed the conversion.