In the late 1950s, he met Joan Didion in New York City, where she was an editor at Vogue. In a 2005 interview Didion recalled, "We amused each other and I thought he was smart. He knew a lot of stuff that I didn't know, like politics and history – I had managed to go through school without learning much except a lot of poems." He invited her to travel to Connecticut one weekend in 1963 to visit his family: New England Irish Catholic, with six children. Didion said she "liked the set-up, liked being there, and liked him."
He was father to Quintana Roo Dunne, who died in 2005 after a series of illnesses. He was uncle to actors Griffin Dunne (who co-starred in An American Werewolf in London) and Dominique Dunne (who co-starred in Poltergeist).