Matthau was married twice; first to Grace Geraldine Johnson from 1948 to 1958, and then to Carol Marcus from 1959 until his death in 2000. He had two children, Jenny and David, by his first wife, and a son, Charlie Matthau, with his second wife. Matthau also helped raise his stepchildren, Aram Saroyan and Lucy Saroyan.
In addition to colon cancer, Matthau had atherosclerotic heart disease during the last years of his life. In the late evening of June 30, 2000, he had a heart attack at his home and was taken by ambulance to the St. John's Health Center in Santa Monica where he died a few hours later at 1:42 a.m. on July 1, 2000. He was 79 years old. He was buried in Westwood Village Memorial Park Cemetery in Los Angeles. Matthau's wife Carol Marcus died in 2003, and her body was interred in the same grave as her husband.