After the New York City draft riots in 1863, Smith and his family were among prominent African Americans who left Manhattan and moved to Brooklyn, then still a separate city. He no longer felt safe in their old neighborhood.
In July 1863, during the New York Draft Riots, Irish rioters attacked blacks throughout the city and burned down the orphan asylum. The children were saved by the staff and Union troops in the city. During its nearly 30 years, the orphan asylum had admitted 1310 children, and typically had about 200 in residence at a time.
In 1863 Smith was appointed as professor of anthropology at Wilberforce College, the first African American-owned and operated college in the United States. Smith was too ill to take the position. He died two years later, on November 17, 1865, at the age of 52, from congestive heart failure. This was nineteen days before ratification of the Thirteenth Amendment to the United States Constitution, which abolished slavery. He was buried at Cypress Hills Cemetery in Brooklyn. Smith was survived by his widow, Malvina, and five children.