Not much is known about Van Buren's relationship with his wife Hannah, largely because he rarely mentioned her in his private correspondence (and not at all in his memoirs.) Historian Donald Cole, a leading Van Buren scholar, has concluded, though, that the marriage was likely a happy one. Hannah passed away in 1819, leaving Van Buren to raise his four sons. They spent much of their childhood with close relatives and Van Buren frequently expressed regret at not being more involved in their upbringing. It should be noted, however, that he did provide for their education and well-being.

As the children grew older, they spent increasing time with their father, in both Washington and Albany. Indeed, as adults, they became his trusted aides and advisers. When their father became President, Abraham and Martin Jr. served as his private secretaries, while John and Smith stayed in Albany and kept their father informed of the political goings-on in his home state; John Van Buren actively promoted his father's candidacy in 1848. In the 1850s, after Van Buren receded from the political scene, the ex-President spent more and more time with his sons and their families. Van Buren's son Martin passed away in 1855 after a long illness.