Madison left the White House and retired to his Virginia plantation, Montpelier, where he spent his remaining years supervising his large plantation holdings and slaves. Being a gentleman planter scarcely utilized all his energies, however, and the sixty-eight-year-old former President exercised his quill, a pen made from a feather, and gave his voice to several causes. High on his list of activities was Jefferson's University of Virginia, which Jefferson founded after leaving office. Madison served on its Board of Visitors and succeeded Jefferson as rector, or head, of the university in 1826.

Three years later, Madison served as a delegate at the Virginia Constitutional Convention, negotiating once again, as he had done in youth, compromises between large slaveholders and western farmers. In the great constitutional debate over the high protective tariff passed in 1828, Madison denounced the doctrine of nullification, the right of states to declare federal laws unconstitutional when they undermined state interests. Additionally, Madison was a founding member of the American Colonization Society, which favored a gradual abolition of slavery and the resettlement of slaves and free blacks in Africa.

Death took the aging President quietly at his breakfast on June 28, 1836, after having been confined to his room for chronic rheumatism and severe attacks from liver dysfunction for six months. His family and much of the nation had hoped that the eighty-five-year-old Madison would live to July 4, so as to join Jefferson and Adams in the list of former Presidents who had died on that historic date. More than 100 slaves, family friends, and relatives attended his burial the next day at the family cemetery at Montpelier.