In March 1817, at sixty-six years old, Madison left Washington and public politics, honoring the two-term precedent established by Washington and Jefferson. He returned to Montpelier with Dolley. His mother, Nelly, lived in her own wing of the house, his father having passed away just before Madison became secretary of state. He began to manage his plantation and its enslaved community full time for essentially the first time in his life.

Never content with the life of only running his plantation, Madison plunged into other activities to engage with public life. Almost immediately after his presidency, he was elected president of the Agricultural Society of Albemarle. In that position, he advocated progressive farming techniques that would halt the clearing of and replenish Virginia’s exhausted forested areas.

In what modern viewers may consider a tragic paradox, Madison spent more time in retirement thinking about emancipation than he had ever spent as a politician when he was able to effect change. Of course, the political reality was that if Madison had espoused such views during his political years, his political years would likely have been cut abruptly short.

For the rest of his retirement, Madison tried to make sense of the contradictions of liberty and slavery in the new nation and in his private life. He was a founding member and, near the end of his life, president of the American Colonization Society (ACS), an organization dedicated to expatriating free blacks to Africa. Madison thought carefully about an ultimately impractical scheme of selling public lands in the West to raise money to compensate enslavers, free slaves, and fund their expatriation.

Madison advocated the process not so much because he desired to rid the nation of its black population, as some members of the ACS undoubtedly wanted. Rather, Madison was convinced that life would be better for black people in Africa than among a white population that he believed would never allow them full rights and would live in constant fear of retribution for the wrongs they had committed. Madison saw firsthand the impracticability of the plan when he failed to convince some enslaved workers on his own plantation to accept freedom in exchange for leaving the land of their birth and risking an unknown, precarious life an ocean away.

However high-minded his views compared to a large swath of Southern enslavers, Madison refused to stake his reputation on emancipation, even in retirement. Madison was part of a former generation that still considered slavery evil, as opposed to many of the new generation of enslavers that considered it a positive good. Undesirous to stir a hornet nest, Madison spoke circumspectly about slavery in public. He insisted that enslavers be compensated for emancipation (even if just as a practical matter to gain support) even as he pled that Southerners recognize the humanity of the enslaved.

In 1829, Madison was invited to a convention to revise the Virginia Constitution. At the convention, Madison recognized that the enslaved community was a part, however marginalized, of the national family. He advocated for political protections for enslavers against the non-slaveholding community, advancing Madison’s complex legacy as an anti-slavery enslaver.

The Constitution never far from his mind, Madison spent his retirement years organizing the notes he had taken at the Constitutional Convention from 1787, with Dolley’s help. Like many Virginia planters, Madison found farming in Virginia a precarious business, and Montpelier rarely turned much of a profit. He intended posthumous publication of the Notes of Debates in the Federal Convention to support Dolley after his death. Although not perfectly reliable as an impartial record of the proceedings, given that Madison curated the notes a bit as he organized them, the notes remain the most important record of the convention. They continue to have significance not only for the study of history but of constitutional law in the United States.

Thomas Jefferson’s University of Virginia demanded most of Madison’s attention in retirement. Like Jefferson, Madison insisted that republics depended on a citizenry educated in the virtues of liberty and republicanism. Even before he had retired from the presidency, he had accepted an appointment as a trustee of the university. Upon Jefferson’s death in 1826 until 1834, he served as its rector.

Madison died two years later on June 28, 1836. His estate, including the enslaved population, passed to Dolley. He had hoped that Dolley would not sell slaves, which would wrench them from friends and family, except upon a slave’s request or for misbehavior. Facing a financial crisis from Montpelier’s indebtedness, however, Dolley sold Montpelier and its enslaved community and moved to Washington, DC.