Of Jimmy and Rosalynn Carter's four children, three sons were grown by the time he became president. His daughter Amy lived with her parents at the White House and attended public schools in Washington. She was later admitted to Brown University.

The Carters were, by and large, unpretentious people. Much of the pomp and circumstance that had been a feature of previous administrations was frowned upon. Richard Nixon's huge presidential limousine was retired and replaced by a smaller one. The presidential yacht Sequoia was retired from service. Rosalynn Carter never bothered to commission her own china pattern for the executive mansion—a usual custom for First Ladies. And the Carters did not spend the full allowance an incoming resident and his wife are given to refit the White House living quarters to their liking. Jimmy and Rosalynn Carter were also workaholics. They enjoyed weekends at Camp David, although even there, they tended to bring along "homework" and engage in policy and political discussions with key aides.

Rosalynn Carter died in November 2023. She and Jimmy were married for 77 years, the longest marriage in the history of the American presidency.