Life in the White House for Andrew Johnson's family was an ongoing cavalcade of visitors and activity. Because his wife, Eliza, was a semi-invalid and kept to her room most of the time, suffering from tuberculosis, Johnson asked his two daughters, Martha and Mary, a widow, to live in the White House and serve as official hostesses. They brought their five children and Martha's husband, David T. Patterson, who later became a U.S. senator from Tennessee, with them.

Johnson's sons were tragic figures: The oldest, Charles, died in 1863 after being thrown from a horse. He was serving at the time with the Middle Tennessee Union Infantry as an assistant surgeon. Johnson's next son, Robert, suffered from alcoholism. His drunken escapades led to his retirement from the First Tennessee Union Cavalry, after which he served as Johnson's private secretary. He died from his affliction in 1869 at age thirty-five. The youngest boy, Andrew Jr., a teenager during the White House years, liked to write and tried his hand at journalism after the war, founding the Greeneville Intelligencer. It failed after two years, and he died soon after at age twenty-seven.

No clearly established routine dominated daily life in the White House. The President rose early and worked late. He was not a religious man, although he sometimes attended Methodist services with his wife. He liked best the Baptist faith because of its democratic structure. But he also admired Catholic services because all Catholics had equal access to church pews regardless of their money.

For entertainment, Johnson practiced politics, talked for hours with old friends who would come to visit, played an occasional game of checkers, and enjoyed circuses and minstrel shows. He probably took some of his racist banter on the stump from the humor poked at blacks in the popular minstrel shows of the day. He also enjoyed drinking Tennessee bourbon, and he suffered from perhaps an undeserved reputation for overindulging. At the grand inauguration ball in 1864, Johnson, while suffering from a severe cold and fever, had taken a whiskey just before making his formal speech. He looked and sounded drunk to the embarrassment of his family and President Lincoln. At several other times during his presidency, Johnson appeared in public in what looked to be an inebriated state. He never lived these incidents down, although historians contend that they were greatly exaggerated.