"He said that was the awfulest place he'd ever seen in his life - he said people were dying there like flies," Booth said.
Booth, who sometimes raises eyebrows when people learn he is the son of a Confederate soldier, has told the story many times over the years.
His father, Isham Johnson Booth, was born in 1847 and enlisted in 1863 when he was 16.
After Booth's first wife died, he married the younger Miranda Lou Dunn.
"She was 38 years old when I was born, and Daddy was 72," said Henry Booth, who turned 92 last December.
Booth is one of a dwindling number of still-living sons and daughters of Civil War veterans.
Mississippi resident Larry McCluney Jr., a historian for the National Headquarters of Sons of Confederate Veterans, said about 30 sons still are alive, but he had no figures for daughters. The Sons of Union Veterans of the Civil War updated its website in June, listing 37 known living sons and daughters of Union veterans.
Typically, they are children of Civil War veterans who in the 20th century married younger women and continued to have children.
Henry Booth, who lived in the Vanna community of Hart County for 40 years, is a veteran himself. He served in the U.S. Navy during World War II, assigned to a landing craft ship that took equipment like tanks to the beaches. He was at the major battle sites of Iwo Jima, Guam and Okinawa.
His father didn't talk much with his young son about his experiences in the war, but Booth said he overheard him talking with visitors to their home. Booth's father was a sharecropper and never purchased his own house.
Isham Booth, who had 12 children, including one who died at birth, died in 1934 and is buried at Antioch Baptist Church, north of Elberton.
"He was quite a man," Booth recalled about his father. "He'd rent a farm - a two-horse farm - and put out some oats, corn and wheat. He always liked a big branch on it.
"He was a land renter. He didn't want to own nothing. He said a poor man didn't need nothing but a burial lot," Booth said. "He hated to pay taxes more than any man you've ever seen in your life."
Booth heard his father talk about a stream that coursed through the prison camp and provided water to the prisoners.
"I've seen that branch. I've been down to it two or three times. It came into the camp at one end with a pretty good flow of water, and he said when it came out the other end there wasn't nothing. They used the water up," he said.
While serving as a guard, the teenage soldier became sick with a fever. He was sent home by horseback to recover and ordered to return when well, Booth said.
His father recovered and headed back to Andersonville, but had traveled only about 16 miles from Elberton when he learned at a store that the war had ended, so he returned home.
In 1865, Booth was listed as a deserter. In 1927, he was urged to have the record changed because despite not reporting back to the camp he had never actually deserted.
"He had to go to some old people who knew the story and get a sworn affidavit on it," Booth said.
The old soldier once insisted he would never go back to Andersonville, but in 1931 he did, his son said. A World War I veteran who lived in Elberton, Evan Fleming, bought a new 1931 Chevrolet with money the government gave him from service in the war.
"He came over to Daddy's house and said, 'OK, Uncle Jonce, let's go,' " Booth said.
Nobody knows what the old man thought about the return visit, but Booth himself visited Andersonville in April for Confederate Memorial Day.
"You have a different feeling of the people that were there," he said. "It's worth the trip."