Davis faces two counts of armed robbery in connection with the heists at Colbert's First Citizens Bank on July 24 and Aug. 27 and is held at the Onslow County Jail in North Carolina.
"He's being held right now to face the extradition process," said Jim Fullington, special agent in charge of the GBI's Athens office.
Investigators found items linking Davis to the robberies during a search of his property at the Marine base, as well as a residence in Madison County where Davis occasionally stays, according to a GBI news release.
Law enforcement officials are not releasing details about what evidence they found or where Davis sometimes stayed in Madison County, Fullington said.
Davis likely will return to Madison County to face the armed robbery charges next week, said Madison County Sheriff Kip Thomas.
"The earliest would be Monday, but it'll probably be later than that," Thomas said.
Davis, a private first class, had been in the military for less than a year, Thomas said. Davis went through boot camp at Parris Island, S.C., and was training at Fort Bragg in Fayetteville, N.C., about 140 miles west of where he was stationed at Camp Lejuene, when military police detained him in the bank robbery investigation, Thomas said.
A 20-year-old Marine from Hull has been arrested and charged with a pair of Colbert, Ga., bank robberies.
Davis was raised in the Madison County area and attended local schools through the ninth grade until he transferred to a school in Jackson County, Thomas said.
"I'm just glad we finally got him," he said.
Investigators received a tip about Davis after local media outlets published information about the bank robberies, bank surveillance photographs and a detailed sketch of the suspect, according to the GBI news release.
Madison County officials last week offered a $5,000 reward for any information leading to the arrest of a suspect in the two bank robberies.
In both heists, the robber pointed a handgun at tellers and demanded they fill his backpack with money.
In July, the robber wore sunglasses and a cap, while in the August's robbery he wore a skull mask.