"I want to make sure we regulate what needs to be regulated, you don't get rid of all regulations," the former Massachusetts governor said. "But you don't so overburden the financial sector that it can't do the job it's intended to do, which is to make loans to small business and put people back to work."
Romney made the comments after touring Kenny's Great Pies in the Atlanta suburb of Smyrna, where he sampled some key lime pie. He said they were delicious. Georgia Attorney General Sam Olens was at his side.
Romney also defended his view that the United States should leave Afghanistan, amid criticism from Republican U.S. Sen. Lindsey Graham of South Carolina that his foreign policy resembled that of former Democratic President Carter.
"I think we did a superb job in removing the Taliban following after 9/11, and that's why we went there to get the Taliban and al-Qaida out," he said. "And now we're moving to a phase where to hold on to the independence and the economic vitality that is beginning to grow in Afghanistan, the Afghanis themselves are going to have to step forward to secure that kind of security."
Romney, the presumed front-runner in the Republican race, was scheduled to headline a fundraiser Thursday night at a private Atlanta residence.
Romney is venturing onto turf fellow GOP presidential hopeful Newt Gingrich wants to claim. The former Georgia congressman has set up campaign headquarters in Atlanta and has the support of Gov. Nathan Deal.
Gingrich's campaign has struggled in recent weeks, as top campaign staff and advisers have abandoned his bid citing differences in strategy.
Former Georgia Gov. Sonny Perdue also left as his national campaign co-chairman, lining up behind former Minnesota Gov. Tim Pawlenty.
Radio talk show host Herman Cain, another presidential hopeful, also hails from Georgia.