Caddell, who is white, moved into the Brownsville neighborhood in June and began flying the flag about a month later. Since then, more than 200 residents have signed a protest petition, and neighbors plan to march today along the street in front of Caddell's house.
"My first reaction was they are going to do what they think they need to do," said Caddell, 50. "My second reaction was I'm not going to be here."
Caddell plans to be on nearby James Island for the wedding of a black friend. She tearfully told the town council this week that she is not racist.
She also flies the American flag from her modest brick house, and her yard has various ornaments, including a gnome, Halloween and Thanksgiving decorations, and a "Confederate Boulevard" sign on her fence.
"That flag means a lot more to me than anything I can describe to you," Caddell said. "It's my heritage, and it's my right. I'm not trying to slam anybody, and I wish I wouldn't be slammed either."
James Patterson, a 43-year-old crane operator who lives in a mobile home next door, said displaying the flag is insensitive.
"I know she has a legal right to do that on her property. But just because it's legally right, doesn't make it morally right," said Patterson, who is black. "You can put up what you want, but if this was a Jewish community and someone moved in and started flying swastika flags, there would be a lot of hell raised about that as well."
Violet Saylor, 74, a retired social worker who lives about three blocks away, said the flag brings back memories of Jim Crow in the neighborhood that she has lived in all her life.
"She shouldn't fly that flag because it represents slavery and the Ku Klux Klan that used to ride through the town and we used to have to turn our lights off and hide behind the shades," Saylor said.
The town of Summerville has said that although Caddell has the right to fly the flag, neighbors also have the right to march.