The public exchange happened Monday between Warner Robins Councilman John Williams, who is white, and Daron Lee, who is black.
During a meeting, Lee said he was being disrespected by Williams. Lee said he wasn't working in a cotton field.
Williams said, "You should be." He later denied the remark was racist.
The National Association for the Advancement of Colored People said city officials should step down if they cannot represent all the city's constituents. The Anti-Defamation League called Williams' remark offensive and is offering diversity training to the council.