Floyd submitted a one-paragraph letter to Southern Cal athletic director Mike Garrett saying he was quitting because he no longer had full enthusiasm for his job.
"I accept Tim's decision and wish him well," Garrett said in a statement.
The Clarion-Ledger, of Jackson, Miss. first reported Floyd's resignation.
The announcement follows allegations that thousands of dollars in cash and gifts were funneled to Mayo by a representative of a sports agency.
Last month, YahooSports.com reported that Louis Johnson, a former associate of Mayo's, told federal and NCAA investigators that Floyd paid to have Mayo sign with the Trojans.
"The university is cooperating fully in the continuing investigation with the NCAA and Pac-10 into all allegations of NCAA and Pac-10 rules violations at USC," Todd Dickey, senior vice president of administration, said in the statement. "The university, the NCAA and Pac-10 have jointly conducted interviews of approximately 50 witnesses. No conclusions have yet been reached. At this point, it would be both inappropriate and premature to comment further."
USC's powerhouse football team is under NCAA investigation because of allegations that Heisman Trophy-winning running back Reggie Bush accepted gifts and his family free rent from would-be marketers who wanted him as a client.
Floyd's resignation is another blow to a basketball program that has had starters DeMar DeRozan, Taj Gibson and Daniel Hackett declare for the upcoming NBA draft and has lost three recruits since the season ended.
Garrett said a search to hire a new coach would begin immediately.
Floyd submitted his letter of resignation Tuesday afternoon.
"As of 1 p.m. today, I am resigning as head basketball coach at the University of Southern California. I deeply appreciate the opportunity afforded me by the university, as well as the chance to know and work with some of the finest young men in college athletics," the letter said. "Unfortunately, I know longer feel I can offer the level of enthusiasm to my duties that is deserved by the university, my coaching staff, my players, their families, and the supporters of Southern Cal. I always promised my self and my family that if I ever felt I could no longer give my full enthusiasm to a job, that I should leave it to others who could. I intend to contact my coaching staff and my players in coming days and weeks to tell them how much each of them means to me. I wish the best to USC and to my successor."
In April, Floyd spurned an offer from Arizona to fill its coaching vacancy, saying he was staying at Southern Cal. A year ago, he was offered the Louisiana State job and turned it down, saying at the time, "This is my last job at SC."
The day he met reporters to say he had turned down Arizona, Floyd said, "This is still my last job."
The 55-year-old coach led the Trojans to the NCAA tournament three consecutive seasons, a first in the program's history, and this year he coached them to the Pac-10 tournament title.
Floyd had three years remaining on his contract. In April, he said, "Hopefully, they'll add a year or two. That would be great."
Floyd's other college stints were at Iowa State, New Orleans and Idaho. He coached the NBA's Chicago Bulls from 1999-2002 and the New Orleans Hornets from 2003-04.