Gilbert, who posted a letter to Cavs fans on the team's Web site shortly after James announced his plans to sign with the Heat, told The Associated Press late Thursday night that it's "accountability time" for James.
"He has gotten a free pass," Gilbert said in a phone interview with The AP. "People have covered up for (James) for way too long. Tonight we saw who he really is."
Gilbert feels James quit on the Cavs during their second-round series against the Boston Celtics, who rallied from a 2-1 deficit to eliminate Cleveland.
"He quit," Gilbert said. "Not just in Game 5, but in Games 2, 4 and 6. Watch the tape. The Boston series was unlike anything in the history of sports for a superstar."
The Cavaliers were beaten by 32 points in Game 5. During the game, James appeared distracted and uninterested, often glaring at Cleveland's coaches as the Cavs tried to foul to get back into the game in the second half. James also made some puzzling postgame comments, saying he had "spoiled" people with his play over seven seasons.
Gilbert also said he believes James quit on the Cavs in Game 6 of their series in 2009 against Orlando.
"Go back and look at the tape," he said. "How many shots did he take?"
Gilbert, who has owned the Cavs for five years, said he was most disappointed by James' behavior in the months leading up to the superstar's announcement that he is going to Miami to play with Dwyane Wade and Chris Bosh.
Gilbert said James never returned a single phone message or text since the end of the season and that the Cavs were not informed of James' decision until he went on the air.
Gilbert said Rich Paul, one of James' business partners, called the Cavs moments before the announcement.
"LeBron James needs to go to another team with two superstars already so he can win a championship," Gilbert said. "We will win a championship before (the Heat) do."