AIKEN - James Brown spent a lifetime building up his wealth, but it's only taken a year to find out how much in disarray were the Godfather of Soul's finances when he died.

The late singer was notorious for having money problems, from owing the IRS millions to defaulting on loans. He was once sued by two of his daughters over royalty disputes.

Since his death last Christmas, there have been rampant rumors of buried money in his backyard and suitcases full of money and even court statements that at one time, he was being given $100,000 to spend every month.

Now there are allegations that, at best, Mr. Brown should have watched his financial advisers more carefully.

At worst, he was robbed.

With all the tabloid developments going on right now - born out of wedlock children surfacing, possible bigamy by his fourth wife and his body still not in its final resting place - one issue tops them all: Where did all of James Brown's money go?

That's what Atlanta attorney Louis Levenson would like to know.

And so would his clients - several of Mr. Brown's children and grandchildren, who believe their patriarch's estate and financial affairs were badly mishandled, and possibly stolen from.

"I don't know where it went," Mr. Levenson said.

But he's been trying for months to find out.

He's demanded a full accounting of Mr. Brown's estate from the three men who oversaw it - Buddy Dallas, Alford Bradley and David Cannon. But so far, he says, he's gotten only a fraction of what he's asked for, "none of which have made sense."

Not only can't Mr. Levenson say where all the money went, he doesn't yet know how much there was to begin with. Mr. Brown had multiple income sources.

Over just the past decade or so, Mr. Brown reportedly earned at least $18 million from performances alone. Royalties from his music brought in millions more every year.

And in 1999, he basically mortgaged his music royalties in a deal that netted him $26 million up front in exchange for repayment as the money rolled in later.

But tax returns from recent years indicate that he claimed just $100,000 a year in earnings, and attorneys involved in the complicated and contentious battle to settle his estate say they've found that some of his accounts were virtually empty when he died.

Two special administrators appointed by the court to oversee the mess say they believe at least one of the three men who'd been entrusted to handle Mr. Brown's financial affairs may have misappropriated millions of the singer's money - allegations he essentially refused to discuss when put on the stand at recent court hearings.

Mr. Cannon, Mr. Brown's longtime accountant and the man most responsible for handling his finances, told The Augusta Chronicle recently that his only mistake was misdirecting a $900,000 royalty check that should have been put toward payoff of the bond. Instead, he placed it in Mr. Brown's trust fund, and the singer paid him $350,000 and apparently spent the rest, he said.

Mr. Cannon paid $350,000 back into the estate when he resigned as a trustee in August. Judge Jack Early, who's been overseeing all the legal challenges, has found him in contempt of court for failing to pay another $373,000. The judge said in a court order that he'll be locked up in a state prison for six months if the sum - plus $50,000 for attorneys' fees and a $10,000 fine - isn't paid by Jan. 25.

Even after death, Mr. Brown is still making money, thanks to a rich catalogue of music and frequent samplings by other artists.

But even as new money rolls in, it's going back out again. Court records indicate that all the legal squabbling has racked up nearly $400,000 in legal bills for the special administrators - so far. The costs could be much higher.

Mr. Levenson believes that family members will one day learn what happened to the soul singer's money - but only after a "forensic" analysis can retrace where all that wealth went.

They've already uncovered what appear on the surface to be fishy transactions made by Mr. Cannon. The two special administrators say at least $7 million could have been misappropriated.

Questionable checks he wrote out to himself or a company he runs landed him on the witness stand - where he repeatedly invoked the Fifth Amendment to refuse answering questions about how much he earned while working for Mr. Brown.

But tax records Mr. Levenson produced in court indicate that the former trustee earned at least $5 million from 2000 to 2006.

Mr. Levenson said he has records that show Mr. Cannon's earnings in 1999 - the same year Mr. Brown borrowed the $26 million against his future royalty earnings - were between $5 million and $7 million.

Besides some South Carolina property holdings and Mr. Cannon's admission in court that he paid a Honduras contractor $866,000 in recent months to build a retirement home in the South American country, Mr. Cannon "seems to have nothing to show for" all the money he reportedly made over the years, Mr. Levenson said.

"Clearly, Mr. Cannon was writing himself checks" without any oversight by anyone else, he added.

Mr. Cannon's agreement with the singer was that he would make 5 percent of everything Mr. Brown earned, Mr. Levenson explained, questioning the fairness of that arrangement.

Mr. Cannon's Columbia attorney, Eric Bland, said Mr. Cannon "vehemently denies misappropriating any money."

Any financial transactions Mr. Cannon did, the attorney said, were done with the authority of Mr. Brown, who considered the former trustee a "loyal, trusted confidante."

"He (Mr. Brown) knew at all times what was going on with his money," Mr. Bland said. "He had his finger on the pulse."

Mr. Bland said he could not comment or elaborate on many questions about Mr. Cannon's tenure as one of the singer's advisers.

Mr. Cannon refused to answer questions at that court hearing, he stressed, because he is the subject of an IRS audit.

Mr. Cannon is also under investigation by both the attorney general's office and the South Carolina Law Enforcement Division, Mr. Bland said.

All three men who'd been the singer's longtime legal and financial advisers are no longer overseeing his estate.

On Nov. 20, Mr. Dallas and Mr. Bradley resigned their positions both as personal representatives of the estate and trustees of two charitable trusts the singer created - right before a hearing to determine whether misconduct allegations lodged against them were true.

However, they are trying to retract those resignations, arguing that they quit only after improper judicial influence from Judge Early.

But all three were personally hired by Mr. Brown - and he named them in his will as the people he wanted to carry out his last wishes and oversee a charitable trust for disadvantaged children, which was to be funded by proceeds from Mr. Brown's likeness and music rights.

Mr. Bland said the singer knew what was going on. Others say Mr. Brown should have watched the trustees more carefully.

Mr. Levenson said there were others who could have spotted any financial wrongdoing, "but no one did."

Except, he says, for one former bookkeeper who was threatened with termination when she raised questions about some of the money dealings.

"Who was watching the watchers?" Mr. Levenson said. "Well, nobody was."

Reach Sandi Martin at (803) 648-1395, ext. 111, or sandi.martin@augustachronicle.com