The trustee, Irving Picard, sought $198.7 million from Madoff's brother, Peter, who had worked at Madoff's Manhattan investment company since 1965, and sons, Mark and Andrew.
Also sued was Shana D. Madoff, Bernard Madoff's niece and Peter Madoff's daughter.
Bernard Madoff, 71, a former Nasdaq chairman, is serving a 150-year sentence for the fraud in connection with a multibillion-dollar Ponzi scheme. No other family members have been criminally charged. The lawsuit said the defendants at least should have seen signs of irregularity and fraud at the company "but either failed to make sufficient inquiry or knew of the fraud, ignored it, and profited from it."
Lawyers for Picard said in papers filed in U.S. Bankruptcy Court in Manhattan that Madoff's family-run business "was operated as if it were the family piggy bank."
They said family members withdrew huge sums of money to fund personal business ventures and pay for expenses ranging from multimillion dollar homes, cars and boats to monthly credit card charges for restaurants, vacations and clothing.
The lawyers said $141 million identified as fraudulent proceeds were received by family members in the six years before Madoff surrendered and revealed his plot last December while at least $58 million was received in the last two years.
Peter Madoff was the company's senior managing director and chief compliance officer while Mark and Andrew shared the title of co-director of trading.
Mark had worked with his father at the company since 1986 while Andrew had been there since 1988. Shana Madoff, a lawyer, had worked there since 1995 as compliance counsel and in-house counsel, the court papers said.
"Senior management — the family defendants — did not do their jobs," Picard's lawyers said. "Otherwise, the Ponzi scheme might have been detected and stopped many years ago."
The lawsuit said the family members frequently held themselves out to be business and securities regulatory compliance managers and principals of Madoff's company.
"Yet the family members were completely derelict in these duties and responsibilities," the lawyers wrote.
"Simply put, if the family members had been doing their jobs — honestly and faithfully — the Madoff Ponzi scheme might never have succeeded, or continued for so long," according to the court papers.
Martin Flumenbaum, an attorney representing Mark and Andrew Madoff, said in a statement that the lawsuit's claims were baseless.
He said the brothers had no prior knowledge of Bernard Madoff's crimes and contacted the U.S. Department of Justice and the Securities and Exchange Commission immediately after their father told them he had defrauded clients.
"By immediately turning him in, the brothers saved the victims of the fraud more than $170 million that their father was about to distribute," Flumenbaum said.
He said Mark and Andrew were cooperating fully with investigators and he noted that they too had "suffered substantial economic losses as a result of their father's crimes."
A lawyer for Madoff's brother did not immediately return a phone call for comment. A message for comment left at Shana Madoff's East Hampton home was not immediately returned.
Picard's lawyers said Madoff's investment business did not register with the SEC as an investment adviser until September 2006, many years after it should have done so, and then did not file paperwork that reflected what investors were told.
For instance, Madoff's private investment business told the SEC in January 2008 that it managed assets worth $17.1 billion in 23 customer accounts when it actually had more than 4,900 open customer accounts and falsely claimed to investors that they were worth about $68 billion, the court papers said.
The lawyers said the company's SEC compliance documents only provided the appearance of compliance efforts, rather than actual compliance with federal securities laws.
"The family members each turned two blind eyes to these duties," Picard's lawyers said.
The court papers took particular aim at Peter Madoff, saying he "failed miserably" to meet his responsibilities to monitor the company's operations and ensure its compliance with federal securities laws.
And they also were highly critical of his daughter Shana, saying that in her role as the company's compliance director: "It would seem impossible for her to carry out her compliance duties, year in and year out, without questioning or wondering whether the company's (investment) business was a fraud."
They added that "she ignored every red flag of the massive fraud taking place right in front of her."
The lawyers wrote that Madoff's sons failed to make the slightest attempt to fulfill their supervisory responsibilities in a company that, despite its claims of being on the cutting edge of technology, used outdated systems and kept paper records of financial activity.
The lawyers said Madoff's Ponzi scheme "massively enriched" the defendants, who received money through paperwork that was falsified, backdated or fabricated to make it seem the money was not customer assets.
The court papers said Peter Madoff has received at least $60 million during the fraud and used fake stock trades to make large withdrawals seem justified.
The lawsuit accused Mark Madoff of using $66 million he received improperly to live a high-end lifestyle with homes in Manhattan, Nantucket and Greenwich, Connecticut. It said he received "astronomical compensation" between 2001 and 2008 including $29 million in salary and bonuses of $4.8 million in 2006 and more than $9 million in 2007.
It said the company's funds "paid for all aspects of his lavish lifestyle from the purchases of his high-end homes to the mattress and box spring he slept on, the television he watched in his home gym, and the outdoor shower in his home."
Still, the lawsuit said, Mark Madoff still seeks $44 million in deferred compensation even though Picard has not discovered any evidence that any of his pay was deferred.
The lawsuit criticized Andrew Madoff the same way, saying he lived a high-end lifestyle after receiving more than $60 million, including $31 million in salary and bonus from 2001 to 2008.
It said the family should have known they were not entitled to "free" company money, especially since they were intimately involved in the business and were close relatives who sometimes traveled, vacationed and spent many holidays together.
The fraud was uncovered last December when Madoff admitted losing billions of dollars for thousands of clients during his half century career.