AI In Law Risky Business For Attorneys

It's a classic tale: a product backfiring spectacularly during a showcase. A recent real-world example involves a prominent law firm that employed an AI chatbot, developed by its own client Anthropic, to assist in drafting expert testimony for the client's defense.

When AI Stumbles in High Stakes Legal Cases

The outcome was far from ideal. Claude, Anthropic's AI, mistakenly identified the title and authors of a cited paper and introduced other textual errors into the expert's statement. These inaccuracies made their way into the court-filed statement in April.

I can’t believe people haven’t yet cottoned to the thought that AI-generated material is full of errors and fabrications, and therefore every citation in a filing needs to be confirmed.

— Eugene Volokh, UCLA law school

These mistakes were significant enough for the plaintiffs suing Anthropic (music publishers alleging copyright infringement by training Claude on lyrics) to request the federal magistrate to dismiss the expert's testimony completely.

This incident could also tarnish the reputation of the major law firm Latham & Watkins, which represented Anthropic and filed the flawed declaration.

Latham argues that the errors were inconsequential, amounting to an “honest citation mistake and not a fabrication.” The firm’s failure to notice the errors before the statement was filed is “an embarrassing and unintentional mistake,” but it shouldn’t be exploited to invalidate the expert’s opinion, the firm told Magistrate Judge Susan van Keulen of San Jose, who is managing the pretrial phase of the lawsuit. The plaintiffs, however, say the errors “fatally undermine the reliability” of the expert’s declaration.

The Troubling Trend of AI Hallucinations in Court

During a telephonic hearing on May 13, Judge van Keulen expressed her own reservations. “There is a world of difference between a missed citation and a hallucination generated by AI, and everyone on this call knows that,” she said, according to a transcript of the hearing cited by the plaintiffs. Judge Van Keulen has not yet ruled on whether to keep the expert’s declaration in the record or whether to impose sanctions on the law firm.

This highlights the core challenge facing judges: court filings increasingly contain significant errors and even outright fabrications, a phenomenon AI experts call "hallucinations."

A roster compiled by French lawyer and data expert Damien Charlotin now details 99 such cases from federal courts across two dozen states, as well as international courts in Europe, Israel, Australia, Canada, and South Africa.

Charlotin believes this is almost certainly an undercount, stating, “I can only cover cases where people got caught.” The number of instances where AI-generated errors have gone undetected is incalculable.

Nearly half of these documented instances involve pro-se litigants (individuals representing themselves without a lawyer). Judges often treat these individuals with more leniency, recognizing their lack of legal experience, typically resulting in case dismissals rather than fines. However, lawyers were responsible in the majority of cases. Alarmingly, around 30 cases involving attorneys, where AI-generated errors were found, occurred or were filed this year alone—long after the propensity of AI bots to "hallucinate" became widely known. This indicates the problem may be escalating.

Courts Push Back: Sanctions and New Rules for AI Use

UCLA law professor Eugene Volokh expresses disbelief: “I can’t believe people haven’t yet cottoned to the thought that AI-generated material is full of errors and fabrications, and therefore every citation in a filing needs to be confirmed.”

Judges are signaling their diminishing patience with AI-generated fabricated quotes, erroneous legal decision references, and citations to non-existent precedents. Filing any document without verifying the accuracy of its factual claims, including case citations, violates Rule 11 of the Federal Rules of Civil Procedure, exposing lawyers to potential monetary sanctions or disciplinary measures.

Some courts have issued standing orders requiring that the use of AI at any point in preparing a filing must be disclosed, along with certification that every reference has been verified. At least one federal judicial district has almost entirely forbidden AI use.

Beyond the Courtroom: AI's Unreliability Problem

The flood of inaccurate references in legal filings underscores a fundamental issue with AI's integration into daily life: its inherent untrustworthiness. It has long been known that even advanced AI systems, when faced with unfamiliar questions or tasks, tend to invent information to fill knowledge gaps.

As other fields utilize AI for important tasks, the consequences can be severe. A team of Stanford researchers wrote last year that many medical patients “can be led astray by hallucinations.” They found that even the most advanced bots could not back up their medical assertions with solid sources 30% of the time.

While fatigue or inattention can affect professionals in any field, attorneys, who often handle high-stakes disputes involving substantial sums, are held to a particularly high standard for fact-checking formal submissions.

Some legal experts suggest there’s a legitimate role for AI in law, even for decisions traditionally made by judges. However, lawyers cannot afford to be unaware of the pitfalls for their profession if they fail to monitor AI outputs.

Lessons Unlearned: Infamous Cases of AI Misuse in Law

The very first sanctions case on Charlotin’s list, Mata vs. Avianca, emerged in June 2023. This New York personal injury case resulted in a $5,000 penalty for two lawyers whose legal brief, largely produced by ChatGPT, cited at least nine nonexistent court decisions. The case received widespread publicity.

One might assume such debacles would deter lawyers from relying on AI chatbots. However, that's not the case. Charlotin suggests that the convincing, authentic-seeming output of AI bots might lead overworked or less diligent lawyers to accept false citations without proper verification. “AI is very good at looking good,” he stated. Legal citations follow a standardized format, making them “easy to mimic in fake citations.”

It's also possible that early sanctions, often just a few thousand dollars, weren't severe enough to grab the legal community's attention. But Volokh argues that the non-monetary repercussions of submitting false citations should be far more concerning than financial penalties. “The main sanctions to each lawyer are the humiliation in front of the judge, in front of the client, in front of supervisors or partners..., possibly in front of opposing counsel, and, if the case hits the news, in front of prospective future clients, other lawyers, etc.,” he explained. “Bad for business and bad for the ego.”

Charlotin’s dataset offers examples that are amusing, albeit mortifying for the lawyers involved, showcasing practitioners seemingly oblivious to their technological environment.

Steven A. Schwartz, the lawyer who prepared the hallucinatory ChatGPT filing in the Avianca case, later testified he was “operating under the false perception that this website could not possibly be fabricating cases on its own.” When he suspected the cited cases were fake, he sought reassurance—from ChatGPT itself. “Is Varghese a real case?” he asked the bot. “Yes, it’s ‘a real case,’” it replied. Schwartz did not respond to requests for comment.

Other cases highlight the dangers of misplaced trust in AI.

Last year, Minnesota Attorney General Keith Ellison hired Stanford communications professor Jeff Hancock for an expert opinion on AI-faked material in politics. Ellison was defending a state law criminalizing such distribution in political campaigns, a law challenged as a free speech infringement.

Hancock, a respected expert on the social harms of AI-generated deepfakes, submitted a declaration. However, Hancock’s declaration contained three hallucinated references, apparently from ChatGPT, which he had consulted. One error even misattributed an article he himself had written; he only noticed after the plaintiffs pointed it out.

Federal Judge Laura M. Provinzino noted what she termed “the irony” of the situation: “Professor Hancock, a credentialed expert on the dangers of AI and misinformation, has fallen victim to the siren call of relying too heavily on AI — in a case that revolves around the dangers of AI, no less.”

This angered the judge. Hancock’s fake citations, she wrote, “shatters his credibility with this Court.” Citing his attestation to the declaration's veracity under penalty of perjury, she discarded his entire expert declaration and denied permission to file a corrected version.

In a statement to the court, Hancock suggested the errors might have resulted from a copy-paste mistake, while maintaining his declaration's points were valid. He did not respond to requests for further comment.

The Real Price of AI Errors: Reputation and Rising Penalties

On February 6, Michael R. Wilner, a former federal magistrate acting as a special master in a California federal case against State Farm Insurance, imposed $31,000 in sanctions on two plaintiff law firms. Their brief contained “numerous false, inaccurate, and misleading legal citations and quotations.”

In this instance, a lawyer used an AI bot to help prepare an outline for associates writing the brief but failed to inform them of the AI's involvement. Consequently, the associates accepted the outline's citations as genuine without double-checking.

Wilner noted that “approximately nine of the 27 legal citations in the ten-page brief were incorrect in some way.” He opted not to sanction individual lawyers, calling it “a collective debacle.”

Wilner added a chilling observation: the bogus citations initially almost convinced him of the plaintiff's case's soundness, until he looked them up. “That’s scary,” he wrote. His monetary sanction for AI misuse is reportedly the largest in a U.S. court to date.