Straight-Up Made-Up: the Ongoing Saga of Problematic AI-Generated Legal Submissions

Don’t include fake cases in court submissions. Don’t miscite cases for wrong propositions of law. Don’t refer to cases in court if you haven’t read them. This is the most basic of lawyering stuff. Probably too basic to even be included in an Advocacy 101 course. Yet, over the last two years, lawyers have made headlines for doing exactly these things. Their common partner in misadventure? Generative AI.

The landscape of AI-generated errors in court submissions

AI-related mishaps first came to the attention of the legal community (and the rest of the world) in May 2023 when an American lawyer misused ChatGPT for legal research and made the front page of the New York Times after he included references to non-existent cases in court filings. When asked by the court to explain himself, the lawyer stated, “I did not comprehend that ChatGPT could fabricate cases” and indicated he was “embarrassed, humiliated and deeply remorseful” about not taking any steps to verify the accuracy of the tool’s outputs. Given the wide coverage this case received, many assumed that the days of lawyers misusing AI were numbered – surely lawyers would have gotten the message not to uncritically trust the outputs of ChatGPT and other generative AI tools.

Unfortunately, more than two years later, we have many more examples of the same thing happening. French researcher Damien Charlotin has created a database to track legal decisions that reference “hallucinated” content. Over 150 instances are currently listed (this number includes cases involving self-represented litigants). Worryingly, the database suggests that the “fake case” trend is actually accelerating, with the vast majority of instances occurring in 2025. And, of course, these numbers no doubt represent an under-counting given that not every instance of problematic AI-generated legal content makes its way to a written decision and not every written legal decision gets publicly reported.

In Canada, there are four reported cases of lawyers including problematic AI-generated content in court submissions that I am aware of:

• In February 2024, a lawyer appearing before the Supreme Court of British Columbia in a family law matter had costs awarded against her personally after she included two non-existent cases, generated by ChatGPT, in a notice of application filed with the Court. Like the American lawyer mentioned above, this lawyer was not aware that the technology could produce fictitious authorities and was “remorseful” and “deeply embarrassed”. (Zhang v Chen, 2024 BCSC 285)
• In April 2025, a lawyer appearing before the Federal Court was admonished after the Court was unable to locate two cases cited in filed materials. The Court observed that the AI tool used in this case (“a professional legal research platform designed specifically for Canadian immigration and refugee law practitioner”) also hallucinated a legal test and “cited, as authority, a case which had no bearing on the issue at all.” This use of generative AI was undeclared, notwithstanding a Federal Court notice requiring disclosure. The Court found that a costs award (in an amount to be determined) was appropriate given “that the use of generative artificial intelligence [was] not only undeclared but, frankly, concealed from the Court”. The Court further found that, with respect to any costs that are awarded, there should be consideration as to whether the lawyer should be required to pay any of those costs personally. (Hussein v. Canada (Immigration, Refugees and Citizenship), 2025 FC 1060)
• In May 2025, a lawyer appearing before the Ontario Superior Court of Justice in a criminal law matter was required to show cause as to why she should not be held in contempt of court after her factum was found to include “references to several non-existent or fake precedent court cases.” Prior to a contempt hearing, the lawyer informed the Court that her staff, unbeknownst to her, had used ChatGPT when preparing the factum. The lawyer took full responsibility, expressed deep regret for what happened, committed to taking continuing education training in technology and legal ethics and said that she implemented new protocols in her office to prevent this from happening again. Finding that the purposes of a contempt hearing were already met, the Court dismissed that proceeding on the conditions that the lawyer: (1) fulfill her commitment to take continuing education training and (2) not bill her client for the research, factum writing and attendance at the underlying motion. (Ko v. Li, 2025 ONSC 2965)
• In May 2025, a lawyer appearing before the Ontario Court of Justice in a criminal law matter was ordered to prepare a new set of defence submissions after the Court observed that the original submissions included: a case that appeared to be fictitious; several case citations to unrelated civil cases; and cases cited for legal propositions not found in the cases. There was no explicit finding in the reasons that generative AI was used to prepare these submissions, with the Court noting that there would be “a discussion at the conclusion of the trial about how the defence submissions were prepared.” That said, the Court’s suspicion that generative AI had been used was apparent in the Court’s specific requirement that “generative AI or commercial legal software that uses GenAI must not be used for legal research for [the new submissions]” ( v. Chand, 2025 ONCJ 282)

Why should we be concerned with problematic AI-generated content?

It goes without saying that fabricated or misleading legal submissions are bad. No one is applauding these lawyers. That said, it is worthwhile to unpack, a little bit, exactly what is at stake when faulty generative AI submissions make their way into Canadian courtrooms.

Most immediately, there are concerns about the proper administration of justice. As noted by Justice Myers in the Ko v. Li case,  “a court decision that is based on fake laws would be an outrageous miscarriage of justice to the parties.” Legal disputes must be adjudicated on the basis of actual, not fabricated, legal authorities.

More systemically, we should also be concerned about how fabricated legal authorities might impact the development of the common law. Our common law system works by building on precedents established in previous cases. The insertion of fake cases, if relied upon, risks our legal system moving forward on the basis of what are essentially fraudulent pretenses. We want the law to develop in accordance with measured judicial reasoning, not machine delusions.

Some good news: we have, in general, a very diligent judiciary with effective systems in place to catch fake cases. I am not aware of any reported decision where a judge has inadvertently relied on a fictitious authority in deciding a case. However, as I’ve previously written in Slaw, generative AI tools may include subtle legal errors in their outputs – a word change here, a legal test slightly altered there. These insidious legal errors can be much harder to catch than straight-up made-up cases but still risk impacting the outcomes of disputes and undermining the development of the common law. I’m less confident that none of these subtle errors have become embedded in reported decisions.

In any case, even if courts catch all fabricated or otherwise erroneous generative AI-produced legal content, there are still other important costs to consider:

• Wasted judicial resources: There are well-established problems of delay and backlog in Canadian courts. Judicial time and other court resources are at a premium. When a judge has to deal with problematic AI-generated legal submissions, their time and attention are hijacked away from other important tasks.
• Public confidence: The public expects that if they have a legal problem, they can turn to lawyers who are experts with extensive (and expensive!) training to help them. When the public sees media reports involving lawyers apparently passing off their work to cheap chatbots, their trust is undermined. As Jordan Furlong has aptly put it, “bringing lawsuits that cite non-existent cases leads the average person to regard lawyers as lazy, credulous, or completely indifferent to the truth.” It may make the public wonder if they would be better off just going it alone.
• Chilling lawyer interest in technology: When lawyers see things going wrong – really wrong in a very public way – when another lawyer uses technology, this can feed into what might already be risk-adverse tendencies in the profession. Lawyers may end up being scared of touching AI at all. This is the wrong reaction. While lawyers should be cautious about using generative AI, they are also well served by being curious, learning more about the technology, and becoming confident in responsible use and best practices (see here for more). Generative AI is entering lawyers’ lives from all directions, from being embedded in long-standing tools, to being used by clients, to arising in evidentiary matters. Burying one’s head in the sand isn’t a viable strategy.

What can be done?

For many lawyers, the fact that colleagues have misused AI and submitted problematic AI-generated to courts is old news. They saw the New York Times headline in 2023; they’ve seen the media reports and the LinkedIn posts. I actually hesitated to write about this topic at all. Why write about the “fake case” cases when there are so many other flashy AI bobbles to ponder? However, after looking at the increasing rates at which lawyers are submitting problematic AI-generated content, and considering the harms noted above, the topic took on new urgency. We need to be talking about this and trying to stem the tide.

One common response to this issue is the observation that these lawyer mishaps are not really about technology but about a lack of basic care and diligence. If the lawyers actually tried to read the cases that they were citing – an important step in any litigation matter – they would have discovered that the cases weren’t real and would have presumably quickly deleted them from their submissions. There is no indication that any of these lawyers were purposely including fictitious citations. The root problem is sloppiness, not malfeasance.

This is all true. However, I also think that we do ourselves a disservice if we don’t examine the technology angle to all of this. Generative AI tools have features that create a perfect storm for this sort of thing to happen: they are deceptive, unpredictable and easy to access. Understanding and responding to these dynamics is part of an effective solution.

For example, the interface of a tool like ChatGPT looks like a search bar that we’re very familiar with from online search engines of databases. Our default use for such search bars has been to look up information and we are accustomed to the output being either a set of (real) results or an answer that no results are available. But generative AI doesn’t work this way. It is fundamentally a probabilistic technology that is guessing which words go best together. Tools built on this technology “are based on complex mathematical systems that learn their skills by analyzing enormous amounts of digital data [and]…do not — and cannot — decide what is true and what is false.” Another way of putting this is that tools like ChatGPT are not best understood as information retrieval devices, but rather as word-creation machines.

To be sure, such tools can be extraordinarily powerful text creation machines. And, to be clear, there are lots of interesting ways for lawyers to responsibly use generative AI (a topic for another column!). At the same time, the sophistication of this technology is, in certain respects, what makes it particularly dangerous when it comes to legal research. Generative AI tools can produce fabricated citations that look extremely convincing – they can be in the correct citation format and have plausible sounding case names. In Ko v. Li, the lawyer was even given weblinks to CanLII (one of which returned a “404 Error – Page not found” and another which directs to an entirely different case than the case named).

This kind of deception is paired with unpredictability – a generative AI tool may give correct legal information and accurate citations to certain inquiries but make unexpected errors at other times – sometimes even within the same response. Indeed, in some ways, the better the technology gets, the riskier it gets. The more a user sees accurate outputs, the more they may gain a false sense of confidence, particularly when the inaccurate outputs that pop up are so facially persuasive or are of the more subtle variety.

On top of this, we are used to vetting content for human errors and not AI errors, which can be much different. A low-quality factum written by a human is likely to contain weak writing or lack citations to authorities. A low-quality AI-written factum can have excellent writing and many supporting authorities (with hyperlinks to boot!). If someone is unacquainted with the limits of the technology, they can be easily fooled. We are used to using form as a proxy for quality, but AI turns this heuristic on its head.

These risks are compounded by how easy it is to use the technology even if one doesn’t fully understand its limitations. Previously, AI tools for legal professionals were largely gatekept by vendors – a lawyer or other legal professional had to make a conscious decision to purchase the technology and spend a significant amount of money to acquire it. Presumably, those willing to spend the money on AI tools exercised a degree of diligence and learned about the technology before they started using it. Now, anyone can upload ChatGPT for free on their computer. And, increasingly, generative AI functionality is being pushed out into commonly used legal tech tools. While such tools often bring particular safeguards, studies still generally show persistent problems with accuracy when it comes to generative AI and legal research.

The ease and speed with which one can produce good-looking outputs is tempting for lawyers working under enormous time pressure and stress. And without a keen understanding of the technology’s limits, generative AI is a bright and shiny “EASY” button just waiting to be pressed.

We need to disrupt this temptation with appropriate education. To their credit, most Canadian law societies have issued warnings to lawyers about the possibility of fabricated outputs and the need to take care when relying on AI tools. Courts have issued practice directions and notices to this same effect. Clearly, however, not everyone has gotten the message. The risks need to be loudly and regularly broadcast. Here are some additional suggestions:

• Law societies and bar associations should continue to publicize their guidance and offer free or low-cost education events.
• Judges should strongly consider issuing and publicising written reasons in cases where they encounter problematic AI-generated content. The more lawyers hear about this happening and see serious consequences befalling their colleagues, the more they will avoid falling into the same trap.
• Legal workplaces should institute AI-use and disclosure policies. Some of the existing cases involve staff or outsourced professionals mis-using generative AI in a manner unknown to the lawyer who ultimately files the problematic submission. While, ex post, there is no doubt that the lawyer still bears responsibility for the content submitted, there are also ex ante steps that can and should be taken to prevent this from happening in the first place. Staff need to be alerted to the risks of AI and have clear guidance about whether and when to use it. Both the Law Society of Ontario and the Law Society of Alberta now offer guidance for firms about how to craft internal AI use policies (see here and here). When lawyers outsource legal research or drafting work to third parties, they should consider including explicit provisions about AI use (including disclosure expectations or perhaps even non-use) in their contracts.

We also need to think more deeply about how lawyers interact with AI technologies. A common refrain in AI guidance to lawyers is to “verify outputs” and/or ensure that there is a “human-in-the-loop.” I have no quibble with the general sentiment here, but equally important is building and promoting a more nuanced understanding about the evolving relationship between lawyers and AI beyond “add human and stir.” A promising thread can be found in the concept of “verification drift”, a term coined by Armin Alimardani. As he describes it:

Why did some students, even after receiving training and feedback on responsible GenAI use, fail to adhere to the provided instructions? This question is complex, but one possible explanation is a phenomenon I termed “verification drift.” This occurs when users initially approach AI-generated content cautiously, aware of the risks of inaccuracy and the potential for hallucination. However, as they engage with the content, they gradually become overconfident in its reliability and find verification less necessary. This misplaced trust may stem from GenAI’s authoritative tone and ability to present incorrect details alongside accurate, well-articulated data. This suggests that the challenge is not just a lack of awareness but a cognitive bias that lulls users into a false sense of security.

Understanding the human-machine interaction and the cognitive biases at play is essential. (Brad Wendel’s excellent Substack post from last month on “fake case” cases makes a compelling call along these same lines and is worth a read).

Ultimately, though, I can’t see any simple way to quickly inoculate our courts from the scourge of straight-up made-up cases and other problematic AI-generative content. Lawyers taking short-cuts isn’t a new problem. And, in this context, it isn’t an easy problem to solve, especially when a shiny new technology is so willing to help them do so. I expect the examples in Charlotin’s database to continue to grow. It will take some time to right this ship and it may well be an ongoing battle. Courts must continue to be vigilant in vetting the legal submissions they receive and, indeed, given the risk of subtle hallucinations, I would argue that some amount of judicial hyper-vigilance is warranted. Sneakily-wrong AI can fool the best of us. Given the stakes, both lawyers and judges must be on guard.