Deceptive Dynamics of Generative AI: Beyond the “First-Year Associate” Framing

Guidance for lawyers on generative AI use consistently urges careful verification of outputs. One popular framing advises treating AI as a “first-year associate”—smart and keen, but inexperienced and needing supervision. In this column, I take the position that, while this framing helpfully encourages caution, it obscures how generative AI can be deceptive in ways that make it fundamentally dissimilar to an inexperienced first-year associate. How is AI deceptive? In short, generative AI can fail in unpredictable ways and sometimes in ways that mimic reliability, making errors harder to detect than those flowing from simple inexperience.

Before elaborating, three important caveats about scope. First, this column focuses on guidance given to lawyers in response to generative AI’s unreliability. This focus is not meant to imply that other concerns relevant to responsible AI use—like data security or potential bias—are unimportant. Second, the risks posed by generative AI’s unreliability vary by task; accuracy matters less when brainstorming alternative wording for a factum than when conducting substantive legal research. The cautions offered below will be more or less salient depending on how generative AI is being used. Third, concerns about unreliability vary by tool. The generative legal AI landscape is vast and diverse, encompassing hundreds of specialized legal products, general-purpose tools such as ChatGPT and Microsoft Copilot, and AI features built into software like videoconferencing platforms and PDF readers. The deceptive dynamics detailed below will not manifest equally across all tools, both because tools serve different functions and because developers employ different interfaces and safeguards.

With that context in mind, how might generative AI tools and outputs be deceptive in ways that aren’t obvious in the portrayal of AI as a first-year associate?

First, and most obviously, generative AI outputs can be fabricated. A lawyer reviewing a first-year associate’s work likely expects some errors flowing from inadequate research or an incomplete understanding of the law. They do not suspect straight-up fictitious content. As Jason Harkess observes, “[h]uman errors typically involve misinterpretations, incomplete research, or analytical missteps. AI hallucinations, by contrast, often involve complete fabrications that appear superficially correct but lack any basis in reality.”

Many lawyers now know that generative AI can generate fake case citations, so this form of deception is increasingly well-understood. However, continued submissions of fabricated authorities to courts suggest the lesson hasn’t been universally absorbed. In any case, there is also the lesser-known prospect of subtle hallucinations: a date altered here, part of a legal test changed there. These more subtle hallucinations are harder to detect and mean that where accuracy is paramount, extreme caution and rigourous verification is warranted when relying on AI outputs. In some situations, the vetting burden may, in fact, outweigh any efficiency gains. It isn’t always a simple matter of double-checking a tool’s outputs – it may be that using generative AI doesn’t make sense in the first place or that you need to consider using it in a more constrained way. Contrast, for example, using generative AI in a legal research task to arrive at useful Boolean search terms (h/t Katarina Daniels) versus trying to use the technology to generate a full research memo or factum from scratch.

Second, generative AI outputs are often fluent. The text these tools produce is generally polished, characterized by clean formatting, correct grammar, and familiar language patterns. This is a strength of these tools – no one wants sloppy or error-ridden text. The problem is that psychological research shows that processing fluency—the ease with which we read text—can breed overconfidence in a text’s substantive truth and trustworthiness. When reviewing a first-year associate’s work, a lawyer is likely to take comfort in well-presented materials. Careful grammar, writing, and editing demonstrate diligence which, at the very least, indicates that the work was not hurried. With generative AI outputs, this heuristic breaks down. As Jack Shepherd has observed, “…humans do sometimes produce bad but well-presented work. L[arge] L[anguage] M[odel]s always do. That makes it harder to spot the errors.” The refinement of generative AI outputs can mask quality issues for the unsuspecting.

Third, generative AI can be fickle – or at least appear so to users who lack deep technical understanding. The types of errors produced by generative AI technology may be unexpected, meaning we are not primed to look for or recognize them. As a 2023 Harvard Business School study notes, generative AI has a “jagged frontier” in which tasks that we might expect to be difficult can often prove easy, while seemingly simple tasks sometimes end up posing big challenges. Take, for example, ChatGPT’s strong performance in creating poems that mimic Shakespeare’s style versus its inability, at least at certain points in time, to correctly count the number of times that the letter “r” appears in the word “strawberry”. This dynamic complicates verification. Lawyers may not realize they need to double-check outputs on tasks that would be trivial or very easy for a first-year associate. While technical explanations exist for these “common sense” failures, they are likely to be unexpected by non-experts (and most lawyers are non-experts in AI).

Fourth, and related to the above, generative AI outputs can be fragile. To quote Shepherd again, “large language models are designed to produce different output each time, even if you use the same prompt. The level of randomness can be adjusted through the ‘temperature’ parameter, although if you turn it down too much, the output becomes very robotic and unhuman-like. It is inherent in the design of these models that the output differs each time.” While we might start expecting regularity and consistent outputs once an associate does a task several times, the same will not necessarily hold for a generative AI tool. This reality of the technology also means that it can experience unexpected failures even on tasks it has previously handled well. For lawyers, this sort of fragility means that past performance cannot always be used reliably as a proxy for future quality.

Fifth, generative AI technology can lack faithfulness. “Faithfulness” is a technical term for a specific type of AI deception, helpfully summarized in this paper by Katie Matton et al.:

Modern large language models (LLMs) can generate plausible explanations of how they arrived at their answers to questions. And these explanations can lead users to trust the answers. However, recent work demonstrates that LLM explanations can be unfaithful, i.e., they can misrepresent the true reason why the LLM arrived at the answer.

In other words, when generative AI explains its reasoning, it is possible that the explanation given does not reflect how it actually produced the output. To be clear, the faithfulness problem does not arise with all tools, or attach to all explanations, and the precise conditions under which unfaithful explanations arise is an ongoing area of research. Even with this wobbliness, it strikes me that lawyers should be aware of the possibility of unfaithful explanations. The practical issue, to quote Matton et al again, is that “misleading explanations can provide users with false confidence in LLM responses, leading them to fail to recognize when the reasons behind model recommendations are misaligned with the user’s values and intent.” This dynamic is particularly troubling because lawyers are likely to consider the reasoning and process used to reach a conclusion as providing some indication of the reliability and appropriateness of the conclusion. When AI provides explanations that appear logical but are fundamentally unfaithful, it exploits this professional instinct, potentially leading lawyers to place unwarranted trust in flawed outputs.

By pointing out that generative AI tools and their outputs can be deceptive through fabrication, fluency, fickleness, fragility, and unfaithfulness, my point here is not that this technology is so irredeemable that it should be eradicated from lawyers’ offices. Again, let’s recall that not all generative AI uses for lawyers hinge on generating factually accurate outputs. Also again, the world of legal AI is large and some tools may include features or safeguards that mitigate certain of these risks. On top of this, we know that all sorts of shortcomings can attach to human performance. We cannot engage with anywhere near the amount of data that AI systems can. We bring our own biases to tasks. We get tired and hungry.

The intended take-away here is not “machines bad, humans good”. The relatively modest plea I am making is to appreciate that, if we are directing lawyers to verify generative AI outputs – an undoubtedly wise direction – the “first-year associate” framing only takes us so far. What unites the five deceptive dynamics outlined above is that they all involve forms of deception that we are not primed to necessarily expect (or to expect in the same form) from human colleagues, even inexperienced ones. Generative AI outputs can be fabricated but appear authentic, polished but flawed, error-prone in unexpected ways, successful until they’re not, and accompanied by explanations that may themselves be unreliable. Understanding these distinctive deceptive dynamics equips lawyers to develop verification practices tailored to this technology’s particularities, rather than relying on instincts developed for supervising human colleagues. The “first-year associate” framing asks lawyers to supervise; understanding these dynamics reveals what they must supervise for—and why that task can sometimes be more challenging than the framing suggests. Better understanding breeds better practice, and, ultimately, safer use. Indeed, as these deceptive dynamics become better known and incorporated into practice, their power to deceive will diminish.