How Hard Could It Be to Write a New U.S. Bar Exam?

The bar examination landscape in the United States seems to be in a state of upheaval. The National Conference of Bar Examiners (NCBE) is in the midst of developing a new bar exam, called the Next Gen Bar. Meanwhile, California’s State Bar considered scrapping NCBE entirely and hiring Kaplan to write an exam for California.^[1] Though California has now paused this plan, it’s clear that they were looking for other options and thought it might be possible to write a new exam.

I once heard a teaching expert joke that she could teach anything. She would take a bet that she could teach a dog to read: she could certainly stand in front of the dog and deliver a lecture on the methods of reading. She would win the bet, she said, because no one specified that the dog must learn! Learning requires much more than listening to a lecture, but too often lecturers focus on their own actions and fail to consider whether students are engaging with the material or just sitting, dog-tired, with uncomprehending minds.

The idea of creating a bar examination quickly seems to be the domain of those who conflate teaching with learning. Could California or another state hire a company to write an exam in a few months? Certainly! (If they just wait a few more months they can probably use AI to write it for them). Would that examination be a fair and valid way to distinguish test-takers? I would have some concerns.^[2] Writing examination-style questions is only a moderately difficult task, but writing questions at the correct level of difficulty to evaluate professional-level abilities is nearly impossible. If a question is too easy, a “good” examinee may overthink it and choose the wrong answer and if a question is too hard, a “bad” examinee may have as much luck as a good one at guessing the right answer. A question must be Goldilocks-perfect in order to reliably separate the test-takers.

Before NCBE announced it was writing a new bar examination, 41 jurisdictions were using the Uniform Bar Exam, NCBE’s current test offering. If NCBE had not chosen to innovate, its exam would have fallen out of alignment with the practice of law and the skills needed by modern lawyers. But by developing a new exam, the NCBE seems to have given jurisdictions the idea that it is possible to write a new exam themselves and bypass the NCBE. Hopefully they will not conclude that it would be easy. While California has paused its plan for now, only 19 jurisdictions have signed on to the Next Gen Bar Exam, indicating that California may not be the only jurisdiction praying that another option will miraculously appear.^[3]

Meanwhile, the NCBE, while far from perfect, at least appears to understand that the questions must be tested and the process cannot be rushed. They have a plan for pilot testing, field testing, and a prototype examination.^[4] The NCBE conducted its pilot testing from August 2022 to April 2023 and has just released the results in recent months. The NCBE noted that during its pilot testing, test-takers “identified interpretive issues in the questions, instructions, and fact-pattern language.” In plain language, they identified that the questions were not perfect and needed to be revised. Any jurisdiction planning to use a bar exam should be asking how the test-writers intend to evaluate and revise their test. How will they measure its validity? Assuming NCBE’s plan and timeline is reasonable, how would another test-maker do a reasonable amount of pilot testing and field testing after having begun two years later?

State boards of bar examiners would do well to remember that their choices will affect people’s lives. They should evaluate the NCBE’s test and any other options by remembering that the test must be valid. It must separate those with the knowledge and skills to practice law from those who do not have those qualities. And it must do this repeatedly and reliably. Because many law school graduates receive job offers which are contingent on their first-time bar exam passage, failure on the exam results in lost careers, not just lost time. With those stakes, designing a bar exam cannot be considered easy.

___________

^[1] Karen Sloan, https://www.reuters.com/legal/government/california-shelves-plan-create-its-own-bar-exam-2024-05-16/

^[2] My concerns and views are my own and do not reflect the viewpoint of my employer.

^[3] https://nextgenbarexam.ncbex.org/illinois-adopts-nextgen-bar-exam/

^[4] https://nextgenbarexam.ncbex.org/reports/research-brief-pilot-testing/