Numbers and Stats: Evaluation and the Legal Profession

Many people in the legal sector are working toward making the legal profession, and society as a whole, more equitable, which has led to calls to increase diversity in the legal profession. Employers and educators have tried to achieve more impartiality by integrating quantitative metrics, such as LSAT scores, grades, and litigation statistics, into their selection processes.

This is better than selecting candidates based primarily on criteria like family legacy, but these processes still aren’t perfect. The selection of athletes a a good comparison. Elite athletes are a group of people with well-defined and generally available statistics going back to childhood, and they are assessed by systems designed to ensure their performance is objectively evaluated. However, deeper analysis shows clear patterns of inequity.

In a famous example from his book Outliers, Malcolm Gladwell observed that though the births of people with particular aptitudes can be expected to be distributed randomly in the year, the most common birth months for major league hockey players are in the first quarter. This is because when they start playing, they were bigger and stronger than players born later in the year. The selection processes in place for junior players mean they receive better coaching and opportunities. This solidifies their advantages as they mature, creating the type of pattern that can also exist in developing talent in other fields.

Leah Howie and Brea Lowenberger, whose book Creating a Seat at the Table: Reflections from Women in Law was published in October, expressed concerns about how apparently objective measures affect access for marginalized groups in law. Law schools serve as important gatekeepers for access to the legal profession and by extension many positions of power, and their processes of evaluation for admission and evaluation can be problematic.

The LSAT is one example of an evaluation tool that assigns a numerical score to something that is difficult to measure and for which correlation with long term outcomes is not clear. It is designed to predict success based on grades in the first year of law school. It also advantages some applicants while disadvantaging others, and there is limited discussion on whether grades in the first year of law school are a good benchmark for admissions.

Grades are also used in law school admissions and for selection for jobs and other competitive roles, but high grades may not be the best way to predict success over a career. As Howie observed: “An A doesn’t necessarily mean that someone learned more: they performed better in a three-hour window on a random day in December.” Doing well in exams in particular is associated with lack of anxiety, and as law schools grade on a curve, law professors are ranking students in these situations in ways that may affect the rest of their careers. Many law schools have recognized this and have moved away from grading entirely based on exams.

The performance statistics used in many situations are not equitable ways to allocate competitive positions. Some of the most important attributes for success are a willingness to learn and explore, and interpersonal skills. Lowenberger observed that defining what is being measured and why that outcome is desirable is required to make fairer systems of evaluation. Measures should be designed to gauge individual skills. According to Lowenberger, high grades may be correlated with self-awareness – where students understand what they are likely to do well in and choosing those opportunities accordingly. This can create pressure to avoid trying new things. People in power tend to have benefited from existing structures, and these systems are comfortable for them, so they are unlikely to initiate change.

The example of hockey players shows that streaming people into tracks with expected outcomes happens early, and it is difficult for individuals to switch once this happens. The longer the timelines over which metrics are collected, the more difficult it is to change people’s career trajectories. When asked how someone can approach this situation, Howie said that much of it is related to having the right sponsors and mentors. However, Lowenberger cautioned that though this is true, bad advice can derail a career.

The desire to use objective measurements to evaluate potential in legal careers as a way to reduce bias is admirable, but it is not adequate to require individuals to successfully navigate systems that use metrics that measure things that are not tied to relevant outcomes. Howie suggested clarifying what a fair outcome would look like and how to evaluate it, saying it may be necessary to look outside the legal discipline for these answers. Lowenberger suggested looking around the table to see who is making decisions and ensuring different perspectives are represented as a way to develop better processes of evaluation.