There is a growing trend to bring the tools of data analysis into legal practice, and much of the media coverage references Michael Lewis’ 2003 book Moneyball: The Art of Winning an Unfair Game*, or perhaps the movie based on the book:
With so many things being described as being “like moneyball”, I started to wonder what Moneyball actually said. It also made me wonder what analytics can best be used to illuminate legal information, and where the data could come from, so I read the book:
Lewis discusses the development of the statistics required to adequately measure baseball players at length. The original statistics that were used to measure teams’ and players’ performance, like batting average, were not closely correlated with what really matters in baseball — winning games. It turns out that easily measured things like the ability to run fast are not the useful statistics to use when assessing whether to recruit a player and how much to pay, but: “As bad as they may have been, the statistics used to evaluate baseball players were probably far more accurate than anything used to measure the value of people who didn’t play baseball for a living.”
Baseball leagues keep detailed records of every game and every interaction, which can be analyzed for patterns of what leads to a winning team. In Moneyball Lewis explains that when people wanted to analyze the statistical relationships between particular elements of play and game outcomes that the leagues weren’t maintaining, they had to compile them themselves. When they had better insights into what elements of play led to more wins better than most of the league, it created an opportunity for Billy Beane and the Oakland A’s to create a winning team in the early 2000s that fit their budget with under-valued players by focusing on offensive play and disregarding defensive play, which was less important for winning baseball games. It was the information asymmetry in analysis of players’ value that created this opportunity. I am informed that the Oakland A’s now have excellent defence, because the value of defence isn’t as overvalued as it used to be, and apparently this shifts on an ongoing basis (thank you Xavier).
Extrapolating this observation to evaluation of individuals in the legal profession, it is difficult for any individual lawyer or judge to have sufficiently large numbers of matters of any particular type to have a viable record for statistical analysis to assess effectiveness with a high level of confidence. The Oakland A’s were able to assess players based on thousands of times at bat in similar conditions. Even among the elite baseball players they considered drafting, the college age players were better prospects than high school age players, because they had a longer track record, which were more predictive of their success as major league players.
This same strategy is more difficult to implement in law, even if we disregard the relatively small number of interactions lawyers and judges have to count. The majority of “at bats” for a lawyer are not publicly accounted for: unlike in baseball, when a matter settles or a routine judgment is not published as a written judgment, that statistic is lost for the purposes of analysis. The Oakland A’s evaluated what skills in a player make a team win, and the best metric they could find was the ability to get on base. This is not the same as batting average — walks are an excellent way to get on base, but they weren’t being counted. They found that one of the most difficult skills to teach and under valued attributes in a baseball player is the ability to evaluate pitches and to know what is and what isn’t in the strike zone. Court judgements always represent something that was swung at.
There are traditional legal research tools that parse the numerical information in court judgments into statistical information to guide decision making in certain types of matters. The main examples of this kind of tool are quantum services, which provide numerical data with qualifying information on particulars in cases in areas like criminal sentencing, and personal injury claims. It is difficult to draw statistically valid inferences on what outcomes are most common from this data, because the majority of routine decisions of this type are not published. This means that in order to parse data for all matters, one would have to retrieve the information manually from individual case files. For the purposes of legal practice this doesn’t really matter: judges and juries have discretion to decide on an outcome that will generally fall within a particular range or to explain their reasons for going outside it. Knowing the range and what factors influence the decision is sufficient, and the reported non-routine cases are usually the ones that determine the range.
A further problem of using court decisions to statistically evaluate lawyers is that looking at outcomes is less predictive than looking at process. According to Moneyball “Too many people make decisions based on outcomes rather than process.” In movies lawyers are often described as having “never having lost a case”; I’ve heard it observed that in that case the lawyer can’t have taken any interesting or difficult cases. Having a good process to deal with all cases is much more predictive of future success. This is why I am excited about the move among law firms to do better analytics on their own internal data. In large firms it is possible to know what works incrementally better over a large scale with data analytics: the difference between a good hitter and an average hitter is small — so small that it is impossible to observe by just watching games. 40% of the time the average hitter will score more. In a closed system like law firms, it is unlikely organizations will make their data of this sort public, but it is becoming a competitive advantage to be known to use it.
The distillation of what made an offensive player valuable is a player who could discern exceptionally well whether a ball was in the strike zone, who had the restraint to not swing at those pitches, and who could combine that with the power to hit home runs. I don’t have a great deal of experience in hiring lawyers, but that seems like a good analogy for what I want in someone who would represent me. I want a lawyer who knows what not to swing at, and there’s no way to assess that ability from the data corpus of written decisions.
* M Lewis, Moneyball: The Art of Winning an Unfair Game (New York: W. W. Norton, 2003).