Measuring Legal Service Value, Part 2

What makes a great law firm? How can one quantify just how great a firm is, and compare it to its competitors? Last time in this space I suggested that legal service value has four elements (full paper here):

• To the extent that a firm gets good legal results for its clients, it has effectiveness value.
• To the extent that the firm’s fees are low and easy to pay, it has affordability value.
• The more the firm’s practices minimize clients’ time and stress costs, the more client experience value it has.
• Finally, if the firm’s work has many benefits and few costs for people other than its clients, it has high third party value.

In principle, a firm’s performance on these four elements of value can be quantified. If we could actually create accurate charts like these for legal services providers, we would have more empowered consumers, better self-awareness of strengths and weaknesses within law firms, and more evidence-based regulation. This project can also move us toward a more meritocratic legal profession, in which individual success is less dependent on racial and socioeconomic privilege.

An accurate quantification of value will require multiple metrics, and the array of metrics will depend on the legal niche in question. This column outlines three basic types of metric for quantifying legal service value: output metrics, internal metrics, and input metrics. Output metrics measure what comes out of the legal service provider. Internal metrics examine what happens inside the firm. Finally, input metrics analyze the people who walk in (or log in) to the firm each morning.

Output Metrics

Output metrics may analyze the outcomes created by law firms, or their work products. A simple example is win/loss rate, which can be informative in some administrative litigation niches if properly designed. Judges in England & Wales are now evaluating the advocacy of all lawyers who appear before them in criminal matters.

The best proof of a pudding is in the eating. Output metrics are, in principle, the best way to evaluate legal service value. If we can tell from its outputs that a firm consistently produces great, highly affordable results for its clients, while leaving them happy and doing good in the world, we can be confident that it is a great law firm.

Unfortunately, output metrics for legal services are methodologically troublesome. They must be valid — they must measure what we are actually interested in instead of something else. Measuring affordability value by comparing the average bills of three law firms over the last 10 cases handled by each would not work, because any differences would be just as likely to reflect the complexity of the cases they handled. Especially in contested matters, the outcome depends on a wide range of factors other than the inherent value of the legal service provided.

One egregious example of an invalid output metric is the personal injury “Litigator Awards” handed out by the “Trial Lawyers’ Board of Regents” in the United States. This metric is based on whether or not a firm has settled cases for more than certain amounts within a set period. However no account is taken of the inherent strength of the cases in which these results were obtained. Thus, receiving a “Litigator Award” may well speak more to the prowess of a personal injury firm in attracting high-value cases than it does to the effectiveness of its advocacy for those clients.

Even if valid, an output metric may still lack reliability if the sample size is too small. For example, surveying past clients can be a good way to assess client experience value. However, because clients have idiosyncratic expectations of their lawyers, and some client evaluations are unreasonable or biased, the results cannot be relied upon unless a sufficient number of clients were surveyed.

Internal Metrics

Looking at what happens inside law firms is an an alternative to scrutinizing outputs in the effort to identify value. Most consumers aren’t interested in what goes on inside — they care about the outputs. However certain practices and structures demonstrably increase the chance that high-value services will be delivered. Checking for these practices and structures can be a helpful way to identify the firms likely to consistently produce great value, especially when output measures are methodologically impossible.

Some internal metrics focus on processes — what lawyers, and other people involved in legal service provision, actually do when working for a client. Common sense practices such as actively listening to clients and giving them a single point of contact within the firm have been empirically demonstrated to produce client satisfaction.

Structural attributes can also be the basis for good internal metrics. A firm that is free from harassment, racism, sexism, conflict, and excessive turnover is likely to produce better value.

The methodological challenge of internal metrics is that the information is private. Self-reporting is one way to get at this data. Peer file review and audit are somewhat more forceful techniques sometimes used by regulators and legal aid funders.

Input Metrics

How many years of experience does the lawyer providing the service have? What were her bar exam scores? How many practice hours have the firm’s staff accumulated in the niche? These are examples of input metrics, focused on the attributes of the people who do the work.

The advantage of such input metrics is that the data are relatively easy to gather and compare. The disadvantage is that this data seems to have a weak relationship at best with meaningful elements of value. Potentially more fruitful are surveys identifying the knowledge or skills that are most important for other lawyers to possess. If a variety of substantive knowledge (e.g. of the Criminal Code) or a personal skill (e.g. persuasive writing) demonstrably helps practitioners provide high quality services in a certain niche. If we have data about the extent to which different practitioners have these attributes, that can help us quantify and compare the value they offer.

A Steep Path, Worth Climbing

The value of a legal service is a complex aggregate of its effectiveness, affordability, client experience, and third party effects. Measuring it accurately requires gathering data from multiple sources, using a variety of methodologies. A “tyranny of metrics” that are misleading or counterproductive might be even worse than the status quo, in which the true relative value offered by different firms is mysterious, even to insiders.

However the author’s view is that it is entirely possible – and entirely worth the effort — to create accurate, objective methods to quantify the value propositions of different firms in different legal niches.

For a more detailed account, please see https://ssrn.com/abstract=3144771