“Noise” and Decision-Making – Why Consistency in Decisions Matters

The divergence between the law on the books and the law as applied — and the uncertainty and unpredictability that result — exacts a price paid in the coin of injustice. ….

R. v. Ferguson, 2008 SCC 6 at paragraph 72

The Rule of Law requires that the law be accessible and “so far as possible intelligible, clear and predictable” (Lord Bingham). Daniel Kahneman, Oliver Sibony and Cass R. Sunstein have written an important book on the unexplained inconsistencies that get in the way of predictability in decision-making: Noise: A Flaw in Human Judgement. This is not a book about the practice of law, although it does discuss criminal sentencing. The relevant insights for administrative justice practitioners are how tribunals as institutions can think about inconsistencies in decision-making. The solutions proposed in this book are, for the most part, difficult to translate into the administrative justice world. The challenge for tribunals is how to combat those inconsistencies while respecting the independence of decision-makers as well as the important exercise of discretion.

What is “noise” in decision-making? Noise is different than bias. Bias is systematic deviation from the norm while noise is random and scattered. The authors use many examples, but the one that will be familiar to the legal profession is the inconsistency in both sentencing and refugee determinations. Some inconsistencies may well be due to bias, but the authors contend that much of it can be explained by noise. Bias is when most errors in a set of judgments are in the same direction. Eliminating bias, which is an admirable goal, will not, however, eliminate all errors. The errors that remain when bias is removed are the unwanted divergence of judgments. Noise is, in other words, the variability in judgments that should otherwise be identical.

We focus on bias because it is a causal explanation for errors – and the human mind resorts to easily understood causal explanations. Only a statistical view allows us to see noise. The authors contend that the absence of statistical thinking is one reason why noise receives so much less attention than bias.

The authors identify two types of noise – level noise and pattern noise. Level noise is the variability of average judgments made by different individuals. The example they give is some judges are generally more severe than others and others are more lenient. Pattern noise (or statistical interaction) is the personal, idiosyncratic responses of decision-makers to the same case. As the authors note, subtle differences among people are often enjoyable, but those differences become problematic when decision-makers operate within a system that assumes consistency.

In an interview in the Guardian, Kahneman made the important point that the noise that impacts organizations is not a phenomenon within an individual but within the organization itself. It therefore follows that, unlike bias, the answer to system noise does not depend on individual actions of decision-makers.

The authors also point out that uniformity in decision making is not always desirable. Rules applied slavishly can eliminate discretion, for example. This can lead to perceived unfairness.

The book contains some useful suggestions for non-adjudicative organizations to combat system noise. Most of these are not relevant for tribunals because they interfere with the independence of decision-makers. However, there are a few suggestions that merit some discussion for the tribunal setting.

Choosing who gets to make decisions is an important part of combating noise. Expertise or knowledge of the subject area is clearly important in decision-making, as is experience. Typically, when selecting a decision-maker, these are the areas that selection panels focus on. The skills and abilities requirements for tribunal positions can vary, but usually there is an emphasis on analytical and writing skills, as well as hearing management skills. Required personal attributes likely also included are impartiality and sound judgment.

The authors of “Noise” would agree with these criteria but likely point out that they do not go far enough. They note that judgments are both less noisy and less biased when those making the decisions are well trained, intelligent, and have the right cognitive style: “good judgments depend on what you know, how well you think and how you think.”

The authors review various tests that have been used for assessing cognitive styles and their implications for decision-making. For example, people who score high on the question “how much do you like to think hard about problems” tend to be less susceptible to cognitive biases. Also, people who score higher on critical thinking skills and who are not over-confident are also more reliable decision-makers. However, the authors focus on the only measure of cognitive style or personality that can predict forecasting performance – a measure of “actively open-minded thinking”. Those who are actively open minded routinely search for new information that could contradict their prior beliefs, are methodical in integrating that information into their current perspective, and are willing (sometimes even eager) to change their minds as a result. Of course, this cognitive style does not fit the stereotype of a “decisive leader”. To reduce error in judgment, it is better to remain open to counterarguments and to accept that you might be wrong – being decisive should come at the end of the process not at the beginning. The authors note that there is some evidence that actively open-minded thinking is a skill that can be taught.

The importance of getting the right decision-maker from the very beginning cannot be overemphasized. That is because there are significant limitations on a tribunal’s ability to ensure consistency once that decision-maker is making decisions. Paul Daly has highlighted some of the tensions in ensuring consistency in a column from 2019. However, as Professor Daly notes, it is not impossible to put in place systems for assisting with consistency – although not enforcing it.

The authors of “Noise” set out a useful concept of “decision hygiene”. Since noise is an unpredictable error that cannot easily be seen or explained, strategies for noise reduction must be designed to prevent an unspecified range of potential errors before they occur. The authors use the idea of “decision hygiene” as similar to washing your hands – you may not know precisely which germs you are avoiding; you just know that washing them is a good preventative step against a variety of germs. The authors do not discuss decision hygiene in the law context. Based on their review of techniques in a variety of other areas, including medicine and hiring, they come up with an approach that is not workable in most administrative justice decision-making. Their “mediating assessment protocol” mostly involves building consensus from multiple views which is not easily transferable to a decision-maker of one. However, they do set out some general principles that could have application to a decision hygiene model for tribunals. I have listed the ones that could have application:

The goal of judgment is accuracy, not individual expression

“When the goal is accuracy and you expect others to agree with you, you should also consider what other competent judges would think if they were in your place”.

The authors, without much explanation, touch on the positive role that algorithms could play in judgments (a topic for another column). They also state that guidelines to constrain the discretion of judges can be of assistance (we already know that guidelines cannot be binding on decision-makers).

Think statistically, and take the outside view of the case

The outside view of a case is considering a case as a member of a reference class of similar cases, rather than as a unique problem.

Structure judgments into several independent tasks

This is designed to combat “excessive coherence” which causes people to distort or ignore information that does not fit a pre-existing or emerging story. “Overall accuracy suffers when impressions of distinct aspects of a case contaminate each other. For an analogy, think of what happens to the evidentiary value of a set of witnesses when they are allowed to communicate”.

Resist premature intuitions

“Intuition need not be banned, but it should be informed, disciplined, and delayed.”

Kahneman is sceptical about the ability of the judicial system to address noise. In his interview in the Guardian, he stated:

The judicial system, I think, is special in a way, because it’s some “wise” person who is deciding. You have a lot of noise in medicine, but in medicine, there is an objective criterion of truth.

…

…But I have had many conversations with judges about the possibility of doing research on how noise affects their judgment. But, you know, it’s not in the interest of the judicial community to investigate themselves.

I agree that it is hard for the administrative justice system to “investigate itself”, but it is not impossible. Investigating noise, much like investigating unconscious bias, is challenging because implied in that investigation is the idea that decision-makers have not been consistent. This is a hard pill to swallow for many and understandably so. However, as the authors have pointed out, noise is largely a structural issue and not an individual one.

The concept of “decision hygiene” is also helpful. As when we wash our hands to remove unidentified germs, practicing good decision hygiene does not require the identification of the noise to eliminate or reduce it.

This book serves as a useful first step for a tribunal in discussing noise in its decision making. In future columns I will highlight some of the tools available in the tribunal setting to make things a bit quieter.