The Problem With Experts
Litigation and arbitration are teeming with experts these days.
There are technical experts to explain what happened. Others to say whose fault it was. And another bunch to quantify the damages.
Almost every sizable case has at least one expert on the witness list. Well, never just one. Each side must have their own expert. And, of course, they never agree.
That’s the problem with experts. Recent studies have shown that people have a very hard time understanding what experts say and giving appropriate weight to conflicting expert opinions. Adjudicators are no different from anyone else.
Derek Koehler, a psychology professor at University of Waterloo, recently wrote in The New York Times about experiments he conducted to assess how scientific debates are reported in the news media.
Media try to give “balance” to their reporting by including the views of dissenting experts. Often the weight of the expert opinion is very strongly on one side of an issue – for example, climate change or the safety of vaccines or certain foods. But the very fact of including the dissenting opinion may give a false impression of disagreement when there is really almost unanimous consensus on a particular point.
In one study Koehler conducted, participants were given a numerical summary of a range of expert opinion on various economic issues. On some issues, a large majority of experts agreed on a conclusion; on others there was more disagreement. (For example, on one issue 93 experts agreed, 2 disagreed and 5 were uncertain; while on another the split was 38/36/27.)
The study found that, when participants were given a written comment from an expert on each side of the question, in addition to the raw numbers, they had much more difficulty distinguishing between the high-consensus and low-consensus opinions. The participants gave much more weight to the dissenting opinions than the raw numbers warranted.
“This distorting influence affected not only the participants’ perception of the degree of consensus, but also their judgments of whether there was sufficient consensus to use it to guide public policy,” Koehler concluded.
What causes this response? According to Koehler:
One possibility is that when we are presented with comments from experts on either side of an issue, we produce a mental representation of the disagreement that takes the form of one person on either side, which somehow contaminates our impression of the distribution of opinions in the larger population of experts. Another possibility is that we may just have difficulty discounting the weight of a plausible argument, even when we know it comes from an expert whose opinion is held by only a small fraction of his or her peers. It’s also possible that the mere presence of conflict (in the form of contradictory expert comments) triggers a general sense of uncertainty in our minds, which in turn colors our perceptions of the accuracy of current expert understanding of an issue.
This is not only a problem for media reporting on public policy issues such as how to deal with climate change, drug or food safety. It has profound implications for expert evidence in arbitration or litigation as well.
What is an adjudicator to think when faced with experts on either side who are well-qualified, articulate and credible in their opinions?
Piling on more experts doesn’t help: “I’ll see your PhD and raise you two…”
This simply adds to the cost for each of the parties and does nothing to resolve the problem.
Perhaps, in situations where there is a strong consensus of opinion on one side or the other, a party can present evidence on that consensus, but I think the situations where that would resolve a disputed issue are very rare.
Usually, the experts are being asked to apply their expertise to the facts of a particular case. So while there may be a consensus on the basic principles that apply, there is none on the final conclusion. There is just an opinion on either side – and each opinion depends on the assumptions, experience and analysis of that particular expert with respect to those specific facts.
Counsel may try to undermine the opposing expert by attacking their assumptions or the methodology they used to arrive at a conclusion. Maybe they can drag up some prior inconsistent research or statement made by the expert. Or perhaps a series of hypotheticals can be presented to the expert which cause him or her to hedge or qualify the opinion to some degree.
None of which really answers the question the adjudicator must face: Which expert is right?
So what does the decision-maker do?
Maybe some of the other evidence in the case is weak. Or perhaps there are credibility issues with other witnesses. So the decision-maker relies on that evidence to come to a conclusion on the critical issues. The expert evidence on one side or the other can then be thrown in to support the reasons for the decision that has been made.
If that’s what really happens, parties are spending an awful lot of money on experts for very little benefit.
But if the case truly turns on the experts, the adjudicator must be given better tools to weigh the conflicting expert evidence.
Counsel need to present the expert evidence in ways that allow adjudicators to better assess when an expert is basing an opinion on a set of facts or principles on which there is a strong consensus and when the expert is expressing a minority or dissenting opinion that is not widely supported.
Having said that, we must also keep in mind that there are cases where the contrarian opinion is the correct one. Many of the greatest scientists fought against the tide of accepted wisdom until their observations, experiments, arguments and theories were accepted and, in turn, became the consensus.
The psychological research shows that we are unduly influenced by a good story, even if it runs counter to the expert consensus on a particular issue. We have difficulty simply determining what that consensus is and what it means. We are also unduly influenced by experts who opinions agree with our own perception or world view.
Adjudicators must always be aware of this and guard against their own biases in assessing expert evidence.
Nowhere is the expert witness problem worse than it is in the Ontario auto accident injury (civil justice) context. The system suffers from a long-lamented proliferation of “hired gun” rogue, medico-legal experts who earn a handsome living selling “independent” opinion evidence to deep-pocketed auto insurers. Little has been done to transform the 2010 Rules od Civil Justice promise of impartial and qualified (medico-legal) experts from high-sounding rhetoric into the reality which vulnerable, injured litigants actually experience. Jaw-dropping partisanship on the part of expert witnesses, even when chronicled in (quasi)judicial decisions, is deliberately and stubbornly ignored by subsequent triers of fact who refuse to allow these experts to be cross examined on prior adverse judicial comments which go to bias or which may even go to a lack of competency/qualifications. The judges argue that just because an expert was found to be partisan in one case (or even in several previous cases); prior adverse judicial comments don’t constitute “proof” the expert won’t be impartial in the case at hand. In this way judges and arbitrators give long-time “hired gun” medico-legal experts a perpetual tabula rasa. In so doing they throw the dice and gamble the expert won’t proffer highly biased (pro-insurer) opinion evidence just as as she/he was found to have done by previous judges in previous cases. Sadly, if the gamble is lost, it is the injured litigant – not the trier of fact – who pays the price. Even when there is prior adverse comments chronicled in previous cases in which this or that expert has been found to have exaggerated their qualifications or to have conceded to a lack of competency and training in the very area in which she/he is again being called to proffer opinion evidence – judges and even the Ontario Civil Justice Rules Committee take the position that these prior adverse judicial comments aren’t relevant and so cannot be adduced so as to challenge the qualifications of impartiality of medico-legal expert witnesses. Who would ever imagine that an expert can concede, under cross-examination, to a lack of training and competency in a case and that subsequent triers of fact would then take the position that this information – chronicled in the form of adverse judicial commentary – isn’t admissible as a means to help screen out that same “expert’s unqualified opinion evidence in the next case? Judges fail in their gatekeeping of experts function when they refuse to consider the prior adverse comments of previous triers of fact. The whole notion of gate-keeping of expert testimony has been reduced to a hoax on the public. The Ontario Civil Justice system clearly learned nothing from the Goudge Inquiry. By refusing to consider the adverse comments of prior judges – thus giving bogus experts a free pass – the promise in the 2010 Ontario Rules of Justice (as they apply to experts) has become a misleading hoax on the public. Further, this mind-numbing refusal has set the stage for extended battles of experts which include partisan and even unqualified “experts”. This is a waste of time, money and court resources. And yet we hear the very same judges complaining about diminished resources and ridiculous wait times. Partisan and unqualified (medico-legal) experts undermine trust in the system, undermine justice, and drive up costs to the system. Put simply – they are an “access to justice” issue. An access to justice issue that has never been included in any the many “access to justice” columns appearing on the Slaw website. Why is that?
The problem is that the lawyers think that each side needs their own experts; the plaintiff doesn’t. An expert is not taking the time to know the plaintiff or the file and is producing a general and in some cases generic defence medical reports that can sometimes prejudice the plaintiff. You will sometimes see reports where they forgot to change the name in a cut and paste; or a female is called he throughout the report.
The plaintiffs should be relying on the treating regulated health professionals. Recent court decisions find the courts relying more and more on the treating health professionals.
The problem is also that the regulating body, the Financial Services Commission of Ontario (“FSCO”) who is responsible for the regulatory oversight of auto insurance in Ontario – is failing to meet its obligation to ensure that the insurance companies and their medical experts are complying with the regulations (see 2011 Ontario Auditor General’s Report on Auto Insurance (chpt. 3.01) and its 2013 Follow-up (chpt. 4.01)).
And the problem is no one is addressing the elephant in the room – the third party “broker” clinics who facilitate the process for the insurance companies and who according to various court and FSCO Tribunal decisions manipulate the process and the report and in some instances outright change what the expert actually reported (see FAIR (For Accident Insurance Reform) “The Independent Insurer Medical Examination IME/IE”). In one case I was involved with, the Insurance Company hired a broker to arrange a Paper Review Assessment and gave them 86 documents to review and consider and in turn Sibley gave the assessing Occupational Therapist only 8 documents for her Paper Review; and no one was the wiser until the insured filed an Access Request under PIPEDA. In another case the Occupational Therapist conducting a Determination of Catastrophic Impairment assessment under the SABS conducted it in the insured’s hotel room (no kitchenette) by interview only and the physician who prepared the Catastrophic Impairment Rating report told the CPSO that his role and report was “purely administrative, not interpretive.” .
The problem is FSCO doesn’t investigate complaints against the brokering facility and/or the expert regulated health professionals who edit and/or prepare the expert reports on behalf of the assessing or examining experts; and the College’s don’t regulate the Third Party Report and expert assessments conducted by their members (see K.I., MD and L.K. 2013 ON HPARB 332 (CanLII); and B.M. v. T.F. 2009 ON HPARB 89213 (CanLII) and are not meeting their obligation to protect the public. There problem is there is no governance of the Third Party expert reports and no consequences to the experts or the brokers or the insurance companies that participate in the process. If there was some consequence to the breaches of the various regulations covering the Third Party expert medical reports the problem could be addressed.