If the Mouse Roared, Then the Court Whimpered
Mark Twain wrote in Mark Twain, “Chapters from My Autobiography”, 598 North American Review (Sept. 7, 1906):
I wrote the rest of “The Innocents Abroad” in sixty days, and I could have added a fortnight’s labor with the pen and gotten along without the letters altogether. I was very young in those days, exceedingly young, marvellously young, younger than I am now, younger than I shall ever be again, by hundreds of years. I worked every night from eleven or twelve until broad day in the morning, and as I did two hundred thousand words in the sixty days, the average was more than three thousand words a day–nothing for Sir Walter Scott, nothing for Louis Stevenson, nothing for plenty of other people, but quite handsome for me. In 1897, when we were living in Tedworth Square, London, and I was writing the book called “Following the Equator” my average was eighteen hundred words a day; here in Florence (1904), my average seems to be fourteen hundred words per sitting of four or five hours.
I was deducing from the above that I have been slowing down steadily in these thirty-six years, but I perceive that my statistics have a defect: three thousand words in the spring of 1868 when I was working seven or eight or nine hours at a sitting has little or no advantage over the sitting of to-day, covering half the time and producing half the output. Figures often beguile me, particularly when I have the arranging of them myself; in which case the remark attributed to Disraeli would often apply with justice and force: “There are three kinds of lies: lies, damned lies, and statistics.”
Ian Binnie wrote, extra-judicially, while still a judge of the Supreme Court of Canada, in The Honourable Mr. Justice Ian Binnie, “Science in the Courtroom: The Mouse that Roared” (2007) 56 U.N.B.L.J. 307 at 326-27:
Finally, the parties to these complex cases have a legitimate expectation that the reasons for judgment (where the case is tried without a jury) will make it clear the judge has heard and understood the technical aspects of the case. While in Canada, jurors cannot be asked about their deliberations (therefore no “wah wah wah”), a case heard by a judge alone should result in a decision that traces the judge’s pathway through the scientific evidence to the result. In this way, the trial judge establishes a better record for review by the appeal court, and provides assurance to the parties as well as the broader public that the issues have been properly addressed. Appeal courts should be prepared to consider rejection of boilerplate reasons from a trial judge that fail to come to grips with the scientific debate under-lying the issues being litigated.
In Goodman v. Viljoen, 2012 ONCA 896 aff’g 2011 ONSC 821 the Ontario Court of Appeal affirmed, by a 2-1 majority, a trial judgment in which the trial judge’s finding of factual causation on the balance of probability was explicitly based on her acceptance of the plaintiffs’ experts’ theory of factual causation. That theory was based on a method of analyzing data known as the Bayes Theorem, which is used to produce conditional statistical probabilities. The majority of the Court of Appeal affirmed the decision on the basis that there was nothing about the trial decision that allowed the court to reverse or vary the trial decision.
[142] The standard of review in this case is to defer to the evidentiary findings of the trial judge unless her findings have no basis in the evidence. That is not the case here. In my view, the appellants are asking this court to re-examine all the evidence and to come to different conclusions than those reached by the trial judge – i.e., to retry the case. That is not the role of an appellate court. See Stein v. Kathy K (The), [1976] 2 S.C.R. 802; Housen v. Nikolaisen, 2002 SCC 33, [2002] 2 S.C.R. 235, at paras. 15-18; and H.L. v. Canada (Attorney General), 2005 SCC 25, [2005] 1 S.C.R. 401.
[143] This was a complex medical malpractice case. The causation issue was particularly difficult because, as is not uncommon, there were no specific scientific studies on which the experts or the parties could rely for definitive answers to the specific issues before the court. The administration of steroids prenatally in the case of premature births has had such a significant positive effect, including on the incidence of CP from certain causes, that the accepted standard of care is to administer the steroids in all such cases. However, that does not mean that the failure to do so in any particular case caused the CP in that case.
[144] The trial judge well understood both the deficiencies in the available experimental evidence and the analysis she had to conduct to be satisfied of causation to the required standard.
[145] In my view, the trial judge made no error of law or palpable and overriding error in the apprehension of the evidence, including the expert opinion evidence, or in her findings of fact based on the robust and pragmatic approach.
The trial judge provided some explanation of why she accepted that analysis, summarized in this paragraph:
[203] Dr. Barrett testified to, and I accept his explanation, as to why the Bayesian analysis is appropriate in these circumstances.
The majority must have been satisfied that the judge’s analysis was both consistent with the evidence and legally valid. However, the majority did not discuss the meaning of Bayesian probability analysis theory. You’ll find explanations here, here, and for the mathematically able and not-statistically challenged, here.
The Stanford Encyclopedia of Philosophy has an explanation which does not require a mathematical background.
Bayes’ Theorem is a simple mathematical formula used for calculating conditional probabilities. It figures prominently in subjectivist or Bayesian approaches to epistemology, statistics, and inductive logic. Subjectivists, who maintain that rational belief is governed by the laws of probability, lean heavily on conditional probabilities in their theories of evidence and their models of empirical learning. Bayes’ Theorem is central to these enterprises both because it simplifies the calculation of conditional probabilities and because it clarifies significant features of subjectivist position. Indeed, the Theorem’s central insight — that a hypothesis is confirmed by any body of data that its truth renders probable — is the cornerstone of all subjectivist methodology.
…
The probability of a hypothesis H conditional on a given body of data E is the ratio of the unconditional probability of the conjunction of the hypothesis with the data to the unconditional probability of the data alone.
In Goodman, the “hypothesis H” of the plaintiffs’ experts was that the defendant’s negligence was a necessary part of the cause of the infants’ cerebral palsy.
There is a bit of an O. Henry twist to the trial judge’s acceptance of the plaintiffs’ theory. The Bayesian analysis did not produce a probability greater than 50% that the negligence was a necessary part of the cause. The probability was only 40%. However, the plaintiff’s experts (Drs. Barrett and Pearlman) used that figure, in addition to their clinical knowledge, as the basis for their conclusion that the defendant’s conduct (which was held to be negligent) was a necessary part of the cause. The trial judge wrote:
[193] Having accepted that the Cochrane data establishes a 40% reduction in CP, the court must address the defendant’s submission that this reduction does not meet the necessary legal burden of proof of more likely than not. Instead, this 40% magnitude of reduction results only in a loss of chance which is not compensable in medical malpractice cases (See: Cottrelle et al v. Gerrard (2003), O.J. No. 4194, (C.A.).
[194] Perhaps if the court were to look at this figure in isolation, this argument may have some merit. However, to do so would be to hold the plaintiffs to a standard of scientific certainty which is not required by law and to ignore the relevant context.
[195] The statistical information is but one piece of the puzzle …
[196] Since the figure of 40% represents an all or nothing proposal, it does not accurately reflect the total impact of ACS on CP. Based on this evidence, it is a logical conclusion that if one were able to measure the total effect of ACS on CP, the statistical measure of that effect would be inflated beyond 40%.
[197] Unfortunately, this common sense conclusion has never and can never be tested by science. As Dr. Perlman testified, such a study would be impossible to conduct because it would require pre-identification of those persons who go on to develop CP. Furthermore, because the short term benefits of ACS are now widely accepted, it would be unethical to withhold steroids to conduct further studies on long term outcomes.
[198] I agree with the plaintiffs’ submission that they cannot be denied recovery because science has not advanced far enough to answer the question to a degree of scientific precision, particularly in the face of a very persuasive trend. …
[199] Although science can never prove this logical inference, it does find support when the Cochrane data is viewed using a Bayesian approach to statistical analysis. The court heard from two experts qualified as bio-statisticians as to the appropriate statistical approach to this data.
[200] Both Dr. Willan and Dr. Platt presented their evidence in a fair and balanced manner. Again, the court is faced with two equally competent experts disagreeing. The real issue of contention between the two gentlemen is the statistical approach used by each. The actual mathematical calculations by either were not at issue. For the reasons that follow, in the end, I prefer the evidence of Dr. Willan.
…
[202] The Bayesian approach used by Dr. Willan was heartily endorsed by Dr. Barrett. For reasons already mentioned, I found Dr. Barrett to be exceedingly knowledgeable and particularly well placed to give an opinion on the medical issues relevant to this trial and I preferred his evidence to that of the other medical experts. His endorsement of Dr. Willan’s analysis therefore carries some weight.
[203] Dr. Barrett testified to, and I accept his explanation, as to why the Bayesian analysis is appropriate in these circumstances.
…
[206] This statistical information is of course just one piece of evidence the court must consider in determining the issue of causation. Statistics represent generalizations and not the particular circumstances of a given case. At best, they are numbers which are dependent on the underlying data and the limits of scientific knowledge. Context, human experience and interpretation are required if statistics are to have any real meaning.
[207] Both Drs. Barrett and Perlman, exercising their clinical judgment and expertise, considering the statistical evidence, answered unequivocally that it is more likely than not that had the twins received a full course of steroids, they would not have suffered the injuries they did, or the magnitude of those injuries would have been reduced. I agree with this conclusion. It is supported by the statistical and medical evidence and the common sense inferences that flow therefrom. I therefore find that the plaintiffs have established on a balance of probabilities that the defendant’s negligence caused the CP from which they now suffer.
I have provided a link to the ONCA decision in Cottrelle. Tea leaf readers might want to look at who two of the three members of the ONCA panel were and consider who they are now, if pondering the likelihood that leave to appeal will be granted by the SCC (assuming leave is sought).
The majority reasons contain only one explicit reference to the trial judge’s acceptance of a Bayesian analysis of data.
[130] In my view, the trial judge made no error by accepting Dr. Perlman’s evidence in her consideration of this issue. Dr. Perlman gave his opinion in this case based not only on the Cochrane analysis of 48 cases, but also on his knowledge of the earlier 2004 study and analysis, his knowledge of animal studies showing a relationship between ACS and CP, his expertise and experience, as well as the biological plausibility approach and the Bayesian probability analysis by Dr. Willan. The biological plausibility approach in particular addresses the likelihood that steroids have a maturational effect on all membranes including the brain membranes whose disruption leads to PVL.
There are more references to the Bayes Theory in the dissenting reasons; however, the dissent does not discuss the acceptability, in law, of Bayesian analyses. It may well be that the dissenting judge was prepared to accept it could be valid for the purposes of the dissent. The substance of the dissent was that the plaintiff’s evidence did not connect increased risk to probable causation. The facts of the case are outlined in the dissent. The majority accepted the accuracy of that outline: see para. 110.
[13] The twins had developed CP by the time they were 18 months old. CP is a non-progressive nerve disorder and describes a cluster of conditions that includes motor difficulties, such as spasticity of the limbs, and sometimes includes cognitive problems. Daniel, the first born, is physically disabled but does very well in school. Calvin has more significant medical problems than Daniel. He also has some behavioural problems.
[14] CP develops in the months or even years following birth. CP is sometimes associated with medical problems that occurred during or shortly after birth. For example, some CP is associated with respiratory distress syndrome (“RDS”) experienced by some premature babies during the birth process. RDS can have a direct connection to the subsequent development of CP, or it can lead to a variety of other medical problems that in turn are associated with the later development of CP. For example, some kinds of intraventricular hemorrhaging (“IVH”) are associated with RDS and the later development of CP in premature babies.
[15] In many cases, especially in full-term pregnancies, CP cannot be attributed to any specific event or medical cause. The risk of CP is known to be much higher among premature infants than full-term babies. Premature twins are more susceptible to CP than are single premature babies.
[16] The experts all agreed that the twins’ CP was caused by a condition known as diffuse periventricular leukomalacia (“PVL”). PVL is among the many problems that premature babies face in making the adjustment to life outside of the womb.
[17] PVL involves the inadequate blood supply to an area of the premature baby’s brain referred to as the watershed zone. Arterial blood supplies to the brain meet in the watershed zone. In premature infants, the arterial membranes may not develop fully. The brain cells affected by PVL are unstable and vulnerable. The expert evidence suggested that the damage to the affected areas of the brain caused by PVL occurs during delivery or in the first few days following birth.
[18] Babies born before 34 weeks gestation are more prone to PVL than other babies. PVL is the most common cause of brain injuries suffered by premature babies.
[19] PVL is associated with the later onset of CP. The descending nerve tracts to the legs and arms pass through the area of the brain adversely affected by PVL.
[20] PVL may be caused by hypoxia, that is reduced oxygen in the blood flow circulating to the affected area of the brain, or ischemia, that is a deficiency in the blood supply to the affected area of the brain due to reduced blood pressure. Either hypoxia or ischemia results in the death of brain cells in the affected area of the brain. The twins’ PVL was likely caused by ischemia.
…
[98] In my view, the absence of any evidence capable of quantifying the reduced risk of PVL flowing from a course of ACS precluded a finding that the failure to administer ACS caused the twins’ PVL which in turn caused their CP. That was the primary basis on which the trial judge found causation.
[99] The trial judge did address the quantification of the reduction of the risk of developing CP in the context of her alternative finding of causation based on her determination that the data in the Cochrane analysis established that the risk of CP would be reduced regardless of the specific medical cause of the CP. That analysis cannot remedy the trial judge’s failure to address the quantification of the risk in the context of her finding that the failure to administer a course of ACS caused the twins’ PVL. Nor can it salvage her finding that the CP was caused by the failure to administer ACS regardless of its immediate medical cause. I have already explained, at paras. 63-68, why in my view that finding is unsupported by the evidence. For the sake of completeness, however, I will address this aspect of her reasons.
[100] The trial judge accepted that the Cochrane analysis data established that a course of ACS would reduce the risk of CP in premature babies by 40 per cent. The trial judge also recognized that a 40 per cent reduction in the risk would not establish legal causation. She then went on to hold that the Cochrane analysis data did not identify those cases in which babies who had developed CP, even though their mothers had received a course of ACS, had developed significantly less severe disabilities than those babies who had developed CP and whose mothers had not received a course of ACS. In the trial judge’s assessment, this category of babies who were less seriously handicapped was sufficiently large to inflate the effect of the impact of ACS on CP beyond the 40 per cent reduction in CP. According to the trial judge, many babies who developed CP would be less seriously handicapped because of the administration of ACS to the mother prenatally. The cumulative effect of the two categories, that is babies who did not suffer CP and babies whose disabilities were significantly lessened, was sufficiently large to reach the required legal threshold of a balance of probabilities.
[101] The Cochrane analysis in fact provided no evidence that a course of ACS had any effect on the severity of the symptoms of those babies who developed CP. The Cochrane analysis did not describe the level of disability of any of the 48 babies referred to in the studies it considered. Clearly, the Cochrane analysis made no attempt to compare the level of disability among the 28 babies whose mothers had not received ACS with the level of disabilities among the 20 babies who had developed CP despite the fact that their mothers had received a course of ACS.
[102] Although the Cochrane analysis data itself could offer no support for the assertion that a course of ACS would reduce the severity of the disabilities suffered by babies who developed CP, the plaintiffs’ experts, applying I think the “biological plausibility” explanation, had testified that it only made common sense that if a course of ACS eliminated CP in 40 per cent of the cases, it would reduce the severity of the disabilities flowing from CP in some unspecified number of other cases. The plaintiffs’ experts acknowledged that the reduction in the severity of symptoms associated with CP in babies who had received a course of ACS had never been studied much less reported in the scientific literature.
[103] Although the evidence to support the reduced severity contention is far from strong, I think it was open to the trial judge to accept that evidence. In my view, however, it did not get the plaintiffs over their causation difficulties.
[104] The trial judge began her causation analysis by accepting that the Cochrane analysis data showed a 40 per cent reduction in the occurrence of CP when the mothers had received a course of ACS. She then placed on top of that 40 per cent an additional unquantified number of cases in which the severity of the disabilities suffered by the babies had been materially reduced. The two combined met the balance of probabilities standard.
[105] For reasons set out above, the Cochrane analysis data cannot reasonably support a finding that a course of ACS would reduce by 40 per cent the risk that a premature baby would develop CP regardless of the underlying medical cause of the CP. The data goes no further than to suggest that a course of ACS may reduce the risk of CP in cases other than those in which a connection has been shown between the underlying medical cause of CP and a course of ACS. The quantification of that reduction in premature babies at 40 per cent, the trial judge’s starting point in her causation calculus, has no basis in the evidence.
[106] Putting the causation case at its best for the plaintiffs, there was evidence suggesting that a course of ACS would reduce the risk that premature babies would develop CP in an unspecified number of cases. There was also evidence that in an unspecified number of cases, a course of ACS may have a positive effect on the seriousness of the disabilities suffered by babies who did develop CP. Placing these two unquantifiable amounts together could not demonstrate the requisite causal connection on the balance of probabilities. Pushing the evidence to the edge of its reasonableness limits does not provide a basis upon which the trial judge could reasonably find that it was more likely than not that the twins would not have suffered CP or that their disabilities associated with CP would have been significantly reduced had their mother received a full course of ACS prenatally.
One would never know it from either level of decision in Goodman, or indeed from the content of any other reported Canadian decision, but most courts in the U.K, Australia, and the U.S. have held that Bayes Theorem – regardless of its validity in the world of science – is not appropriate in the court room. The issue usually arises (when it does) in criminal law cases. (Simon Fodden has reminded me that he mentioned one of the U.K. cases, recently, here, and wrote a piece on Bayes Theorem even earlier. He provided the link.)
The issue is not whether Bayes Theorem is junk science. It is not. The trial judge briefly dealt with this issue, without referring to case law.
[125] Dr. Willan described different statistical approaches and in particular, the frequentist or classical approach and the Bayesian approach which differ in their respective definitions of probability. Simply, the classical approach allows you to test the hypothesis that there is no difference between the treatment and a placebo. Assuming that there is no difference, [that] allows one to make statements about the probability that the results are not due to chance alone. To reach statistical significance, a standard of 95% is required. A new treatment will not be adopted into practice unless there is less than a 5% chance that the results are due to chance alone (rather than due to true treatment effect).
[126] Dr. Willan explained the results of the Cochrane meta-analysis in frequentist terms using the CP analysis as an example (Exhibit 10, page 341). The risk ratio (RR) is calculated by taking the probability of CP in the treatment group divided by the probability of CP in the placebo group. A risk ratio that is equal to one means that there is no treatment effect at all – in other words, the placebo and the treatment have the same effect.
[127] In the case of CP, the RR is 0.60 or a 40% risk reduction in CP where there has been administration of antenatal corticosteroids. The upper limit of the confidence level just crosses one at 1.03, which means that the analysis just fails to reach the level of statistical significance in order to justify a treatment policy change. …
[128] [Dr Willan] testified that it is now impossible to undertake further clinical trials because ethically, such studies would never be approved as a standard of care is already in practice for the use of steroids. Because no further trials are possible, there is strong evidence that the use of steroids is beneficial on a long-term basis. The short-term data already confirms this. In order to determine the probability that the risk of CP is reduced, one must use the Bayesian method which uses a different definition of probability. It is an expression of the degree of belief about the unknown.
[129] Dr. Willan acknowledged that in clinical trials the standard is to use the classical or frequentist approach. However, the Bayesian approach is a highly acceptable, reliable, statistical method. His work using the Bayesian approach has been peer reviewed. In his opinion, the Bayesian approach is appropriate in the circumstances of this case, to make a probability statement about whether the risk of developing CP is reduced with the use of steroids.
…
[135] In cross-examination, Dr. Willan acknowledged that the classical approach is the standard used for the purpose of treatment. He further acknowledged that there have been no studies involving the impact of steroids using the Bayesian approach.
…
[138] Dr. Willan agreed that the best estimate of the magnitude of risk reduction is 40% in the case of CP using the frequentist approach. The doctor noted that it would be very rare in clinical practice and research to find a new treatment that would ever half the risk of an outcome. It has only happened to him once in his lengthy career.
[139] Dr. Willan did not calculate the magnitude of the reduction in risk using the Bayesian method. However, he agreed that the 70% increase in risk represents the same 40% risk reduction.
[140] Dr. Willan further acknowledged that he did not review the studies underlying the Cochrane analysis and could offer no assistance on the quality of these papers. He did not apply his statistical analysis to the clinical factors, but agreed that such an exercise should be undertaken by the court to consider a particular case.
…
[146] When questioned about Dr. Willan’s Bayesian analysis and his theory that ACS probably reduces the risk of CP, Professor Platt opined that the quality of the underlying data does not support the plaintiffs’ conclusions. However, Dr. Platt did acknowledge that the Bayesian approach is a well-known and perfectly valid statistical method.
[147] Dr. Platt also rejected Dr. Barrett’s assertion that an increase in sample size would achieve statistical significance
A search for any reported mention of the Bayes Theorem in Canadian jurisprudence (on CanLII and Westlaw Canada’s Canadian court decision databases) produces only one hit: Goodman. Extending the search to Westlaw Canada’s articles database produces 18 more hits, most of which are articles which discuss, in various ways, the asserted conflict (or at least tension) between the assumptions that underlie the Bayesian analysis and those which underlie judicial fact finding, whether by judge or by jury.
Professor Russell Brown, of the University of Alberta Law School, has succinctly explained why in one of the articles available through the Westlaw Canada search: Brown, “The Possibility of ‘Inference Causation’: Inferring Cause-In-Fact And The Nature Of Legal Fact-Finding” (2010) 55 McGill L. J. 1 at 27ff [Brown, “Inference Causation”]. Brown provides, at 27-28, a useful general explanation of how the Bayesian analysis is performed.
In essence, Bayes’s Theory furnishes a mechanism for incrementally revising probability estimates in light of new information, thereby allowing a fact-finder to update continually an opinion about the relative likelihood of a fact. This requires two pieces of data, the first being an a priori estimate of the probability that a fact is proven or not. Bayesians posit that a fact-finder begins with an original estimate of the likelihood of a fact, such as a causal link. That prior probability is then continually revised to reflect the statistical impact of new, relevant information as it is received and incorporated into the probabilistic calculation of the likelihood of the fact. In Bayesian terms, the statistical impact of new information constitutes the second piece of data, being a likelihood ratio, which is multiplied by the prior probability to create the posterior possibility. The process is repeated until finally, the fact-finder supposedly arrives at the final modification of the probability of the causal linkage.
Brown then explains at 28-30 why the premises of the Bayesian analysis are “incompatible with legal fact-finding”:
Bayesian methodology suffers from several defects, however, making it incompatible with legal fact-finding. The principal objection challenges prior probability: it is in essence a reference class, since it forms the a priori basis from which the probability of a proposition such as a causal link is made. Everything that follows —the continual updates of the prior estimate in light of discovered bits of evidence relevant to cause-in-fact, and the final modification of the assessment of the proposition’s likelihood — depends upon that starting point. Unless, however, that preexisting probability is truly a priori — that is, an analytic proposition that derives from logic and reason, as opposed to a synthetic proposition based on experience and observations — there is no reason to accept that pre-existing probability’s status within Bayesian methodology is inherently reliable. … Given the scientific fact-finder’s claim to formal objectivity, the Bayesian process is ironically totally dependent upon the intuition that probabilists eschew as fuzzy and unreliable, and which statistical evidence ostensibly allows fact-finders to avoid.
Another objection to statistical evidence and to its processing in litigation in a Bayesian manner derives from the empirical limitations of human computational capacity. The notion of a fact-finder who continually and mathematically refines accumulating data is simply implausible. The nature of evidence itself, moreover, defies easy Bayesian statistical reduction. Take again the problem of soft variables in evidence. Bayesian analysis cannot account for the significance to be ascribed to the meaning of a witness’s smile in describing a possible causal sequence because both the factual question of whether it was more a smirk than a smile, and the contrasting hypotheses (derision or humour) pose difficulties and perhaps impossibilities for statistical reduction. …
[At 28-29, internal footnotes omitted.]
Brown’s article provides references to the literature on the issue of the compatibility or incompatibility of Bayesian analysis and evidence theory. There are proponents of compatibility. The literature outside of Canada, particularly in the U.S., is extensive. There’s useful Australian literature too that’s easily found by a search on AUSTLII, even a very recent piece arguing that Bayesian analyses are consistent with the premises of legal fact-finding: James Franklin, “The Objective Bayersian Conceptualisation of Proof and Reference Class Problems” (2011) 33 Sydney Law Review 545. (The link will get you to the site to download the paper.) This is the abstract.
The objective Bayesian view of proof (or logical probability, or evidential support) is explained and defended: that the relation of evidence to hypothesis (in legal trials, science etc) is a strictly logical one, comparable to deductive logic. This view is distinguished from the thesis, which had some popularity in law in the 1980s, that legal evidence ought to be evaluated using numerical probabilities and formulas. While numbers are not always useful, a central role is played in uncertain reasoning by the ‘proportional syllogism’, or argument from frequencies, such as ‘nearly all aeroplane flights arrive safely, so my flight is very likely to arrive safely’. Such arguments raise the ‘problem of the reference class’, arising from the fact that an individual case may be a member of many different classes in which frequencies differ. For example, if 15 per cent of swans are black and 60 per cent of fauna in the zoo is black, what should I think about the likelihood of a swan in the zoo being black? The nature of the problem is explained, and legal cases where it arises are given. It is explained how recent work in data mining on the relevance of features for prediction provides a solution to the reference class problem.
As mentioned, in Goodman, neither the trial judge, nor the majority, nor the dissent, discussed whether the Bayesian methodology was legally valid, regardless of its statistical validity. The majority, though, has to have accepted that the Bayesian analysis was legally valid as well as scientifically valid in order to validly affirm the trial verdict. I leave it to others to decide whether Goodman deals, adequately, with the former issue, particularly if we take the comments of then Justice Binnie as expressing what the views of the Supreme Court of Canada would be if asked to address the question.
I should clarify a point that I (perhaps) didn’t make clear enough in the posting. I attempted to do so in the last paragraph.
I am not dealing with the issue of whether the trier of fact (judge or jury) should employ some form of Bayesian analysis in deciding whether it is probable or not that some thing occurred in the past. That is not what happened in Goodman. The trial judge did not purport to apply, herself, some form of Bayesian analysis of the data.
Rather, she accepted an expert opinion on the probability of some fact where the opinion was based on a Bayesian analysis of the data. The issue I have raised – which is why I led with the quotations from Twain and Binnie – is whether the trial reasons, and appellate reasons, adequately explain why an opinion based on a Bayesian analysis of the data was admissible at all. The majority reasons stand or fall with the trial reasons, because the majority didn’t provide an independent explanation.
I’m going to repeat a portion of the passage I quoted from ex-SCC Justice Binnie’s article:
It is true that the trial judge explained, in some detail, what it was the plaintiffs’ experts said the Bayesian analysis of the data (the statistics) meant and how the result compared to traditional methods of statistical analysis. However, I saw nothing in the trial reasons when I first read them, and still see nothing – for whatever that is worth in the scheme of things – that convinces me that the trial judge understood the principles of Bayesian methodology enough to know why there is tension, if not in conflict, between that methodology and the methodology of fact finding in law.
Bayes Theory is not “junk science”. The trial judge adverted to that issue when she mentioned that the work of the expert who provided the Bayesian analysis of the data had been been peer reviewed. But the fact that it is not junk science – the fact that it is an accepted methodology in science – does not, of itself, mean that there is no conflict between Bayesian methodology and the principles that law uses in determining what is admissible evidence and the weight to be given that evidence. Professor Brown highlighted that problem in the excerpt that I quoted. I’ll repeat the key passage.
I don’t see anything in the trial judge’s reasons that satisfies me that the “prior probability” – the pre-existing probability – upon which the plaintiff’s theory of “biological plausibility” relied was inherently reliable. That is because there is no discussion of that issue, at all, in the reasons. (Perhaps it is hinted at when the judge explained why additional testing was no longer permitted, on ethical grounds.) Perhaps that issue was canvassed in the evidence. But we don’t know that. The majority appellate reasons don’t indicate whether it was.
The trial judge may accurately set out what it was the experts said, but that does not mean the trial judge understood that evidence. At the end of the day, then, without something in the trial reasons that shows that the trial judge understood the principles upon which Bayesian methodology is based, the trial reasons are inadequate. If the trial reasons are, then so are the majority reasons in the ONCA. We are not dealing with a principle of law which allows us to apply the presumption of regularity to the trial or majority reasons.
It is open to a trial judge to reject or accept admissible expert evidence on valid grounds. But what we don’t see, I suggest, in the Goodman trial reasons is why the Bayesian analysis was legally valid in the circumstances [just] because the trial judge accepted that it was medically valid, preferring the evidence of the plaintiffs’ experts over that of the defendant’s experts. Medical validity – here, statistical validity – doesn’t entail legal validity. What we see, I think, is the judge presuming the Bayesian analysis was legally valid as a matter of evidence theory. Perhaps that is because of the way the case was presented. I don’t know. Make what you will of this paragraph in the trial reasons:
I have added the emphasis. That paragraph could be read to mean that the defence did not challenge the basic admissibility of the opinion – did not raise the issue of whether a Bayesian analyisis is compatible with law’s principles of evidence – but only its weight.
I’m going to quote, again, from the trial reasons where, I think, we see where the judge made the jump from the argument about medical validity to the assumption of legal validity. That will put para. 200 in context.
I have added the emphasis
DC
I will finish by adding my standard complaint about judicial claims that “common sense” somehow provides proof and justification for the claim that the evidence shows a valid in law causal connection between wrong and injury. (This apart from the fact that the judge used the phrase because the Supreme Court has said that common sense is the underlying principle by which the but-for test is to be applied, hence the judge needs to at least mention that mantra.)
Let’s look at how the trial judge referred to “common sense” other than when quoting or paraphrasing case law. That occurs in para. 12.
Bear in mind that what we’re considering is the significance of evidence whose relevance has been explained to the fact-finder (here a judge alone) by an expert, who was permitted to testify and whose opinion is relevant only because the significance of that evidence is not something a judge (or jury) is assumed to understand without that opinion. So, when a judge asserts that common sense (whatever the judge means by common sense – the term is not explained: nor was it in Goodman) supports the expert’s opinion of a causal connection, aren’t we left to wonder how the judge would know that? If, however, the expert says it’s common sense, then the expert means common sense based on the expert’s knowledge, which has to be the proverbial horse of a different colour. (But not a zebra.)
I’ve added the emphasis. Remember the point that Professor Brown made about the validity of the Bayesian method depending on the assumption of the validity of the prior probability. What does the sentence in bold tell us?
All emphasis added by me.
What does the trial judge’s use of “common sense” add in any of these paragraphs to the question of whether the evidence established a causal connection, on the balance of probability, between the negligence and the injury?. If its only the judge’s adjective describing her view of the validity of the expert’s opinion, it adds nothing. If it is something more, what is that something more? Bear in mind that, when the trial judge quoted case law, one of the passages she quoted (at para. 12) was from an Ontario Court of Appeal decision – Fisher v. Victoria Hospital 2008 ONCA 759 at para. 59 where the ONCA reiterated that “common sense cannot become a substitute for resort to the evidence. … The mere application of “common sense” cannot conjure up a proper basis for inferring that an injury must have been caused in one way rather than another.” [Internal quotation marks omitted.]
So, let’s go back to the lines in paras. 207 and 210 that seem to summarize the trial judge’s use of “common sense”.
We are entitled to ask, about para. 207, what “common sense inferences” if these are anything other than what the expert(s) said. If they are only common sense conclusions by the experts then “common sense” adds nothing, not the least because these conclusions aren’t common sense.
This seems to be the judge’s common sense. If it is, then is it not using “common sense” as a substitute for evidence? Is it not using “common sense” to “conjure up” a basis for finding a causal connection?
After all, the plaintiffs had to resort to the non-traditional Bayesian analysis precisely because the traditional methods of statistical analysis did not produce the answer that the plaintiffs needed to succeed if the but-for test was applied.
Why did the plaintiffs have to use the but-for test? Because the trial judge ruled that the effect of Resurfice Corp v. Hanke 2007 SCC 7 was that the Athey material contribution test (whatever it now meant as a result of Resurfice: my interpolation – DC) was not applicable. Presciently, the trial judge, after quoting the usual paragraphs from Athey, wrote, anticipating the Clements two-tortfeasor requirement.
As I have written, elsewhere, it defies reality – some might say common sense – to suggest that the trial judge would NOT have applied Athey material contribution to injury as the test before Resurfice.
I’ll end with another oddity in the trial reasons. Recall that the posting which starts this thread had a quotation from the discussion in Prof. Russell Brown’s “Inference Causation” of the incompatibility between Bayesian methodology and legal fact finding. The trial judge in Goodman cited Professor Brown’s article – but only on the issue of the scope of the material contribution test after Resurfice.
The reference to Clements is to the British Columbia Court of Appeal decision.
For those who care, what I expect will be my last – for a very long time – significant (well … I think it is significant: opinions may vary – I thought “Snark” was significant, too) contribution to the literature on the meaning of factual causation in negligence, in Canadian law, is now available on SSRN. Its partial title is “Black Holes, Aether, and Negligence in the Air”. I uploaded a revised version on December 25, 2012. Wait until that appears. The “Aether” is not a misspelling, a typo, or a conceit.
As some of you know, the Supreme Court of Canada has yet another case with causation issues under reserve: Ediger v. Johnston, docket no. 34408, appeal argued Dec 4, 2012. There is only one tortfeasor in Ediger so I don’t expect the Court to revisit material contribution principle. While circumstances might change, what the Court might do with (or to) but-for principle I plan to leave to others.
I still wonder why this was not argued as a Snell case- does the impossibility of proving causation from either side take it out of that paradigm?
Brian:
I assume what you mean is “why wasn’t the case argued under Snell without reference to the Bayes Theorem?” since the trial judge applied the robust & pragmatic approach – see paras. 70-77, 121-122 of the ONCA reasons.
Reading between the lines of the trial reasons, I suspect the Goodmans’ lawyer made that argument too – that Snell supported the inference even without the use of a Bayesian analysis of the statistics, but the trial judge didn’t mention it. if you look at the trial reasons, you’ll see both Drs. Barrett and Perlman testified for the plaintiffs on causation. The trial judge doesn’t mention Bayes in setting out Perlman’s opinion, summarized in paras. 68-69, 76
Bayes first appears in the context of Dr. Barrett’s evidence.
What seems clear enough from the trial reasons, as written, is that the trial judge wasn’t prepared to hold that the evidence was enough, in total, even applying the robust & pragmatic approach, without the interpretation of the statistical evidence provided by the Bayesian analysis.
My reading of the ONCA reasons is that the Goodmans’ case was argued in the alternative there, too. The ONCA majority reasons, however, make it seem that the Bayes analysis was also part of the basis of Dr. Perlman’s evidence (see paras. 130-131.)
David
Brian:
Just in case I misread your comment:
If it was impossible to prove or disprove factual causation on the balance of probability, using the but-for test – since after Clements that seemingly all we have – then the action should have been dismissed. There’d have been a gap in the evidence from possibility to probability but no way of validly leaping it, and no alternative method of satisfiying the causation requirement since Clements says explicity that the material contribution test requires two tortfeasors pointing their fingers – you choose the digit- at each other.
(Actually, if you choose the right digit they’d be giving the finger to the plaintiff, but that’s a different image.)
David
There are a number of points that could be made in response to this comment. I will restrict myself to the main points. There is substantial literature on this issue and it is disappointing that the author chooses to quote only one misconceived article.
1. The author refers inconsistently to Bayes Theorem and Bayes Theory. It is a theorem, it is logically true.
2. It is unclear how legal and scientific decision making can differ. The question is what inferences can be drawn from evidence in conditions of uncertainty. Whatever method is adopted must comply with the requirements of logic. These do not vary according to context.
3. No one piece of evidence can prove a proposition, it can only make the proposition more or less probable than it would be without the evidence. The measure of how much it does so is determined by the probability that the evidence would be found if the proposition is true divided by the probability that the evidence would be found if some other proposition is true. This is called the Likelihood Ratio and is one component of Bayes Theorem.
4. I cannot find a statutory definition of “relevance” in Canada but in R. v. Cloutier 1979 CanLII 25 (SCC), (1979), 48 C.C.C. (2d) 1 (S.C.C.) at 27 it is said that relevance “requires a determination whether, as a matter of human experience and logic, the existence of a particular fact, directly or indirectly, makes the existence or non-existence of a material fact more probable than it would be otherwise”. This corresponds with point 3 above.
5. The point of the Bayesian approach to evidence is precisely that the expert does NOT assume a prior probability or a posterior probability, those are matters for the tribunal of fact. The witness should state only the Likelihood Ratio, the amount by which the evidence should logically alter the assessment of probability of the proposition.
6. One example of past bad practice was the “probability of paternity”. Witnesses used in many jurisdictions, and still do in some, to give this probability. This involved multiplying the “paternity index” which is a Likelihood Ratio, by a prior probability arbitrarily chosen by the witnesses of 50:50. This form of evidence involved assuming a prior probability and assuming an alternative proposition. The latter should depend on the arguments in the case and in particular on the defence theory. Thus the pre-Bayesian methods of evidence were logically invalid and usurped the role of the tribunal of fact, but courts accepted them and lawyers and judges express great nostalgia for them.
7. Pre-Bayesian methods also led to strange judgments in which judges said things such as that legal and mathematical probability were different, as they could not work out how to combine the evidence with the other evidence in the case. There is no logical way of doing so.
8. The judgments in the UK and elsewhere which have criticised Bayesian reasoning have been subjected to vigorous criticism in the literature. They are riddled with basic errors and misunderstandings. This is not surprising as they consist of judges issuing their more or less untutored views. So far as I know, there has never been an appellate case in which counsel gave prepared argument on the subject.
9. If Bayesian reasoning is to be rejected, it is incumbent on the critic to suggest some other coherent system which presents logical criteria for analysing decisions. The author does not attempt to do so.
Mr. Robertson,
1. You are right. The use of Bayes “theory” was sloppiness on my part.
2. With respect, if you are who your name and comment suggests you are, you know exactly how legal and scientific decision making do differ, particularly if you were referring to the truth of the decision as opposed to the process of decision-making. And, there is a difference in the latter, too. Ultimately, since in law “ought” gets to trump “is”, there may always be a difference both definition of “truth” and the definition of validity.
3. Law gets to define the minimum requirements for legal truth. In that sense, law is like religion, not science. See 2.
4. There is none. Relevance means whatever the particular legal system defines it to mean. See 2.
5. It was not my intent to argue the merits of the Bayes Theorem. There isn’t enough in the reasons (at least for me) to form an opinion as to whether the Bayesian analysis was done validly – in the procedural sense- by the expert who testified. My premise in my posting was that it was but that the defence position was that the data the expert relied on was insufficient for the valid application of Bayes Theorem, even if the calculations were performed properly on the data that existed. In addition, my point was that one cannot tell from the reasons (or at least I can’t) what the evidence was upon which the trial judge decided what the prior probability was (beyond accepting whatever was in the report that the particular expert delivered.) Again, the evidence might have been adduced at trial. But, it wasn’t in the reasons.
Also, I point out, I attempted to distinguish between the trier of fact”s use of Bayesian analysis to decide questions of probability – the point you address (about which there is extensive literature as you mention) – and the trier of fact’s acceptance of the probability of the occurrence (or non-occurrence) of some fact, which the trier then uses in a so-called “common sense” conclusion as to the truth of some proposition.
6. I don’t dispute your point that what passes for logical analysis in law – whether statistical or otherwise- often isn’t at all. I’d quote the concluding (about law and logic) from Lord Halsbury’s Quinn v. Leathem aphorism; however, you likely know it as well as I do and I’d be misusing it.
7. True. But a valid Bayesian analysis presumes the decision-maker has taken into account all of the evidence that could affect the prior probability. It’s not open to the decision maker to reject the evidence because allowing it to be considered would bring the field of science to which the data relates into disrepute.
8. Bear in mind in Canada we still use the jury system for both criminal and civil trials so one would never know how, in cases tried with a jury, whether the jury understood or not even if there is a finding consistent with a Bayesian analsyis of the data; nor, for that matter, why the jury chose to accept or not accept any bit of evidence. We do not use the U.S. system of long detailed questions to the jury which might provide a hint.
9. If, by the author, you mean me, you are right that I did not attempt to do so since that wasn’t the point about which I was writing; nor would I have attempt what you propose on this site even were I inclined to such an endeavour.
I’ll make 2 points in conclusion. You wrote, in your first paragraph, “There is substantial literature on this issue and it is disappointing that the author chooses to quote only one misconceived article”.
1. Indeed there is, but “this issue” is the appropriateness of judicial trier of fact use of Bayesian analyse to make the probability decision. I mentioned there was extensive literature. I didn’t see any need to quote it given the purpose of my postings (which I attempted to clarify) in my additions to the initial posting.
2. I quoted from two articles.
Kind regards,
David Cheifetz
Serendipitously (since I was writing about legal decision-making in Canadian common law jurisdictions), a recent Alberta appellate decision emphasizes the point that legal truths are what the law holds that truth to be, not what the truth might be in reality. It does not matter that the court was interpreting the meaning of a causation in workers’ compensation legislation. The court accepted this explanation of the meaning of the but-for test in Canadian common law.
The court said elsewhere, too, the correct answers to the causation question also depend on the policy of the the legislation.
DC
A recent, neutral, wry, overview piece discussing Bayesian analysis and its use in law is here: http://ssrn.com/abstract=2020052
At any moment in the life of a dispute which is before the court, it does not matter what commentators think the correct answer ought to be. What matters is what the current law is. Mr. Robertson’s second point is:
[My emphasis.]
Now consider this passage from David Goldstein, “How Science Works” in Federal Judicial Center, Reference Manual on Scientific Evidence, 3d ed. (Washington D.C.: The National Academies Press, 2011) at 53.
If Mr. Goldstein is correct, then there is no requirement that legal and scientific decision making be the same.
Russell Brown (the author of the article deprecated by Mr. Robertson) has written, elsewhere:
see Russell Brown, “Inferring Cause-in-Fact and the Search for Legal ‘Truth’”, in Richard Goldberg, ed., Perspectives on Causation (Oxford: Hart Publishing, 2011) 93 at 97. [Internal footnotes omitted.]
And, the former Mr. Justice Ian Binne of the Supreme Court of Canada wrote, in the article I quoted from initially:
Binnie, “Science in the Courtroom”, at 308-09 [internal footnotes omitted].
In any event, there is ample authority for the position that judges, in Canada and in the United States, at least, believe that there is something inherently different as between legal and scientific decision making. In the trial process, the judge has the last word.
DC