A Canadian Appeal Courts Citation Analysis
Brock Rutter, a member of the New York and Vermont bars and former research assistant at the Berkman Center and now enrolled in the master’s program at McGill Law School, wrote to me about his proposed thesis topic, wondering whether he might run it by Slaw’s readers for any advice they might have. I thought it would make sense to present his work and proposal in Q & A format:
Q. You say you’re working on “a bibliometric analysis of citations between provincial courts of appeals.” Can you explain what it means in plainer terms?
A. I will pick a year, probably in the early 1990s, and count every citation from a court of appeal to a court outside that province. I think this information will be interesting because I hope it will provide insights into the relations between the various courts. I will be collecting all this information on a spreadsheet. Imagine each row is the citing court. Each column could be a number indicating a number of citations to another province’s courts, the Canadian federal courts, or other.
I need to work out a lot of variables. For example, I am tempted to ignore “self-citations” – that is citations to earlier decisions of the same court – but might count them as it might be an interesting gauge of “introspection” or “extroversion” of the court. I am not sure how to handle for citations outside Canada, or to the Canadian federal courts, but will probably handle them similarly. I may also categorize by type of case, especially differentiating areas, like criminal, where the substantive law is federal. Also, I expect I will ignore citations in dissents, but count negative or disapproving citations similarly to positive citations.
I could also “weight” the data, perhaps doing a map with the sizes of the provinces altered to show the relative degree to which other courts cite each province. Since the output of the highest courts of the provinces ranges from a handful a year to over a thousand, I will probably also weigh these in terms out overall sample size.
Q. What do you hope to get out of such a study? What practical importance might it have?
A. I think that citation networks are changing rapidly as the result of new ways of “finding” the law. In the U.S., we are witnessing the waning days of the West Keynote system as the dominant paradigm. Before looking at how things are changing, we need to get an idea of what they are. I personally believe that, despite touted flattening effects, algorithm-based search (think Google) will actually increase network effects, that is, increase the relative prominence of a small set of leading cases as against the mass of un-cited or quickly forgotten cases.
At the very least, this has value as a training project in citation analysis in Canada. For example, I am aware of a scholar at McGill who has a theory that European legal systems are getting more parochial and inward looking. He would be interested in examining this through bibliometrics, essentially proving or disproving his hypothesis based on the frequency of citations between national court systems. Although I wasn’t interested in doing Europe-focused work, this idea influenced me.
Q. You mention a similar study in the U.S. Can you summarize the results of that study?
A. I am influenced (inspired?) by the study by William Landes, Larry Lessig, and Michael Solimine, “Judicial Influence: A Citation Analysis of Federal Courts of Appeals Judges” [PDF]. The writers were able to crunch a lot of information, which they got from the 1995 Shephard’s CD ROM sets. It seems to me that the paper is more important as a proof of concept than for any substantive conclusions drawn from it. The paper focuses mostly on the measured rankings of individual judges, but I am particularly interested in some of the appendices dealing with the circuit courts.
Judge Richard A. Posner also makes numerous references to the Landes study in explaining legal citation analysis of case law generally. He explains, “I have suggested that weighting the number of decisions of a federal court of appeals by the number of citations to those decisions by other courts of appeals, which is to say courts not bound as a matter of stare decisis to follow the cited court’s decisions, yields a meaningful measure of judicial output.” (An Economic Analysis of the Use of Citations in the Law.
Q. How can Slaw readers help you?
A. Although I’m Canadian, I’ve lived most of my life in the U.S., including for college and law school. I don’t have the depth of knowledge of the Canadian legal system that your readers do so the biggest help I can imagine is 1). directing me to any previous Canadian bibliometric / citation analysis work. (I am aware of some citation analysis work by Rosalie Fox: http://www.legalaccess.eu/IMG/pdf/fox_journees_europeennes_e-2.pdf; and 2). alerting me of any “bad choices” for a survey year, or any especially good years. (1992, for example, saw 2,608 cases from the highest courts of each province, the extremes in terms of output being three cases for Newfoundland and Labrador and 1,164 for Quebec, as found on CanLII).
Of course, I’d welcome other suggestions and comments. And I’d be glad to provide any subsets of data I collect if it would help someone else’s research.
Pre-selecting a sample period on the basis of high level descriptive statistics (e.g., “not enough decisions in year X”) invites an increased risk one or more unidentified confounds will be mixed up in the analysis. This will tend to reduce the power and validity of possible inferences about other periods that can be drawn from analysis of the sample.
It will probably prove helpful to consult someone with knowledge about statistics early on in the design of any such analysis.
This sounds like a great project. Back in 2002, Jane Parkinson (former librarian at Parlee McLaws, then with the Calgary office of Stikeman Elliot) published results of a citation analysis project that she undertook with assistance from a CALL/ACBD Research Grant (http://www.callacbd.ca/index.php/publisher/articleview/frmArticleID/204/). The CALL Research database offers the following citation:
Title: Citation Analysis Project
Researcher: Jane Parkinson
Funding
Agency: Committee to Promote Research, Canadian Association of Law Libraries
Abstract: The purpose of the project is to analyze the average frequency of citations for each of the following years: 1940, 1950, 1960, 1970, 1980 and 1990 and the length of time over which decisions are cited. A pattern of longevity will be determined for each record and for each year. If time permits, some analysis of frequently-cited and/or long-lived decisions will be presented. Ongoing.
This research has been published as:
■”Fine Wine in Dusty Bottles? A Study of the Age of Cited Decisions” in Canadian Law Libraries 27 (Summer 2002) 63-66 and
■”The Labels on the Bottles: Or, Where to Find Cited Cases” in Canadian Law Libaries 28 (Summer 2003) 62-64.
Though Jane’s project was not appeal specific, the research had a very interesting outcome and will offer some historical comparisons for you.
Best of luck with your project!
I supervised a graduate student at the University of Toronto some 30 years ago who analyzed the Supreme Court’s use of foreign and academic writings. He compared the court’s use of these sources at two distinct periods and the comparison was at the heart of his analysis. (I still have a copy of his thesis.)
It will be much more useful to compare the courts’ practices at (at least) two distinct periods; one, pre-(widespread)QL, Westlaw and CanLII, and one afterwards. What my impressionistic view is that courts, both trial courts and courts of appeal, are now much more “inward” looking than they were. In the “old days” (when we still had periods in citations!), the D.L.R.’s and the W.W.R.’s forced a national (or regional) view of the law, as did the topical law reports. This last fact was probably more important in practice, since national series like the D.L.R. have always had a very small circulation, being limited to academic libraries and those of the larger firms. With the advent of provincial reports, B.C.L.R., Alta. L.R., etc., and, in particular, on-line resources, there are now enough easily accessible decisions from all provinces—perhaps not in P.E.I., though I haven’t checked—that in many reasons for judgment at both the trial level and court of appeal the only decisions referred to are from the province in which the court sits.
My impression is that this fact is true with respect to the decisions of the Federal Court, at least in decisions emanating from Quebec, and in matters where provincial law may be applicable. (Canada does not have and, so far as I am aware, has never even debated whether we should adopt the Erie doctrine with respect to the provincial law to be applied by the Federal Court.)
In part, the recent inward looking practice may be due to the fact that for counsel researching the law it is easier and cheaper to limit his or her research to one province. Judges, who have, I believe, free access to QL, etc., are often not motivated to go wider than counsel has—and, of course, there are serious risks to the integrity of the judicial process if the court refers to and, in particular, relies on cases that counsel did not cite.
In Brock’s first Answer he says, among other things, “I am not sure how to handle for citations outside Canada, or to the Canadian federal courts, but will probably handle them similarly.” His reference to “Canadian federal courts” leads me towhether he might benefit from reading some basic texts on the structure and organization of the Canadian court system.
I think it would, in fact, be quite interesting to know how much attention the Federal Court of Appeal pays to earlier decisions of the provincial court of appeal of a province from which it is hearing the case. In cases dealing with Indian Act provisions plaintiffs often have a choice between bringing their case before the superior court of the province or the Federal Court. One might then expect that both the FCA and the provincial CA had rulings on the same or similar provisions.
Angela Swan wrote:
You are certainly right about the “inward looking” trend. This will cause problems, too. There are are risks to the integrity of the process where, for example, the governing decision is a Supreme Court of Canada decision which purports to apply to the common law of all of the provinces – maybe even the civil law by analogy – where the decision has been considered by the appellate courts (and the trial courts) of other provinces, and one would never now that from a particular provinces’s jurisprudence.
Peter McCormick has authored or co-authored several studies of citation practices, including one of provincial C.A.s in Final Appeal
You could look at decisions that were cited only once(or less than any small limiit of times) and see if they were potentially applicable to more cases, and the party in whose favour they were cited got an unusual advantage.
Similarly, you could, if you used Quicklaw descriptions of citations or ones by C.A.s similar to those the Supreme Court of Canada’s staff do for citations in nondissenting opinions, look at whether and how often cases were “distinguished” versus “followed” and so on.
I am trying to prepare for publication an article on the citations Quicklaw recorded (on a day in 2003) of a group of Supreme Court of Canada cases.
Thanks to all of you for the helpful suggestions.
I’m familiar with the literature on citation analysis referred to here. While I agree with Angela Swan about some more focus to the proposed research, I have to question its utility the way the proposed search is presently framed. It leaves open the question that the data will be descriptive but not give us much that we don’t already know, albeit anecdotedly: e.g. BC appellate courts generally defer to decisions from the Courts of Appeal of Alberta or Ontario, then to the other western provinces, then to the maritimes, and then to Quebec, with some variation.
The more interesting study I would think would be a citation analysis of the citing of English House of Lords/Supreme Court common law decisions since the UK joined the European Union. I number of us who have thought about this question wether the English common law thread has been broken by the infusion of EU law, and that the place to look for development of common law principles are the appellate courts of the Commonwealth excluding the UK, e.g. Canada, Australia, NZ etc.
As well, a citation of the extent to which foreign, comparative, and international law is cited with approval in Canadian appellate courts would be informative and helpful to legal scholars and legal researchers generally.
So good idea, change the direction of the topic.