Brock Rutter, a member of the New York and Vermont bars and former research assistant at the Berkman Center and now enrolled in the master's program at McGill Law School, wrote to me about his proposed thesis topic, wondering whether he might run it by Slaw's readers for any advice they might have. I thought it would make sense to present his work and proposal in Q & A format:
Q. You say you're working on "a bibliometric analysis of citations between provincial courts of appeals." Can you explain what it means in plainer terms?
A. I will pick a year, probably in the early 1990s, and count every citation from a court of appeal to a court outside that province. I think this information will be interesting because I hope it will provide insights into the relations between the various courts. I will be collecting all this information on a spreadsheet. Imagine each row is the citing court. Each column could be a number indicating a number of citations to another province’s courts, the Canadian federal courts, or other.
I need to work out a lot of variables. For example, I am tempted to ignore “self-citations” – that is citations to earlier decisions of the same court – but might count them as it might be an interesting gauge of “introspection” or “extroversion” of the court. I am not sure how to handle for citations outside Canada, or to the Canadian federal courts, but will probably handle them similarly. I may also categorize by type of case, especially differentiating areas, like criminal, where the substantive law is federal. Also, I expect I will ignore citations in dissents, but count negative or disapproving citations similarly to positive citations.
I could also “weight” the data, perhaps doing a map with the sizes of the provinces altered to show the relative degree to which other courts cite each province. Since the output of the highest courts of the provinces ranges from a handful a year to over a thousand, I will probably also weigh these in terms out overall sample size.
Q. What do you hope to get out of such a study? What practical importance might it have?
A. I think that citation networks are changing rapidly as the result of new ways of “finding” the law. In the U.S., we are witnessing the waning days of the West Keynote system as the dominant paradigm. Before looking at how things are changing, we need to get an idea of what they are. I personally believe that, despite touted flattening effects, algorithm-based search (think Google) will actually increase network effects, that is, increase the relative prominence of a small set of leading cases as against the mass of un-cited or quickly forgotten cases.
At the very least, this has value as a training project in citation analysis in Canada. For example, I am aware of a scholar at McGill who has a theory that European legal systems are getting more parochial and inward looking. He would be interested in examining this through bibliometrics, essentially proving or disproving his hypothesis based on the frequency of citations between national court systems. Although I wasn’t interested in doing Europe-focused work, this idea influenced me.
Q. You mention a similar study in the U.S. Can you summarize the results of that study?
A. I am influenced (inspired?) by the study by William Landes, Larry Lessig, and Michael Solimine, "Judicial Influence: A Citation Analysis of Federal Courts of Appeals Judges" [PDF]. The writers were able to crunch a lot of information, which they got from the 1995 Shephard’s CD ROM sets. It seems to me that the paper is more important as a proof of concept than for any substantive conclusions drawn from it. The paper focuses mostly on the measured rankings of individual judges, but I am particularly interested in some of the appendices dealing with the circuit courts.
Judge Richard A. Posner also makes numerous references to the Landes study in explaining legal citation analysis of case law generally. He explains, “I have suggested that weighting the number of decisions of a federal court of appeals by the number of citations to those decisions by other courts of appeals, which is to say courts not bound as a matter of stare decisis to follow the cited court’s decisions, yields a meaningful measure of judicial output.” (An Economic Analysis of the Use of Citations in the Law.
Q. How can Slaw readers help you?
A. Although I'm Canadian, I've lived most of my life in the U.S., including for college and law school. I don't have the depth of knowledge of the Canadian legal system that your readers do so the biggest help I can imagine is 1). directing me to any previous Canadian bibliometric / citation analysis work. (I am aware of some citation analysis work by Rosalie Fox: http://www.legalaccess.eu/IMG/pdf/fox_journees_europeennes_e-2.pdf; and 2). alerting me of any “bad choices” for a survey year, or any especially good years. (1992, for example, saw 2,608 cases from the highest courts of each province, the extremes in terms of output being three cases for Newfoundland and Labrador and 1,164 for Quebec, as found on CanLII).
Of course, I'd welcome other suggestions and comments. And I'd be glad to provide any subsets of data I collect if it would help someone else's research.