Column

Do You Believe in Peer-Review?

Peer-review is a widely accepted process in scholarly publishing. It’s seen as a sign of quality and a way to establish legitimacy. There are, however, drawbacks to this process too. It takes time and doesn’t always give consistent results. What benefits do we get from the peer-review process and is it worth the costs? Are the benefits the same for legal information as they are for other disciplines?

Many journals, whether scientific or legal, open access or behind a paywall, use peer-review because it provides status that can help writers with tenure promotion or securing grants and scholarships. There are many other benefits to peer review as well:

It provides a quality check.

  • Peer-review makes sure the information is vetted by experts in the field. Review from peers establishes the validity of research based upon their expert knowledge. 

It provides valuable feedback to authors.

  • Not only can peer-reviewers with the right expertise check the facts, they can also share statements and opinions. This feedback helps authors critically reflect on their work and improve it before publication.
  • This knowledge exchange provides an opportunity for mentorship and community building among scholars/researchers/practitioners with similar interests.

It can help publishers decide whether or not to accept a manuscript. 

It spreads awareness of an upcoming resource.

On the other hand, there are also commercial legal publishers that have opted out of the peer-review process altogether for legal texts such conference proceedings etc. Some academic publishers decide to skip this step in exchange for post-publication community-driven feedback. Some drawbacks of peer-review are:

There are a number of ways peer-review can go wrong and no clear consensus on how to go about it. 

  • What is peer-review and who is a “peer”? Is it a lawyer working in the same field (so perhaps a direct competitor?) An expert on the subject? 
  • What is “review”? Someone saying “looks good to me” or filling out a questionnaire? Critiquing every detail of an argument, checking references, and making detailed suggestions for improvement?
  • Will it be anonymous? How many reviewers should you have? Will they look at the full-text, just a chapter, the book proposal? Will it be done pre- or post- publication? Do you have guidelines? How will reviewers be compensated and/or acknowledged? There is a lot to consider and some methods can be perceived as better than others (e.g. double-blind vs. single-blind peer review).

It’s time consuming and slows down the publication process. 

  • For legal material, publishing current information is key to staying properly informed.

It can get expensive.

  • Whether you are compensating a review manager or individual peer-reviewers, this process can easily add up. However, it is often the case that peer reviewers aren’t compensated financially. Even non-financial rewards such as acknowledgements can’t always be expected.

It’s not consistent.

  • Two peer reviewers can come back with completely different feedback and it’s not a guarantee that they will find every mistake. Peer-reviewed does not guarantee the published work is free from errors. 
  • In science, while feedback will contain opinions, reviewers can check the more objective aspects of the work such as the design of an experiment, methodology etc. In law, reviewers can assess whether a text includes the expected cases and legal topics, but the application of the law can be subjective. You could argue that this allows for more inconsistencies in reviews of legal texts compared to scientific ones.

It’s open to bias.

  • When there is only a small group of people who are experts in a particular area, there is the risk of protecting established opinions and not being open to new ideas. This makes it difficult to protect anonymity and can make room for bias, whether positive or negative. This also means a smaller pool of reviewers to pick from. 

It relies strongly on trust.

  • On the one hand, this reliance on trust can help bring people closer (think of the “trust fall” exercise). In order to be constructive, peer-review requires rigor, objectiveness, and transparency from both the authors and reviewers. On the other hand, some people might abuse this part of the peer-review process. For example, reviewers might provide very little or unhelpful feedback, but still ask for compensation, or in extreme cases steal ideas and republish them as their own.

What about legal information outside of academia, such as practice manuals, that appeal to a wider audience and where there is an advantage to publishing quickly? Are there instances where we are better off without peer-review for legal material? 

It’s clear that the peer-review process is not perfect, but it’s deeply embedded in the scholarly publication lifecycle. We should be critical of this process and look for ways to improve it, or for alternatives, so that we can have more reliable results. 

If you’ve been a peer-reviewer, had your work peer-reviewed, or have any thoughts on this process, please let me know!

Comments

  1. I definitely believe in peer review, at least for scholarly writing. Very often I have laboured on a piece of my own research until I feel it’s impeccable, but then received peer reviews of it that expose vast and undeniable room for improvement. We humans are highly prone to tunnel vision, especially re work product that we are personally invested in. The double-blinding is very helpful as a way to remove ego and bias from the process, and promote an authentic knowledge exchange between experts.

    I edited a peer-reviewed journal (the Windsor Yearbook of Access to Justice) for several years, and this blog identifies many of the issues that Editors must grapple with. We did not pay reviewers. Fortunately, enough academics understand that peer reviewing is a core part of the job. Contradictory reviews of a single manuscript could be challenging. However this is often an opportunity for the author to revise their piece to respond to the negative review before it is eventually published. I was consistently gratified by the improvement in manuscripts, between initial submission and publication, brought about by the peer review process.

  2. Hi Noel! Thanks so much for sharing your insight on this. It’s great to know that, despite the issues, you were grateful for this process.