Avoiding the Guise of an Anonymous Review

The scientific community takes the current structure of the peer review system as a given. But what if there were a better way to conduct reviews? We are two colleagues, a Ph.D. student and an assistant professor, at Canada’s University of Saskatchewan in Saskatoon. In summer, our daily work routine often ends with a leisurely walk on our way homeward along the South Saskatchewan River, strolling along footpaths shaded by trees. We enjoy something we all do: engaging in a good conversation with a good companion. One day, we struck up a conversation on the role and quality of the peer review process in the scientific community, and we wondered which peer review system is the best. As we continued this conversation, we realized several differing ideas around peer review and the challenges of blindness and openness from the points of view of both reviewer and editor. Are anonymous reviews fair? Are they effective? What do you gain or lose by choosing to be anonymous? Our talk revolved around some basic questions. Are anonymous reviews fair? Are they effective? What do you gain or lose by choosing to be anonymous? In practice, can you really be anonymous? Are there alternative ways of conducting quality reviews? From our experiences and the data that we could find, we agreed that there are enough pitfalls in the current system of reviewing anonymously to warrant considering other approaches. We also feel that lifting the veil—allowing your name to be linked to your review and then making that review open to all—would be a self-regulating system where only thoughtful reviews are submitted.

How did we reach these conclusions? Let’s begin with the first author’s experience.

A Beginner’s Experience with Scientific Peer Review

After publishing a few papers, I started getting review requests from several journals. Apart from getting excited, I often find myself on the horns of a dilemma: whether to review a paper anonymously or sign my name.

This dilemma became worse when I received an invitation to review a manuscript written by a renowned researcher. In my review, I had several concerns about the manuscript. However, the author was in a position of power and influence in the scientific community, and I faced (perhaps foolishly) the fear of retaliation. I was thinking subconsciously (or consciously) about who the author is and how he might affect my career path in future, and thus, my review could not be completely objective. Finally, I submitted my review comments anonymously, albeit reluctantly! But if I had revealed my name, it probably would not have affected me during my career.

This story is an example of what may happen when reality collides with the idealism of objectivity. If someone decides when to anonymize a review on the basis of potential risks or benefits, she or he is simply making a personal value judgement on a case-by-case basis, which is against the ethical guidelines for peer reviewers. Therefore, “to be anonymous or not to be” is more than a question, it is a choice.

To Name or Not to Name?

The consensus within the scientific community is that peer review is a major driver of scholarly integrity, credibility of research, and commitment to the advancement of science. Traditionally, the peer review process has been anonymous in a single-blind manner, where the referees’ identities remain hidden from the authors, or a double-blind manner, where the authors’ identities are also withheld from reviewers. An “open review” process, where the identities of both reviewers and authors are known to each other, is an alternative to the traditional process.

Does the anonymous peer review, as currently practiced, fail to do its job properly? This is a controversial topic that has long been debated among scholars. The proponents of anonymous peer review argue that it promotes objectivity, decreases sexism and nepotism, and prevents bland or timid reviews.

More specifically, advocates of double-blind review, which is an extreme view on anonymity, believe that it can decrease systematic biases such as the Matilda effect, in which the contributions of female scientists are overlooked compared with those of their male colleagues, and the Matthew effect, in which eminent scientists take more credit than those who are less well known but are almost equally qualified [Rossiter, 1993; Lerback and Hanson, 2017]. Note that in practice a truly double blind review may be rare because the identity of an author may be recognized by the language used, bibliographical references, access to specific data sets, prior presentations, or other factors.

Nevertheless, the anonymous peer review has been increasingly criticized, mainly because of its lack of transparency. Opponents of anonymous peer review assert that open review stops hostile, inflammatory, and unsubstantiated comments; mitigates discourtesy; and discloses possible conflicts of interest.

Although there are arguments for and against openness, it can enable researchers to see the “who, why, and what” behind editorial decision-making.

Although there are arguments for and against openness, it can enable researchers to see the “who, why, and what” behind editorial decision-making. Also, the general policy of disclosure can lead reviewers to provide more objective comments, think more carefully about the scientific issues, and write thorough and fair reviews rigorously. Additionally, reviewers can further discuss the results and critical issues directly with the authors. Under any of the peer review models, the role of journal editors and the issues they face should not be understated. Editors direct the process, pay attention to detail, and make decisions. Furthermore, editors must confront the big, and perhaps growing, challenge of securing qualified reviewers and encouraging them to continue reviewing regularly. Given the sheer volume of submissions and the ever-increasing number of journals, editors are finding it harder to persuade their colleagues to perform reviews [Helffrich, 2013]. In the second author’s experience, fewer than half of his invitations to review manuscripts are accepted. This rate can further decrease in an open review system, which discourages reviewers who want to remain anonymous [Almquist et al., 2017].

Improving Our System

How might we improve the American Geophysical Union’s (AGU) peer review system? Let’s begin our analysis of AGU’s policy rules by looking into its ethical guidelines for peer reviewers. Per these guidelines, which are based on the Committee on Publication Ethics’ Ethical Guidelines for Peer Reviewers [Hames, 2013], reviewers should do the following:

“decline to review if they have issues with the peer-review model used by a journal”

“determine whether the journal allows them to sign their reviews and, if it does, decide as they feel comfortable doing”

Apparently, these guidelines provide minimal freedom for reviewers to choose their review model of interest. Basically, the single-blind review model is currently adopted by AGU, with the optionfor reviewers to sign their reviews. If a reviewer disagrees with this model, he or she should reject the review request. Also, this model provides no flexibility for authors. For example, an author who prefers a double-blind review will be discouraged from submitting to an AGU journal.

Can AGU adopt an alternative review policy? Although a simple one-size-fits-all solution is unlikely to be achieved, the following strategies may be considered by AGU to improve this important and sensitive process.

Three Proven Models

First, both authors and reviewers need to have the freedom of choice. AGU journals may consider offering alternative review models to authors in the submission process and to reviewers during an invitation to review. Nature Climate Change and Nature Geosciences recognized this need in part, and since 2013, these journals have enabled authors to choose between single- and double-blind reviews. In March 2015, Nature and the monthly Nature research journals started to pursue the same strategy [Nature, 2015].

Second, the transparency of the peer review process can be improved by the inclusion of the “peer review process documents” (possibly excluding minor typographical comments) with published papers. This strategy will make the reviews accessible and referable, thereby increasing accountability. In 2008, the European Molecular Biology Organization began publishing reviewers’ and editors’ comments as well as authors’ responses. This has been a viable and overwhelmingly positive move [Pulverer, 2010]. The European Geosciences Union has also adopted a similar strategy successfully.

Third, AGU may develop an initiative that invites its community members to submit ideas to set up pilot peer review systems. Open to all, this call may seek new ideas that remarkably improve AGU’s current peer review system. In 2012, Elsevier announced the “Peer Review Grand Challenge” competition. This global competition attracted some 800 innovative ideas, from which three winning ideas were announced to be further developed [Lehane, 2012]:

Systematically rewarding reviewers through an official accreditation system that they could cite on their curricula vitae

Developing online and cloud-based platforms to enable reviewers to easily view and annotate submissions

Launching a public reputation system to assign a “credit” to reviewers

Toward Removing the Veil of Anonymity

In 1987, William Hoover, a physicist at Lawrence Livermore National Laboratory, did an experiment. One of his papers had been rejected by two famous journals. Then, he made up an imaginary name, added it to the author list, and changed the title. He resubmitted the paper, and surprisingly, one of those journals accepted the new version. That new name was not only a fake name but also a very rude Italian phrase. The paper still exists, and the imaginary author has been cited in many articles. The lesson to learn from Hoover’s story is that papers must be reviewed purely on the basis of their merits and pitfalls, not on who wrote them.

Each paper resembles a brick used to build up and advance the structure of the scientific enterprise, and it needs to be examined before the next brick is placed on top of it.

Peer review is an integral part of the scientific enterprise. A Persian poet, Saib Tabrizi, once said, “If the first brick is laid askew, the wall will be built askew, even if it goes to the sky.” Each paper resembles a brick used to build up and advance the structure of the scientific enterprise, and it needs to be examined before the next brick is placed on top of it. Screening overall quality, appreciating originality, and sustaining integrity require the collaborative and collective efforts of all the members of the scientific community in their interchanging roles as authors and reviewers. Hiding behind the cloak of anonymity may not be a way out. To promote an open scientific dialogue, protect trust in the scientific community, and mitigate the burden on reviewers and editors, beginning reviewers should be trained through reviewer guidance workshops to have the courage, honesty, and dignity to sign their reviews without being fearful of reprisal. In arts, literature, and policy, it is common for critics to sign their review comments to bring a level of trust to the peer review process. Promoting that culture across the entirety of the AGU community may prove to be a game changer!

References

Almquist, M., et al. (2017), A prospective study on an innovative online forum for peer reviewing of surgical science, PloS One, 12(6), e0179031, https://doi.org/10.1371/journal.pone.0179031.

Hames, I. (2013), COPE ethical guidelines for peer reviewers, Comm. on Publ. Ethics, Norfolk, U.K.

Helffrich, G. (2013), A modestly rewarding proposal concerning peer review, Eos Trans. AGU, 94(48), 459, https://doi.org/10.1002/2013EO480004.

Lerback, J., and B. Hanson (2017), Journals invite too few women to referee, Nature, 541, 455–457, https://doi.org/10.1038/541455a.

Lehane, C. (2012), Peer Review Challenge: Winners unveiled, Elsevier, 30 Oct., https://www.elsevier.com/editors-update/story/peer-review/peer-review-challenge-winners-unveiled.

Nature (2015), Nature journals offer double-blind review, Nature, 518, 274, https://doi.org/10.1038/518274b.

Pulverer, B. (2010), Transparency showcases strength of peer review, Nature, 468, 29–31, https://doi.org/10.1038/468029a.

Rossiter, M. W. (1993), The Matthew Matilda effect in science, Soc. Stud. Sci., 23(2), 325–341, https://doi.org/10.1177/030631293023002004.

—Razi Sheikholeslami (email: razi.sheikholeslami@usask.ca) and Saman Razavi, Global Institute for Water Security, School of Environment and Sustainability, University of Saskatchewan, Saskatoon, Canada

Citation: Sheikholeslami, R., and S. Razavi (2018), Avoiding the guise of an anonymous review, Eos, 99,https://doi.org/10.1029/2018EO098217. Published on 09 May 2018.

This article originally appeared on EOS Earth and Space Science News