Uncategorized

Peer Review Week: Should we use double blind peer review? The evidence…

Non-blind Peer Review Monster

This week is Peer Review Week, the slightly more popular academic celebration than pier review week. Peer review is an essential part of scientific publication and is – like Churchill’s democracy – the worst system to do it. Except for all of the others. The reason it’s imperfect is mainly that it’s done by people, so there is a natural desire to try to improve it.

One suggestion for improvement is to us double blind reviews. At the moment most journals (including Methods in Ecology and Evolution) use single blind reviewing, where the author isn’t told the identity of the reviewers. The obvious question is whether double blind reviewing does actually improve reviews: does it reduce bias, or improve quality? There have been several studies in several disciplines which have looked at this and related questions. After having looked at them, my summary is that double blind reviewing is fairly popular, but makes little or no difference to the quality of the reviews, and reviewers can often identify the authors of the papers.

Does Double Blind Reviewing Improve Review Quality?

I found 13 studies about this (some with replies and re-analyses). Overall, there is little effect of double blinding on quality: in some studies the recommendations to the editor were more negative (i.e. more likely to recommend rejection or major revisions), but of the five studies that looked at quality (mostly by judgment of authors or editors), three found no effect, ad the other two found opposite effects – one with double-blinding higher, one with it lower.

A few studies looked at whether double blinding changed the biases (e.g. due to gender). Almost all effects were negative: Blank (1991) and Isenberg (2009) found no interaction between gender and blinding. Budden et al. (2008) claimed there was an effect, but Webb et al. (2008) came to the opposite conclusion. As the et al. of the latter paper includes two senior editors of MEE (including myself), it is clear what the only correct conclusion is. Isenberg (2009) also found no effects of geography locations, but Ross et al. (2006) found that abstracts submitted from outside the US to a US conference were more likely to be accepted if they were blinded.

There are a few points worth noting. First, if double blind reviewers consistently score papers lower, this may not be a problem: if a journal accepts the ‘top’ 100 papers a year, then it doesn’t matter if the mean rating is 4.2 or 44.2. The problem, then, is whether the blinding changes who gets good score, e.g. does double-blinding mean that womens’ papers are not judged as harshly? In general the answer seems to be no for papers in journals: there may be differences due to experience (Fisher, 1994), but this may be because more experienced researchers are better researchers. Or perhaps it’s because the blinding is not successful…

Is Blinding Successful?

This is an obvious problem with double-blinding: it is often possible to work out who the author is anyway. Several studies have looked at this, and the consistent conclusion is that a large minority of papers can be successfully unblinded, from 22% (Isenberg, 2009) to 46% (Fisher, 1994: 52% if incorrect identifications are included). My favourite study of this is from Hill & Provost (2003), who found that they could train their computers on abstracts, and they identify the authors 40-45% of the time.

The problem here is that if blinding does remove some biases, then imperfect blinding does this imperfectly, and not randomly: a reviewer is more likely to identify an author they know or who is famous. This means that the biases just get shuffled around, possibly in ways which exacerbate the problems. For example, if the only effect of double blind reviewing is that you give lower scores, then more famous people may tend to get higher scores, just because they are easier to recognise. At the very least, this is an issue that needs to be taken seriously when evaluating double blind reviewing: in the end it may not be a problem.

Is Blinding Popular?

In the last decade there have been 8 surveys (that I have found) asking about the popularity of different forms of peer review, in different ways. The surveys asked about review in two ways: “which do you prefer”, and “Is X effective?”: the latter could mean that all forms are considered effective. Of those asking which do you prefer, 35% to 93% of respondents said they preferred double blind reviews. The variation reflects the number of options available, and what form of peer review is currently commonly used. For the latter question, in all cases a majority said that double blind review is effective (or improves peer review): the precise number depends a bit on the question(s) asked, but informally about 70% thinking it makes things better looks about right.

In Conclusion…

Double blind peer review is certainly not a panacea: the evidence is mixed at best, although it is fairly popular. It does feel as if more studies of effectiveness would help: only a couple of studies have looked at effects of gender, experience and location, and it feels as if any effects may vary between fields.

A stupidly long list of references for a blog post

Double blind and quality

  1. Alam M, Kim NA, Havey J, Rademaker A, Ratner D, Tregre B, et al. Blinded vs. unblinded peer review of manuscripts submitted to a dermatology journal: A randomized multi-rater study. Br J Dermatol. 2011;165(3):563–7.
  2. Blank RM. The Effects of Double-Blind versus Single-Blind Reviewing: Experimental Evidence from The American Economic Review. Am Econ Rev [Internet]. 1991;81(5):1041–67. Available from: http://www.jstor.org/stable/10.2307/2006906
  3. Budden A, Tregenza T, Aarssen L, Koricheva J, Leimu R, Lortie C. Double-blind review favours increased representation of female authors. Trends Ecol Evol [Internet]. 2008 Jan;23(1):4–6. Available from: http://linkinghub.elsevier.com/retrieve/pii/S0169534707002704
  4. Carlsson F, Löfgren Å, Sterner T. Discrimination in Scientific Review: A Natural Field Experiment on Blind versus Non-Blind Reviews. Scand J Econ. 2012;114(2):500–19.
  5. Chung KC, Shauver MJ, Malay S, Zhong L, Weinstein A, Rohrich RJ. Is Double-Blinded Peer Review Necessary? The Effect of Blinding on Review Quality. Plast Reconstr Surg [Internet]. 2015 Dec;136(6):1369–77. Available from: http://content.wkhealth.com/linkback/openurl?sid=WKPTLP:landingpage&an=00006534-900000000-99134
  6. Fisher M. The Effects of Blinding on Acceptance of Research Papers by Peer Review. JAMA J Am Med Assoc [Internet]. 1994 Jul 13;272(2):143. Available from: http://jama.ama-assn.org/cgi/doi/10.1001/jama.272.2.143
  7. Godlee F, Gale CR, Martyn CN. Effect on the quality of peer review of blinding reviewers and asking them to sign their reports: a randomized controlled trial. JAMA. 1998;280(3):237–40.
  8. Isenberg SJ, Sanchez E, Zafran KC. The effect of masking manuscripts for the peer-review process of an ophthalmic journal. Br J Ophthalmol [Internet]. 2009;93(7):881–4. Available from: http://bjo.bmj.com/cgi/content/abstract/93/7/881
  9. Jagsi R, Bennett KE, Griffith KA, Decastro R, Grace C, Holliday E, et al. Attitudes toward blinding of peer review and perceptions of efficacy within a small biomedical specialty. Int J Radiat Oncol Biol Phys [Internet]. Elsevier Inc.; 2014;89(5):940–6. Available from: http://dx.doi.org/10.1016/j.ijrobp.2014.04.021
  10. Justice AC, Cho MK, Winker MA, Berlin JA, Rennie D, and the PEER Investigators. Does Masking Author Identity Improve Peer Review Quality? JAMA [Internet]. 1998 Jul 15;280(3):240. Available from: http://jama.jamanetwork.com/article.aspx?doi=10.1001/jama.280.3.240
  11. McNutt RA, Evans AT, Fletcher RH, Fletcher SW. The effects of blinding on the quality of peer review. A randomized trial. JAMA J Am Med Assoc [Internet]. 1990 Mar 9;263(10):1371–6. Available from: http://jama.jamanetwork.com/article.aspx?doi=10.1001/jama.1990.03440100079012
  12. Rooyen S, Godlee F, Evans S, Smith R, Black N. Effect of blinding and unmasking on the quality of peer review. J Gen Intern Med [Internet]. 1999 Oct 15;14(10):622–4. Available from: http://link.springer.com/10.1046/j.1525-1497.1999.09058.x
  13. Ross JS, Gross CP, Desai MM, Hong Y, Grant AO, Daniels SR, et al. Effect of blinded peer review on abstract acceptance. JAMA. 2006;295(14):1675–80.
  14. Webb TJ, O’Hara B, Freckleton RP. Does double-blind review benefit female authors? Trends Ecol Evol [Internet]. 2008 Jul [cited 2012 Mar 18];23(7):351-3-4. Available from: http://www.ncbi.nlm.nih.gov/pubmed/18450323

Effectiveness of Blinding

  1. Baggs JG, Broome ME, Dougherty MC, Freda MC, Kearney MH. Blinding in peer review: The preferences of reviewers for nursing journals. J Adv Nurs. 2008;64(2):131–8.
  2. Blank RM. The Effects of Double-Blind versus Single-Blind Reviewing: Experimental Evidence from The American Economic Review. Am Econ Rev [Internet]. 1991;81(5):1041–67. Available from: http://www.jstor.org/stable/10.2307/2006906
  3. Cho MK, Justice AC, Winker MA, Berlin JA, Waeckerle JF, Callaham ML, et al. Masking Author Identity in Peer Review. JAMA [Internet]. 1998 Jul 15;280(3):243. Available from: http://jama.jamanetwork.com/article.aspx?doi=10.1001/jama.280.3.243
  4. Fisher M. The Effects of Blinding on Acceptance of Research Papers by Peer Review. JAMA J Am Med Assoc [Internet]. 1994 Jul 13;272(2):143. Available from: http://jama.ama-assn.org/cgi/doi/10.1001/jama.272.2.143
  5. Hill S, Provost F. The myth of the double-blind review? ACM SIGKDD Explor Newsl [Internet]. New York, NY, USA: ACM; 2003 Dec 1;5(2):179. Available from: http://doi.acm.org/10.1145/980972.981001
  6. Isenberg SJ, Sanchez E, Zafran KC. The effect of masking manuscripts for the peer-review process of an ophthalmic journal. Br J Ophthalmol [Internet]. 2009;93(7):881–4. Available from: http://bjo.bmj.com/cgi/content/abstract/93/7/881
  7. Justice AC, Cho MK, Winker MA, Berlin JA, Rennie D, and the PEER Investigators. Does Masking Author Identity Improve Peer Review Quality? JAMA [Internet]. 1998 Jul 15;280(3):240. Available from: http://jama.jamanetwork.com/article.aspx?doi=10.1001/jama.280.3.240
  8. McNutt RA, Evans AT, Fletcher RH, Fletcher SW. The effects of blinding on the quality of peer review. A randomized trial. JAMA J Am Med Assoc [Internet]. 1990 Mar 9;263(10):1371–6. Available from: http://jama.jamanetwork.com/article.aspx?doi=10.1001/jama.1990.03440100079012
  9. Yankauer A. How blind is blind review? Am J Public Health [Internet]. 1991 Jul;81(7):843–5. Available from: http://ajph.aphapublications.org/doi/abs/10.2105/AJPH.81.7.843

Popularity

  1. Baggs JG, Broome ME, Dougherty MC, Freda MC, Kearney MH. Blinding in peer review: The preferences of reviewers for nursing journals. J Adv Nurs. 2008;64(2):131–8.
  2. Beller M, Bacchelli A. Double blind reviews in software engineering venues : Practicability , promises and perils Double Blind Reviews in Software Engineering Venues : Practicability , Promises and Perils. 2016; Available from: https://doi.org/10.7287/peerj.preprints.1757v1
  3. Brown RJC. Double anonymity in peer review within the chemistry periodicals community. Learn Publ. 2007;20(2):131–7.
  4. Jagsi R, Bennett KE, Griffith KA, Decastro R, Grace C, Holliday E, et al. Attitudes toward blinding of peer review and perceptions of efficacy within a small biomedical specialty. Int J Radiat Oncol Biol Phys [Internet]. Elsevier Inc.; 2014;89(5):940–6. Available from: http://dx.doi.org/10.1016/j.ijrobp.2014.04.021
  5. Kmietowicz Z. Double blind peer reviews are fairer and more objective, say academics. BMJ [Internet]. 2008 Feb 2;336(7638):241–241. Available from: http://www.bmj.com/cgi/doi/10.1136/bmj.39476.357280.DB
  6. Moylan EC, Harold S, O’Neill C, Kowalczuk MK. Open, single-blind, double-blind: which peer review process do you prefer? BMC Pharmacol Toxicol [Internet]. 2014;15(1):55. Available from: http://www.biomedcentral.com/2050-6511/15/55
  7. Mulligan A, Hall L, Raphael E. Peer review in a changing world: An international study measuring the attitudes of researchers. J Am Soc Inf Sci Technol [Internet]. 2013 Jan;64(1):132–61. Available from: http://doi.wiley.com/10.1002/asi.22798
  8. Regehr G, Bordage G. To blind or not to blind? What authors and reviewers prefer. Med Educ [Internet]. 2006 Sep;40(9):832–9. Available from: http://doi.wiley.com/10.1111/j.1365-2929.2006.02539.x

12 thoughts on “Peer Review Week: Should we use double blind peer review? The evidence…”

  1. Pingback: Peer Review Week: Should we use double blind peer review? The evidence… | The Applied Ecologist's blog

  2. This is really interesting, and thanks so much for putting together the literature on this. One thing that I’m curious about is the relative power of each study, and also the effectiveness of the study design.

    I think there are two ways to evaluate whether or not blind peer review results in less bias. One is to conducted a randomized study and then compare the review quality using some kind of measure. The second way is to actually implement a double-blind policy in a venue where there was not one before. This way, you can have a pre-double blind and post double-blind comparison. I think both approaches are valid, but the latter approach is one that actually gives us the information that we really want to know: does going double blind actually change things in a way that promotes more equity?

    Reply

    1. Thanks. On the power, several studies were really under-powered (this is one issue that a blog post is too short to include), especially with some of the surveys.

      I prefer the experimental approach where the same paper is sent to different reviewers, as that should have more power (although some studies do that and then ignore the paired nature of the study). One problem with implementing a before/after study is that lots of other things can change too, so you can’t be sure that differences are due to double blinding. Look at the Budden et al. study and the responses to see the sorts of problems that can arise. There is evidently quite a bit of heterogeneity between journals, so you can’t conclude much if a single journal is your unit of observation.

      Reply

  3. At Axios Review we give people the choice between single and double blind, and uptake on double blind is around 10% of submissions. There are clearly groups who feel that their identity hampers their ability to receive objective peer review, or that just philosophically prefer double blind, but they are in a minority.

    We’ve recently implemented a parallel workflow for double and single blind within Editorial Manager, and I have moderate hope that it will work out. If it does, maybe more journals should just give authors the option, instead of only doing one or the other?

    Reply

    1. Ah, thanks. I think offering the choice sounds good, but the reason I looked at the double-blind literature is because i think our opinions should be informed by evidence.

      Have you had enough submissions to be able to look at who prefers which system?

      Reply

      1. As you outline above, the evidence is a bit equivocal for the benefits of double blind. That may be because it’s only beneficial for some author groups, or because its positive effects are eroded when reviewers correctly guess the authors’ identities. Or, it may not have any positive effects. Since running double blind review takes more effort than single blind, we opted to let authors choose how they wanted their paper to be reviewed.

        Our sample size of double blind papers is too small still to say very much, but on a quick scan the list contains papers from all over the world.

        Tim

  4. Pingback: Friday links: when you stop believing your own results, and more | Dynamic Ecology
  5. Pingback: Weekend reads: Data sharing fees block access; Machiavellianism and gossip in science; "power pose" redux - Retraction Watch at Retraction Watch

  6. Thank you for the excellent post. Most of your examples were from the biomedical fields for which author blinding may be more practical than for some ecology fields. I suspect that blinding might be harder for place-based, ecological places wherein people are often associated with places. However, most authors cite themselves, such as, “we previously studied aspect A of this problem, and here we investigate aspect B.” Self-citations are a dead giveaway.

    Another article of interest from the ecological literature is Emily Darling’s 2015 commentary advocating mandatory double-blind submissions (Use of double-blind peer review to increase author diversity, Conservation Biology, http://dx.doi.org/10.1111/cobi.12333, paywalled, but can be found). She argued that although the evidence was thin (vanishing thin in my read), double-blind reviews should be required to offset perceptions of gender bias. The journal has adopted the policy, apparently in response to surveys or votes among the Society for Conservation Biology. The post by Frith Jarrad, Managing Editor, indicated a hope that the move may increase submissions from underrepresented groups and early-career professionals.

    Journals compete and Conservation Biology has essentially made a business decision to differentiate itself from competing journals with something unique: double-blind reviews. It is perfectly appropriate to market themselves as being friendly to underrepresented groups and early-career professionals. Further, the Society owns the journal, and it is completely appropriate for journal policies to reflect the values and preferences of its members. However, member preferences, marketing and business choices should not be confused with evidence.

    The hassle factor of double-blind review policies hasn’t been discussed much. This would all fall on the authors to do, and editorial staff to check that all geographic references and any potentially identifying material is stripped from the review submissions, and then restored in the final. Have to redact the de rigueur “Figure 1, Study area” from submissions? Other new policies call for data sharing in open repositories and submitted for review along with the articles. Good metadata is precise about what, where, and whom, and may be required to be hosted on an institutional server. The “new normal” in data curation and openness is good stuff but expensive and tedious hassle for authors as it is; requiring masking and then unmasking would double the hassle and increase costs to authors and the journals to enforce it. Like ecology itself, the ecological publishing landscape is diverse, and I commend Conservation Biology for carving out a new niche. It does not follow that the entire field has to follow suite.

    Reply
  7. Pingback: Chris Mebane – cmebane
  8. Pingback: Methods in Ecology & Evolution Blog | brouwern
  9. Pingback: Professors take part in journal’s test for peer review bias in major international study –

Leave a comment