Lessons learned from open peer review: a publisher’s perspective

Author: Rachel Burley

Vice President, Publishing

On August 29^th ASAPbio published an open letter encouraging more openness in peer review.  As a publisher with a long track-record of open peer review, BMC was delighted to be one of the signatories. 

In 2001, BioMed Central was the first publisher to openly post named peer reviewer reports alongside published articles as part of a ‘pre-publication history’ for all medical journals in the BMC series.

With growing interest in openness and transparency, and as more journals consider open peer review, I wanted to share some of our experiences over the last 17 years.

The BMC Group publishes over 500 journals and of those, 70 operate fully open peer review. Forty-four of the BMC series medical journals use only open peer review, with no opt out for authors or reviewers. These journals publish over 8,000 papers a year, demonstrating that at least in the medical disciplines, there is clearly a community willingness to undertake open peer review. 

Forty-four of the BMC series medical journals use only open peer review, with no opt out for authors or reviewers. These journals publish over 8,000 papers a year, demonstrating that at least in the medical disciplines, there is clearly a community willingness to undertake open peer review. 

So what have we learned about open peer review? 

There have been relatively few large-scale studies looking at the impact of open peer review, but it seems to me that the key issues for journals and publishers considering introducing open peer review fall into the following categories:

1. Review uptake
2. Quality of the review
3. The editorial decision
4. Value to readers

ASAPbio’s FAQs page tells part of the story, where publishers already operating some form of open peer review have added their observations or studies.

Here, we share the lessons learned from open peer review at BMC.

1. Reviewer uptake

   For the BMC series, there is some evidence to suggest that reviewer uptake is lower due to the open peer review policy of the journals. A very small number of reviewers actually cite open peer review as the reason for declining requests, but editors for some journals tell us that it can take longer to recruit reviewers for this reason. A study conducted by the BMJ found a similar trend, where asking reviewers to consent to being identified increased the likelihood of reviewers declining to review.

   It is likely that the BMC open peer review (OPR) policy, which includes reviewers’ identities, is an added factor in lower reviewer uptake, whereas anonymous OPR may have a lower proportion of declined reviewer invitations. We haven’t yet done any research to conclusively prove this.  However, we do have data comparing fully OPR with double and single blind peer review.

   An analysis of data from 500 BMC and SpringerOpen journals between June 2001 to July 2015 found that on average, about half of invited reviewers agreed to review.  Grouped by peer review model, it was 60% for double-blind peer review, 53% for single blind and 42% for open peer review. In other words, more reviewers needed to be invited for OPR journals than for either of the other two models (6 invitations for OPR journals compared to 4 for double blind). 

   There are doubtlessly a number of reasons why more reviewers decline invitations to OPR journals, but the top two are likely to be:

   1.1.      It takes more time to conduct an open peer review: 182 minutes compared to 157 minutes according to this randomized controlled trial.  It is well known that the burden on the peer review community is increasing as research output increases and so anything that adds time to the task is likely to be a disincentive, even if reviewers recognize the value of openness.

   1.2.      Fear of reprisals: for example, reviewers may be reluctant to express their true opinions on a poor quality paper coming from the laboratory of a senior researcher.

   Some reviewers strongly support open peer review and tell us that it is in the interests of the research community to strive for openness and transparency. I believe this is the ultimate goal for all journals, but that it will be a journey rather than an event as the practice becomes more common and researchers are more comfortable with openly critiquing their peers.  By promoting the benefits of OPR we will encourage reviewers to support the journals that use it and create an environment in which more journals are willing to ‘make the leap’.

   Some reviewers strongly support open peer review and tell us that it is in the interests of the research community to strive for openness and transparency. I believe this is the ultimate goal for all journals, but that it will be a journey rather than an event as the practice becomes more common and researchers are more comfortable with openly critiquing their peers. 

   The benefits for open reviewers include recognition for the time and effort they have spent assessing the manuscript.  Credit can be given through systems such as Publons, a free service for reviewers that tracks, verifies and showcases peer review activity across all journals.   Reviewers can use their Publons profile as evidence of their peer review contributions and expertise.

2. How open peer review affects the quality of reports

   There is a school of thought that publishing reviewer reports will encourage better-quality, more constructive comments. But is that actually true?

   A study in BMJ Open in 2015 found that the quality of peer review reports was slightly higher in BMC Infectious Diseases (OPR) compared with BMC Microbiology (single-blind). However, no effect was found in the Journal of Inflammation when it operated open vs single-blind review. Anecdotally, reviewers have told us that they take longer to write a review that they know it will be published and where they will be identified.  This might not always lead to a higher quality report and may be a factor of something else, such as reviewers who do not have English as their first language taking more time to write the reports knowing they will not be copy-edited.

3. The editorial decision

   The study in BMJ Open in 2015 showed that reviewer recommendations were similar in BMC Infectious Diseases and BMC Microbiology, suggesting no difference between open and single blind peer review.  However, in the Journal of Inflammation, it was found that reviewers were more likely to recommend acceptance under open peer review compared with single blind.

4. Reader perception

   For the BMC journals, the reviewer reports receive an average 1.5% of the page views for the journals (including all content types and not just research articles). 

   Since January of last year BMC journals had an average of 25,000 page views per month to open peer review reports.  The numbers are growing though - in July/August this year, average page views were more than 50,000 per month. 

Our long experience of open review has shown us that a pragmatic balance has to be struck between providing an efficient and timely service to authors and the ideal of full transparency, where the mandatory requirement of publishing reviewers’ details may have unintended consequences in discouraging and deterring some researchers from becoming reviewers.

#PeerReviewWeek18 gives us the opportunity to share our views and experiences of peer review, and to recognise the vital role of peer reviewers in building the scientific record. We want to thank the many peer reviewers who have dedicated their time to our journals, and welcome new ones into the reviewer community.