Go to publons.com

Peer review incentives: a simple idea to encourage fast and effective peer review

A version of this article by Publons cofounder Daniel Johnston first appeared in the European Association of Science Editors' journal, European Science Editing (August 2015 edition). It can be cited as: Johnston D. Peer review incentives: a simple idea to encourage fast and effective peer review. European Science Editing 2015; 41 (3): 70-71.

The inherent conflict of peer review is that we expect researchers to put aside their own research priorities to selflessly help with the research of strangers. Picture it -- a typical Friday afternoon. A researcher sits down at her desk, pushes out all thoughts of what she might do with a hint of spare time this weekend, and starts writing up the latest results from the lab. An email notification interrupting her train of thought warrants a groan. Another request to review; the third this week. What takes priority? Finishing her own research article so she can at least tell her employer it has been submitted before her performance evaluation next week, or anonymously helping out with someone else’s work? When we do accept review requests, it is often reluctantly, and only out of a combination of duty, guilt, and hope for reciprocation. Peer review is a chore; a distraction from more career-relevant activities.

This situation leads to predictable problems. Rejection of review invitations is the norm. Editors have to find and screen ten potential reviewers just to secure two peer reviews. Reviewers lack the motivation to return review assignments promptly, and receive the same attribution -- almost none -- whether their reviewing is comprehensive or careless. Review quality is so varied that research on peer review struggles to find any evidence of its effectiveness.[1] Editors are torn between burdening their favourite reviewers with excessive review requests, or gambling on unknown and potentially unsuitable reviewers. Flawed work slips through and leads to embarrassing article retractions. Fabricated reviewers go undetected by busy editors, and we end up reading about it in the New York Times.[2]

Opinions on the state of peer review vary -- former BMJ editor Richard Smith recently declared it time to “slay the sacred cow” of peer review[3] -- but most agree it can be improved. The 2014 Nature Publishing Group Author Insights survey found 77% of researchers agree (or strongly agree) that “traditional peer review processes could be made more efficient”. Seventy percent agreed with the statement “I am frequently frustrated by the length of time the process of peer review takes”.[4] And for good reason: a single peer review takes about four hours, but organising those two reviews takes on average four months or more. We in the industry have become used to this delay to publication, but in the age of the internet it is scarcely believable.

The primary reason for these issues is the absence of incentives for reviewers. Some publishers offer discounts, perks, and the odd certificate, but this clearly is not enough. Paying for peer review is an interesting option, but the real currency of academia is reputation. To motivate peer reviewers we need to provide a way for peer review to improve their academic reputation.

In 2013 Andrew Preston and I founded Publons to improve the speed and quality of peer review by helping researchers to build their academic reputation through their peer review activity. We work with peer reviewers, editors, and publishers to motivate reviewers by giving credit for peer review. For peer reviewers, Publons provides a way to get credit for their contributions (without breaking reviewer anonymity) in a format they can include in job and funding applications.[5] Reviewers have their review activity verified by Publons simply by forwarding the “thank you for reviewing” emails from journals to reviews@publons.com.

The hypothesis is that reviewers that get official recognition for their work are more willing to accept review requests, more willing to prioritise time to do the review quicker, and more likely to do a comprehensive review. The aim is to give reviewers a reason to put aside their own research to review the work of others; not just because journals ask them to, but because they want to contribute to and influence others’ work, and demonstrate their expertise to a wider audience. Most of the problems with peer review -- that reviewers are often slow, careless, and unnecessarily abrasive -- are typical characteristics of any task where people have no incentives to do it, nor to do it well. Give reviewers an incentive to take pride in their reviewing, and they will reciprocate with prompt and high-quality peer review.

The evidence in support of that hypothesis is growing fast: as of 1 June, 38,000 peer reviewers are getting credit for 110,000 reviews across 7,800 journals. Peer reviewers appreciate being able to easily keep a verified record of their peer review contributions to include in performance evaluations, job applications, and even applications for funding. Many use their Publons profile to publish a select few of their reviews (only once the manuscript has been published, and where the journal allows it) to showcase their best work. They also enjoy seeing how their review activity compares to their colleagues and the global average in a range of review metrics, like average word count per review, review frequency, and the range of journals the reviewer has reviewed for.

The best way an editor can make use of Publons is to let their reviewers know all about Publons. Reviewers getting credit means motivated and engaged reviewers; it means less time wasted on finding and screening reviewers that ultimately just say no; less chasing up reviewers; and less having to call in a third reviewer because the first two offered insufficient feedback. Most importantly, it means better quality reviews and faster review and publication times. Our data show that acceptance of review requests increases by 30% once a researcher has signed up to Publons (reducing the time editors have to spend on finding and inviting additional reviewers), and reviewers report they do better quality reviews too.[6] As the popularity of Publons increases, the incentives are structured so that reviewers strive to deliver top quality reviews in order to demonstrate their expertise -- and to be invited to review again.

The benefits to editors and journals of reviewer recognition is made evident by recent partnership announcements with leading publishers and journals such as Wiley, SAGE, PeerJ, eLife, GigaScience, and the American Society for Microbiology -- all signing up to give their peer reviewers credit through the Publons platform. Publons has developed integration options to work with any editorial management system, with minimal implementation costs.

Editors can make more direct use of Publons too. Reviewer profiles give editors a much more comprehensive picture of potential reviewers and their review workload across all journals, while the fast-growing database of motivated peer reviewers has enormous potential for editors to find, screen, and contact the best reviewers all from one place. There are a range of editor tools to help. The reviewer search tool takes manuscript metadata, searches for keywords extracted from review data, and returns suggested reviewers with a contact option. The screening tool checks a supplied email address against those in our database, and returns records of review and publishing history to help editors combat reviewer fraud. The reviewer contact feature makes it possible for approved editors to browse and make contact with active, validated reviewers through the Publons platform.



Peer review is at the heart of science but has been largely ignored, predictably leading to slow and poor quality peer review. We have demanded great work of peer reviewers without offering any reward. It is now possible with Publons for journals and editors to formally recognise reviewers, to enjoy the benefits of motivated reviewers, and to make use of tools to help editors improve the efficiency of their peer review process. The core idea is simple: give better reward to get better results out of peer review. A growing number of reviewers, editors, and journals are acknowledging the advantages of giving credit for peer review, leading to higher quality peer review and a faster publishing process -- a superior outcome for all. The more reviewers we can give credit to the better.


References

1 Hopewell S, Collins G, Boutron I, et al. Impact of peer review on reports of randomised trials published in open peer review journals: retrospective before and after study. BMJ 2014;348:g4145.

2 Fountain H. Science Journal Pulls 60 Papers in Peer-Review Fraud. Available at: http://www.nytimes.com/2014/07/11/science/science-journal-pulls-60-papers-in-peer-review-fraud.html (accessed 10 June 2015).

3 Jump P. Slay peer review ‘sacred cow’, says former BMJ chief. Available at: https://www.timeshighereducation.co.uk/news/slay-peer-review-sacred-cow-says-former-bmj-chief/2019812.article (accessed 10 June 2015).

4 Nature Publishing Group. Author Insights 2014. figshare. DOI: http://dx.doi.org/10.6084/m9.figshare.1204999 (accessed 10 June 2015).

5 Elisabeth M. Bik Publons reviewer profile. Available at: https://publons.com/author/250457/elisabeth-m-bik (accessed 10 June 2015).

6 Roberts S. [@sr320]. (2014, September 19). “It was a little easier spending AM on manuscript review knowing even if comments are ignored - I still get some credit, thanks @Publons! … I also believe I am much more constructive and spend more time explaining issues and offering solutions...”. Available at: https://twitter.com/sr320/status/513052347409235968 and https://twitter.com/sr320/status/513051648076169217 (accessed 10 June 2015).

comments powered by Disqus

Subscribe to our mailing list

* indicates required