Content of review 1, reviewed on April 14, 2015

This timely article by Catarina Ferreira et al covers the history of peer review, surveys the peer review practices of top ecology and evolutionary biology journals, argues the well-documented flaws of the current peer review system are a sign that it has failed to evolve to a changing environment, and proposes a series of “large, multidirectional” changes to fix peer review. I recommend a read for anyone interested in the current state and future of peer review. Of particular interest are the tables detailing the varied review practices and typical timeframes of the 38 surveyed journals, given that journals are usually quite reluctant to publicly share this data.

The most interesting part for discussion purposes is the proposed solutions:

We propose three main directional changes in the current peer review system that we expect will rapidly mitigate the structural flaws identified herein. These include: (i) making peer review mandatory and eventually coupling it with paid reviews; (ii) standardizing the review criteria and guidelines for review by field of expertise; and (iii) creating a Global Peer Review Platform, responsible for the centralization of the process (including the aforementioned standardization).

Discussion on each of the three proposed solutions follows.

(i) making peer review mandatory and eventually coupling it with paid reviews;

There are potential issues with both aspects of this solution (mandatory peer review and paid peer review) that deserved more consideration in the article. PeerJ is given as an example of a mandatory review system (with a very low bar of one review per year), but it is an enormous leap from that to a centralized mandatory review system across all journals to cover the four-million-plus peer reviews conducted each year. What kind of organization could possibly be in a position to set and enforce such a system?

The idea of paying peer reviewers merits further academic discussion also. Anecdotally it is common to hear researchers share concerns that paying for peer review may distort the system, and we have heard numerous journals question whether the economics would ever allow for more than just a ‘token’ payment. Just recently we saw a significant backlash from the Nature Scientific Reports’ pilot with Rubriq [1, 2, 3]. A lot of that backlash was unwarranted, but some of those objections will need to be answered before widespread adoption of pay-for-peer-review.

In addition to Rubriq, it will be worth watching how the recently-launched open access journal Collabra gets on with their pay-for-peer-review system [4].

(ii) standardizing the review criteria and guidelines for review by field of expertise;

Standardization of peer review is an admirable goal (and one that I support), but it remains to be seen how viable it is. As the authors mention, different publishers have different goals for peer review (eg Nature vs PLOS), and a standardized peer review system will need to cater to all of these to ensure uptake.

Getting the entire scientific community to agree upon a standard is another non-trivial hurdle. I suspect we’ll ultimately see a ‘base’ standard emerge -- not from compulsion, but rather from piecemeal adoption of a superior standard -- where journals/publishers add any additional questions to satisfy the non-standard requirements of the journal or field.

One under-stated advantage of a standardized review criteria is the potential for research quality metrics that are standard across all fields, immediate (no need to wait years for the citation count to form), and direct (not a proxy for quality like the publishing journal’s impact factor). This might be the solution to the “publishing by piecemeal” issue, which of course is putting a lot of unnecessary strain on the peer review system.

(iii) creating a Global Peer Review Platform, responsible for the centralization of the process.

I agree to an extent, and this is similar to the impetus behind Publons (I’m a cofounder). A platform that provided a curated database of reviewers and publisher-independent verification of the quality of a journal’s review process (as described by the authors) would solve both the problems of peer review fraud and predatory journals. Where I disagree is in the approach.

The authors propose decoupling peer review from publishers, and after mentioning existing ventures Rubriq and Peerage of Science, ultimately suggest the establishment of an independent, global regulatory authority to handle all peer review. (The International Union for Conservation of Nature is mentioned as an example of a worldwide professional organization to emulate, but ORCID is a more conceivable model.) For this authority to have any effect, it would need buy in from all the major publishers -- and convincing all the major publishers to cede control of peer review would be a mammoth task. As the authors note, "Currently, most of the players in the peer-review game have little incentive to change their behaviour drastically in the short term, meaning that relatively few will willingly transition to a mandatory and standardized system."

That all sounds very force-fed, which is not typically a recipe for a well-functioning system. This approach also puts itself directly at odds with publishers, further harming its chances for even getting off the ground. I would argue that a peer review system that works for all stakeholders and that emerges out of superiority as opposed to compulsion is both more likely to succeed, and more likely to produce a better outcome for science.

Finally, the authors finish the article with a novel suggestion for future research in this area:

Specifically, evolutionary game theory provides an appropriate theoretical framework to discover future directions for the peer-review system to evolve. Important insights could arise by modelling scenarios of non-cooperative versus cooperative games and empirically testing predictions arising from these models.

The incentive structure in this industry would seem pretty hard to model to me, but any such attempts would definitely make for interesting reading!

Note: I am a cofounder of Publons.

Source

    © 2015 the Reviewer (CC BY 4.0).

References

    Catarina, F., Guillaume, B., M., B. A., Hance, E. E., Christine, T., Cayla, A., Adrian, B., R., B. M., Kevin, C., Adrian, F., J., H. T., Kristen, L., Jessica, L., Josee-Anne, O., L., P. M. J., Jason, R., Jacob, S., Cristen, W., Morgan, W., L., M. D. 2016. The evolution of peer review as a basis for scientific publication: directional selection towards a robust discipline?. Biological Reviews.