This manuscript is a herculean effort and enjoyable read. I learned lots, (and I think the paper will be a good resource for anybody interested in the field of peer review) which for me is usually a good sign of a paper’s worth.

The authors report on many aspects of peer review and devote considerable attention to some challenges in the field and the enormous innovation the field is witnessing.

I think the paper can be improved: 1. It is missing a Methods section. It was unclear to me whether the authors conducted a systematic review or whether they used a snowballing technique (starting with seed articles) to identify the content discussed in the paper? Did the authors search electronic databases (and if so which ones?) What were their search strategies and/or did they rely on their own file drawers? Are all the peer review innovations/systems/approaches identified by the authors discussed or did they only discuss some (i.e., was a filter applied?)? With a focus on reproducibility I think the authors need to document their methods.

1. I think the authors missed an important opportunity to discuss more deeply the need for evidence with all the current and emerging peer review systems (the authors reference Rennie 20161 in their conclusions. I think the evidence argument needs to be made more strongly in the body of the paper). I do not think the paper is strong enough regarding the large swaths of the peer review processes (current and innovations) for which there is no evidence2 and it is difficult to gain access to peer reviews to better understand their processes and effectiveness – open the black box of peer review3.

2. There is limited data to inform us about several of the current peer review systems and innovations. In clinical medicine new drugs do not simply enter the market. They need undergo a rigorous series of evaluations, typically randomized trials prior to approval. Shouldn’t we expect something similar for peer review in the marketplace? It seems to me that any peer review process/innovation in development or released should have an evaluation (experimental, whenever possible) component integrated into it. Without evaluation we will miss the opportunity to generate data as to the effectiveness of the different peer review systems and processes. Research is central to getting a better understanding of peer review. It might useful for the authors to mention the existence of some groups/outlets committed to such research – PEERE (http://www.peere.org/) and the International Congress on Peer Review and Scientific Publication (http://www.peerreviewcongress.org/index.html). There is also a new journal committed to publishing peer review research (https://researchintegrityjournal.biomedcentral.com/).

3. In section 1.3 of the paper the authors could add (or replace) Jefferson 20024 with Bruce5. The Bruce paper is also important for two additional reasons not adequately discussed in the paper: how to measure peer review and optimal designs for assessing the effects of peer review.

Concerning measurement of peer review, there is accumulating evidence that there is little agreement as to how best to measure it. Unlike clinical medicine where there is a growing recognition of the need for core outcome set assessments (http://www.comet-initiative.org/) across all studies within specific content areas (e.g., atopic eczema/dermatitis clinical trials) we have not yet developed such an approach for peer review. Without a core outcome set for measuring peer review it will continue to be difficult to know what components of peer review researchers are trying to measure.

Similarly, without a core outcome set it will be difficult to aggregate estimates of peer review across studies (i.e., to do meaningful systematic reviews on peer review).

Concerning the second point – there is little agreement as to an optimal design to evaluate the effectiveness of peer review. This is a critical issue to remedy in any effort to assess the effectiveness of peer review.

1. The paper assumes (at least that’s how I’ve interpreted it – the paper is silent on this issue) that peer reviewers are all similarly proficient in peer reviewing. There is little training for peer reviewers (new efforts by some organizations such as Publons Academy are trying to remedy this). I started my peer-reviewing career without any training, as did many of my colleagues. If we do not train peer reviewers to a minimum globally accepted standard we will fail to make peer review better.

2. Peer review does not function in a vacuum. The larger ecosystem includes other players’ most notably scientific editors. There is little discussion in the paper about this relationship and its potential (dys)function6.

3. In section 2.2.1 you could also add at least one journal has an annual prize for peer reviewing (Journal of Clinical Epidemiology – JCE Reviewer Award: http://www.jclinepi.com/)

4. In the competing interests section of the paper it indicates that the first author works at ScienceOpen although the affiliation given in the paper is Imperial College London. Is this a joint appointment? Clarification is needed. A similar clarification is required for TRH.

Is the topic of the review discussed comprehensively in the context of the current literature?

Yes

Are all factual statements correct and adequately supported by citations?

Yes

Is the review written in accessible language?

Yes

Are the conclusions drawn appropriate in the context of the current research literature?

Yes

References

1. Rennie D: Let’s make peer review scientific. Nature. 2016; 535 (7610): 31-33 Publisher Full Text
2. Galipeau J, Moher D, Campbell C, Hendry P, Cameron DW, Palepu A, Hébert PC: A systematic review highlights a knowledge gap regarding the effectiveness of health-related training programs in journalology.J Clin Epidemiol. 2015; 68 (3): 257-65 PubMed Abstract | Publisher Full Text
3. Lee CJ, Moher D: Promote scientific integrity via journal peer review data.Science. 2017; 357 (6348): 256-257 PubMed Abstract | Publisher Full Text
4. Jefferson T, Wager E, Davidoff F: Measuring the quality of editorial peer review.JAMA. 2002; 287 (21): 2786-90 PubMed Abstract
5. Bruce R, Chauvin A, Trinquart L, Ravaud P, Boutron I: Impact of interventions to improve the quality of peer review of biomedical journals: a systematic review and meta-analysis.BMC Med. 2016; 14 (1): 85 PubMed Abstract | Publisher Full Text
6. Chauvin A, Ravaud P, Baron G, Barnes C, Boutron I: The most important tasks for peer reviewers evaluating a randomized controlled trial are not congruent with the tasks most often requested by journal editors.BMC Med. 2015; 13: 158 PubMed Abstract | Publisher Full Text Competing Interests: I’m a member of the International Congress on Peer Review and Scientific Publication advisory committee; content expert for the Publons Academy; a policy advisory board member for the Journal of Clinical Epidemiology; and on the editorial board of Research Integrity and Peer Review.