Content of review 1, reviewed on December 20, 2020

The manuscript investigates the causal effect of the prestige of journals on the citation rates of articles. Findings suggest that high-impact journals concur to a certain extent to the higher citation rates of articles published therein.
I find the research question very interesting, the methodological approach appropriate, and the presentation of the arguments clear and straightforward. In the following, I propose a few considerations which I hope the authors find helpful to improve their work.

  1. The incipit of the paper “Articles in high-impact journals are, on average, more frequently cited” might cause a certain disagreement on the knowledgeable reader, who is aware that in reality a very small share of articles contributes to the high impact of journals. Referring to averages of extremely high-skewed distributions seems awkward. Formally, I suggest to change the incipit. In the substance though, one might wonder: if high-impact journals contribute to the citation rates of articles published therein, why would a large majority of them show no high citation rates? It would be interesting to partition the dataset of published papers in two groups: one consisting of papers with citation rates below the impact of the journal, and the other consisting of the complement. Applying the same methodology to both groups would provide further insight on the effects of the prestige of journals,

  2. While the heart of the paper is well written and exhaustive, its introduction and conclusions might benefit from a deeper analysis and discussion.

Introduction:
According to the authors, the main implication of the results concerns the bibliometric evaluation of publications. The review of the literature on the appropriateness of the journal impact factor alone or in combination with citations could be improved. Here are few references to that purpose:
Levitt, J. M., & Thelwall, M. (2011). A combined bibliometric indicator to predict article impact. Information Processing and Management, 47(2), 300-308.
Bornmann, L., Leydesdorff, L., & Wang, J. (2014). How to improve the prediction based on citation impact percentiles for years shortly after the publication date? Journal of Informetrics, 8(1), 175-180.
Stern, D.I. (2014). High-ranked social science journal articles can be identified from early citation information. PLoS One, 9(11), 1-11.
Abramo, G., D’Angelo, C.A. (2016). Refrain from adopting the combination of citation and journal metrics to grade publications, as used in the Italian national research assessment exercise (VQR 2011-2014). Scientometrics, 109(3), 2053-2065.
Abramo, G., D’Angelo, C.A., & Felici, G. (2019). Predicting long-term publication impact through a combination of early citations and journal impact factor. Journal of Informetrics, 13(1), 32-49.

Conclusions
Adopting a broader view of the final aim of research, the mission of a scientist is not just having a paper published, rather having the new knowledge embedded in the publication used, i.e. cited. Because in general all papers have the same opportunities to be published in high-impact journals, if the latter represent a distribution channel (to reach the final users) more effective than low-impact ones, then why not to account for their contribution to citation rates when evaluating research?
In the market for products, if a product achieves higher sales because it is sold in 5th Avenue in Manhattan rather than in the suburbs of NYC, should one detract from its value the contribution of the sale point?
Seemingly, a high-impact journal reaches a higher number of potential citers, and possibly achieves higher citation rates, i.e. higher impact on future advancement of knowledge.
Consequently, should we care about possible effects of high impact journals when we evaluate research?

  1. The authors cogently observe that the preprint publication in ArXiv is not necessarily the same as the published version in journal J. It is likely that the peer-review process has added value to the original version of the manuscript. It is also reasonable that high-impact journals’ reviewers add more value than low-impact journals’ ones. The question is whether post-publication citations are affected also by this possible added value. I invite the authors to anticipate this point when they present the model, and acknowledge it as a possible limit of the model.

  2. The authors conduct the analysis at field level, but they do not mention why. Here is a good reason for doing it. The question in fact is whether a field-level analysis would be more correct than an undifferentiated one. The authors cogently observe that preprint duration and post publication citation time window affect the causal effect analysis. To avoid the bias they model the full temporal dynamics. There is another factor though which can distort the analysis, which is the different time distributions of citations across research fields. All other equals, citations of publications in mathematics (for example) accrue more slowly than in other fields, showing a significant inertia in the early stage. As a consequence, the size of the journal citation multiplier would be higher than in other fields, all others equal.

Minor Issues
- At times, Figures are positioned before they are referred to in the text, which is unhandy.
- Typo: by the journal in which it published
- Fig. 2: change “citations” in the y-axis with “cumulative citations”
- “The thin shaded lines represent samples from the posterior predictive distribution of our model”: I cannot see the shaded lines.
- I would not be so sure that authors' reputation (X5) does not affect journal peer review.
- “We provided clear evidence that high-impact journals are highly cited because of two effects”. Reframe, as it is articles that are cited.

Source

    © 2020 the Reviewer.

Content of review 2, reviewed on January 24, 2021

I am satisfied with the reply and integrations made by the authors

Source

    © 2021 the Reviewer.

References

    A., T. V. 2021. Inferring the causal effect of journals on citations. Quantitative Science Studies.