Abstract

Authors who publish in American Economic Review have career paths confined to a few prestigious institutions, and they mostly have exceptional past publication performance. In this paper, we show that authors who have better performance in their past careers receive more citations for their current American Economic Review publications. An opportunistic editor can exploit this relation to substantially improve the citation performance of the journal. The opportunistic editor with perfect foresight can improve the average number of citations by 131.6 percent when he selects the quarter of the articles. The over-representation of authors with certain characteristics is more intense when there is no perfect foresight. The opportunistic editor who relies on the predicted citation performance of articles to select a quarter of the articles increases the ratio of authors who works at the top ten institutions from 34.0 percent to 58.7 percent.


Authors

Tolga Yuret

Publons users who've claimed - I am an author

No Publons users have claimed this paper.

Contributors on Publons
  • 2 reviewers
  • pre-publication peer review (FINAL ROUND)
    Decision Letter
    2022/01/15

    15-Jan-2022

    Dear Dr. Yuret:

    It is a pleasure to accept your manuscript entitled "Predicting the impact of American Economic Review articles by author characteristics" for publication in Quantitative Science Studies. The comments of the reviewers who reviewed your manuscript are included at the foot of this letter. Reviewers 1 and 3 both recommend acceptance of your manuscript. Reviewer 2 was not available to review your revised manuscript.

    I would like to request you to prepare the final version of your manuscript using the checklist available at https://tinyurl.com/qsschecklist. In the final version of your manuscript, please try to address the remaining comments of reviewers 1 and 3. In addition, I would like to ask you to make some improvements to the figures presented in your manuscript. Do not include non-informative decimals in the numbers reported along the vertical axes (e.g., you don't need any decimals in Figure 1 and you need only one decimal in Figures 2 and 3). Also, consider removing the box around each of the figures.

    Please also sign the publication agreement, which can be downloaded from https://tinyurl.com/qssagreement. The final version of your manuscript, along with the completed checklist and the signed publication agreement, can be returned to qss@issi-society.org.

    Thank you for your contribution. On behalf of the Editors of Quantitative Science Studies, I look forward to your continued contributions to the journal.

    Best wishes,
    Dr. Ludo Waltman
    Editor, Quantitative Science Studies
    qss@issi-society.org

    Reviewers' Comments to Author:

    Reviewer: 1

    Comments to the Author
    The author has very thoroughly and comprehensively addressed the issues raised by this referee and thereby substantially improved the manuscripts in various places. The same goes even moreso for the problems I overlooked but the other referees pinpointed, but I'll refrain from remarking more on this. I just noticed the three referees were complementing each other nicely in their assessment of different issues that they saw as needing further work.
    I therefore recommend to accept this manuscript for publication.

    One last little remark:
    P. 4: "The study finds that men get more citations than women however the effect is insignificant." If the effect is insignificant, that means there is no (stat. sign.) difference, i.e. men don't get (stat.) more citations. Their value might be higher but one would hardly expect the values to be exactly equal, so this means nothing by itself.

    Reviewer: 3

    Comments to the Author
    I appreciate the efforts made by the author in revising their paper. Significant improvements have been made. I consider the paper acceptable for publication in Quantitative Science Studies.

    I still have two comments that the author may want to take into consideration.

    First, the author has replaced the Shanghai Ranking by the QS ranking, but a convincing justification for using either of these rankings is still missing. I wonder how sensitive the results presented by the author are to the choice of the ranking. To what extent do the results change if the Shanghai Ranking is used instead of the QS ranking? It would be helpful if the author could add a short paragraph somewhere in the paper discussing the sensitivity of the results to the choice of the ranking (i.e., QS vs. Shanghai). There is no need to report detailed results for both rankings, but a brief discussion of the effect of the choice of the ranking would be useful.

    Second, in the presentation of the results of the regression analysis, the author discusses only the statistical significance of the effects and their direction (positive or negative). A discussion of the effect size (how strong is the effect?) is missing. It is important to also discuss effect size, because effects that are statistically significant but of very small size are of little practical interest. (Apologies, I should have mentioned this point in my first review of the paper.)

    Decision letter by
    Cite this decision letter
    Reviewer report
    2022/01/15

    I appreciate the efforts made by the author in revising their paper. Significant improvements have been made. I consider the paper acceptable for publication in Quantitative Science Studies.

    I still have two comments that the author may want to take into consideration.

    First, the author has replaced the Shanghai Ranking by the QS ranking, but a convincing justification for using either of these rankings is still missing. I wonder how sensitive the results presented by the author are to the choice of the ranking. To what extent do the results change if the Shanghai Ranking is used instead of the QS ranking? It would be helpful if the author could add a short paragraph somewhere in the paper discussing the sensitivity of the results to the choice of the ranking (i.e., QS vs. Shanghai). There is no need to report detailed results for both rankings, but a brief discussion of the effect of the choice of the ranking would be useful.

    Second, in the presentation of the results of the regression analysis, the author discusses only the statistical significance of the effects and their direction (positive or negative). A discussion of the effect size (how strong is the effect?) is missing. It is important to also discuss effect size, because effects that are statistically significant but of very small size are of little practical interest. (Apologies, I should have mentioned this point in my first review of the paper.)

    Reviewed by
    Cite this review
    Reviewer report
    2021/12/06

    The author has very thoroughly and comprehensively addressed the issues raised by this referee and thereby substantially improved the manuscripts in various places. The same goes even moreso for the problems I overlooked but the other referees pinpointed, but I'll refrain from remarking more on this. I just noticed the three referees were complementing each other nicely in their assessment of different issues that they saw as needing further work.
    I therefore recommend to accept this manuscript for publication.

    One last little remark:
    P. 4: "The study finds that men get more citations than women however the effect is insignificant." If the effect is insignificant, that means there is no (stat. sign.) difference, i.e. men don't get (stat.) more citations. Their value might be higher but one would hardly expect the values to be exactly equal, so this means nothing by itself.

    Reviewed by
    Cite this review
    Author Response
    2021/12/02

    Dear Referees,
    Thank you for your valuable comments. I used your comments to write a new and better version of the paper. I report how each of your comments changes the paper. I took the liberty to itemize your comments and number them. First I state your comments, then I state the change and/or comment beginning with three asterisk remarks (***).
    The paragraph, page, table, and figure numbers refer to the new version if otherwise stated.

    [1.1] The present study is concerned with author factors that influence the citation impact of journal articles. Among these, primarily the academic background, i.e. institutional affiliation, is investigated. The study is limited to a single journal, albeit one of the most important ones in its field. This limitation is justified by the effort of data collection, which puts analysis of larger samples beyond the means of single authors or small groups. Moreover, this limitation is somewhat ameliorated by various comparisons to statistics for other journals taken from the literature and from a sample of lower ranked journals.
    (***) Referee #1 summarizes the paper.

    [1.2] I believe that the abstract should be more balanced to the general results and less to the simulation exercise results.
    (***) I agree with Referee #1. I change the abstract to include more about the regression results and less about the simulation results. The new abstract (on page 1) is as follows.
    “Authors who publish in American Economic Review (AER) have career paths confined to a few prestigious institutions, and they mostly have exceptional past publication performance. In this paper, we show that authors who are educated and work in the top ten institutions and have better past publication performance receive more citations for their current AER publications. Authors who have published in the top economic theory journals receive fewer citations even after controlling for the subfield of their AER article. The gender of the authors, years of post-PhD experience, and the location of the affiliated institution do not have any significant effect on the citation performance. An opportunistic editor can exploit the factors that are related to citation performance to substantially improve the citation performance of the journal. Such an opportunistic behavior increases the over-representation of authors with certain characteristics. For example, the opportunistic editor who uses the predicted citation performance of articles to select a quarter of the articles increases the ratio of authors who works at the top ten institutions from 30.8 percent to 52.0 percent.”

    [1.3] The presented literature review is adequate in that it selectively reports on prior work of relevance without getting lost in details or excessively trying to review every single paper published in the rather large topic.

    (***) Referee #1 comments about the literature part of the paper. I have improved the literature part of the paper by replacing older papers with more recent publications (please see comment 3.4), adding new literature about the editorial misconduct (please see comment 2.2), and the lack of heterodox papers in top economics journals (please see comment 3.3). While doing this, I used this comment: I tried not to get lost in details.

    [1.4] The description of the data and the justification of the choice of variables is clear and thorough. However, the naming of the variables could be improved. For example, in my understanding, table 1 rows 5 to 7 are one variable and rows 8 to 10 are another variable. So each level of the variables gets one row. This leads to the problem that variables are not clearly separated and associated with their respective levels. In other words, table 1 and the regression results table (table 3) can easily be misunderstood because the impression is suggested that, e.g., "Affiliated Institution: North America" is one variable, "Affiliated Institution: Europe" is another variable, etc. These variables in both tables should be better labeled and grouped visually to prevent misunderstandings.
    (***) In this version, I grouped the dummy variables that are created from a single variable both in Table 1 (page 6), and Table 3 (page 12). I used borders and subtitles to separate these groups.

    [1.5] Also, I have one issue the data section/regression model specification has left me unclear about. For co-authored papers, how are the variables handled when the values differ? For example, an article of two authors, one of which is from North America and one is from Rest of the World. Which covariate is used? Both? In fact, it is not clear what the level of analysis in the regression model is, authors, articles, or a combination.
    (***) A clear description was missing in the previous version. I also incorporate Referee #2’s comment about clustering observations. Both of these issues made clear in the following paragraph (page 11)
    “There are 1052 articles that are written by 2453 authors. If three authors write a paper, all article-related variables would take the same value for these three observations. Therefore, these observations are not independently and identically distributed (iid). For this reason, we clustered standard errors at the article level. There are authors who publish more than one AER article during the ten years that we included. In fact, there are 1845 distinct authors who write these 1052 articles. However, most author-level variables do not take the same value for the same author. The post-PhD experience, the affiliated institution, and the publication records may be different for the same author who publish in different years. Nevertheless, the observations with the same author would not be iid. Therefore, we also report the standard errors that are clustered at the author level.”

    [1.6] The use of regression analysis on log-transformed citation counts is appropriate for the data and research question.
    (***) Referee #1 makes a general evaluation of the regression method.

    [1.7] The results illustrate the strong association of academic background (affiliation/continent) and citation impact and the unimportance of other author factors such as gender. The simulation excercise also provides some interesting, if hypothetical, outcomes on how relatively easy it could be to increase journal citation statistics by simple selection strategies if editors are willing to sacrifice the total number of published papers. These results also show how the published work in the journal would degenerate to something like a monoculture in terms of institutional affiliations and empirical vs. theoretical contributions.

    (***) Referee #1 makes a general evaluation of the simulation method.

    [1.8] In summary, the article is an interesting and overall well-written contribution to the literature on author factors influencing citation counts. The manuscript also shows that the author has a good command the relevant prior literature and can contextualize his results well with those of other studies. Nevertheless, the manuscript still has some minor flaws that should be worked out before acceptance can be recommended.

    (***) Referee #1 makes a general evaluation of the paper.

    Minor remarks
    [1.9] in several places the authors uses "attained" where it should probably be "obtained" when describing how data was collected
    (***) I have checked all of the places where attain is used and correct it when necessary.

    [1.10] p. 6 the hyperlink to the Shanghai ranking page is already outdated and 404

    () In this version I used the QS rankings (please see comment 3.6), and report a valid web page.
    [1.11] p. 9, l.~31 used "experimental", but perhaps "empirical" was meant, which was also used in the referred to table
    (
    ) I corrected the confusion.

    [1.12] I suggest the author is more careful about using phrases which suggest causal relationships, e.g. p. 11 "Average citations that have been received by authors’ earlier top five journal publications positively affect the citation performance of their current AER article". This could be understood to mean the earlier citations cause the later citation while it would be more appropriate to state that the two variables are associated. Another example on the same page: "Yet, more publications enable authors to receive more citations."

    (***) I checked all the text, and correct all statements that have a causal statement. For example, the sentences Referee #1 notes have been changed as follows.
    “Average citations that have been received by authors’ earlier top-five journal publications are positively related to the citation performance of their current AER article” (page 11)
    “Yet, authors who have more publications receive more citations.” (page 13)

    Reviewer: 2

    Comments to the Author
    Referee report on “Predicting the impact of American Economic Review articles by author characteristics" (QSS-2021-0052)

    [2.1] The paper provides some evidence on the effect of author characteristics on the citation performance of articles published between 2008 and 2017 in the American Economic Review. The study finds that authors with greater past publication and citation performance and from top-ranked affiliation received more citations within the first two years of publication. In the second part of the study, the author conducted a simulation exercise to show an opportunistic editor whose goal is to improve the journal’s average citation performance might discriminate against authors with certain characteristics.

    (***) Referee #2 summarizes the results.
    Main comments:

    [2.2] The discussion of the literature is somewhat limited to the factors driving citation behaviour, yet the more important argument (I think) in this paper is about the potential strategic editorial behaviour. I think the literature review can be improved by incorporating a discussion on this linking to other forms of editorial bias such as publication bias (favouring positive results).
    The paper should also discuss, at least theoretically, the role of the referees. While editors sometimes overrule referees’ recommended decisions, reviewers could exert a large influence on the publication outcome of a paper which may not depend on the quality of the paper. One may even be able to extend the analysis by examining this empirically as the list of reviewers is published in each AER volume.

    (***) I have added literature about editorial misconduct, and the role of referees to the Related Literature section. I also incorporate Referee #3’s comment about the lack of heterodox literature (see comment 3.3). The new part of the section is given in page 4 & 5 as follows.
    “This paper is also related to opportunistic editorial practices. Martin (2016) summarizes the common editorial misconduct that tries to enhance the citation performance of the journals. The editors force authors to cite papers from their journals; they form journal cartels, where journal A cites journal B, and journal B cites journal A; and they create an online queue of accepted papers, so they can choose the papers that are developing a better citation performance to publish.
    There are some opportunistic editorial practices to boost citation performance that cannot be classified as unethical. For example, it is known that positive and strong results are more easily published. Franco et al. (2014) analyze all the projects in a prestigious sponsored program in social sciences, and they show that if the results are not significant and positive, the results are not likely to be written, and the papers are not likely to be published if written. The null results are not published possibly because of their low expected citation performance. The fact that stronger results get more citations is called citation bias (Fanelli et al. 2017).
    The editors may not publish articles in certain subfields of economics because of their citation performance. For example, there are few replication experiments (Andrews & Kasy 2019) in the top economics journals. It is also known that heterodox papers are not published in the top economics journals (Earl & Peng 2012). This may be due to the fact that the heterodox articles do not receive many citations from mainstream articles (Lee 2012).
    The editorial decisions are not taken by the editor alone, the editorial team is important. Card and DellaVigna (2020) find that editors closely follow referee decisions for economics journals. They also note that the referee recommendations correlate strongly with the citation performance of the papers. Naturally, the editors have some discretion. For example, they follow the recommendations of the referees who are more productive more closely, although referees who are less productive have an equally good performance in predicting the citation performance. ”

    [2.3] The citation outcome is quite short-term focus (2-year impact factor), per se, one could extend this to look at longer citation performance (e.g., take the first half of the sample (published before 2012) and look at their 5-year/10-year impact). This can increase the contribution of the paper, potentially to the literature on Sleeping Beauty (SB) in science (e.g., how would the results change by having an editor with long-term foresight?).

    (***) Unfortunately, I could not incorporate this recommendation. There are two reasons for this:
    a-) I changed the simulation section (please see comment 3.7). In this version, the opportunistic editor uses the first seven years of data to predict the last three year’s citation performance. Therefore, the data is not sufficient to make a long-term analysis.
    b-) Sleeping Beauty is an important subject on its own. Unfortunately, I am unable to handle two related but separate subjects within the same article.

    [2.4] The author should consider incorporating the topic of the paper (e.g., using JEL codes, microeconomics or macroeconomics) in the analysis as the citation dynamics could be driven by, whether it is not only theoretical/empirical in nature but also how hot the field is (e.g., maturity of a field). More importantly, this might correlate with author characteristics (e.g., authors from less prestigious universities may go for more novel topics).
    Following the point above, as a general journal in the field of economics, AER would also have an interest in maintaining the diversity of the topics of its article portfolio (other than focusing on its average citations), compared to a specialised field journal. This aspect could be further discussed in the paper.
    (***) I have collected JEL codes of all 1052 articles. I end up not using them for the following reasons.
    a-) The JEL codes are not consistently given. There are large differences between the JEL codes of a working paper and the JEL codes of their published version.
    b-) It is not possible to differentiate between the fields by JEL codes. The articles have given many JEL codes so that many papers can be classified as both micro and macroeconomics according to this self-classification.
    c-) There are no significant citation performance differences between the subfields from JEL codes. Since this paper is mainly about author characteristics, I did not want to include a paper-related factor that is highly insignificant.
    d-) The differences between subfields in economics are blurred, so I could not assign subfields better than what JEL codes have suggested.
    However, there is one improvement that I achieved through this comment. As I was collecting the JEL information, I realized that I could consistently decide on whether the paper is theoretical or empirical. I decided to use this paper-related variable which is highly significant. Therefore, Table 2 (page 10) - which looks at the relation between whether the AER article is theoretical and whether the author has ever written in the top theoretical journal -includes all observations (in the previous version, it had just 20 percent of the sample). Moreover, this variable is included in the regression. (Table 3 page 12)

    [2.5] Given the potential problem posited by the author, I think the paper should include some suggestions or potential solutions to deal with such a problem. How should the journal maintain its editorial check and balance? Would a large editorial board help, particularly in the situation with imperfect information on article quality?

    (***) I add the following paragraph in the conclusion (page 19) for a brief discussion on what can be done.
    “AER considers itself a general-interest economics journal that is among the most scholarly journals in economics. We show that the author characteristics are concentrated in AER. It is a good feature of such a high-quality journal to give a fair chance to authors. The blind review would solve the problem, however, it is getting more difficult to hide the identity of the authors. Another policy would be to give statistics about the authors in papers that are accepted and rejected. For example, if the theorists’ articles are disproportionately rejected, the journal may investigate whether this is the result of a fair procedure. ”

    [2.6] One limitation that should be discussed is that the results are based only on accepted papers, and rejected submissions were not considered. An editor would likely learn about a rejected paper (that was published elsewhere) receiving a large number of citations – so an editor with limited information on the quality of the paper may not only look at past published articles in AER.

    (***) The limitation of the regression analysis was given in the last paragraph of the Regression section (page 13), however, as Referee #2 notes, the limitations to the Simulation section were not given. Now the following paragraph is added to the end of the Simulation section (page 18)
    “There are obvious limitations to the simulation results. We rely on the accepted papers rather than all submitted papers. The editor may learn about the citation performance of the rejected papers and will have a better idea about the citation potential of any given article that we suggest. It is also the case that editors observe some variables that are unobservable to us. For example, the editor may see whether the subject matter of the article is popular. In this case, the opportunistic editor’s selection may be different and citation performance improvements may be less substantial than we suggest. “

    [2.7] The paper should also discuss any potential unobserved factors that correlate with an article’s quality (but observed to an editor), which would influence the outcome of the simulation exercise. Particularly, if such factors are specific to papers that are accepted for publication (and that the top-end of the quality distribution do compose of more authors from the higher-ranked authors), then the articles selected by an opportunistic editor may actually not differ that much from the actual selection, but obviously depends on what information is limited.

    (***) Please see comment 2.6.

    Minor comment:
    [2.8] The unit of analysis is on author-article level, which means that observations are not iid (repeated author who publish more than once and co-author papers), one should cluster the SE at the author as well as paper level to increase the robustness of the findings.
    (***) In this version, I give standard errors clustered at the article-level and author-level in Table 3 (page 12). I also explain why there is a need to cluster (please see comment 1.5)

    [2.9] It would be useful to show how the structure of the author characteristics changed given the opportunistic behaviour.

    (***) I added two new figures. Figure 3 (page 16) shows how the author concentration by the location of the affiliated institutions changes. Figure 5 (page 17) shows how the gender composition changes.

    Reviewer: 3

    Comments to the Author

    [3.1] This paper presents an analysis of the association between the citation impact of articles published in the American Economic Review and the characteristics of the authors of these articles. While I believe the paper has significant potential, it also has a number of important weaknesses that need to be addressed by the author.
    (***) Referee #3 gives a general evaluation of the paper.

    [3.2] Most importantly, in the introduction section of the paper, no clear research question or research objective is presented. The author explains what is done in the paper, but not why it is done. Presenting a clear and well-motivated research question or research objective is essential.

    (***) I agree with referee #3. I added the following paragraph to the introduction to clarify the research questions (page 2)
    “In this paper, we have two aims. First, we want to identify the author characteristics that are strongly correlated with the citation performance of the American Economic Review (AER) articles. In particular, we want to see whether the prevalent author characteristics such as being affiliated in the top ten institutions are strongly and positively related to the citation performance. Second, we want to see whether a hypothetical opportunistic editor can substantially improve the citation performance of AER by using the predicted citation performance. This way, we want to see whether the opportunistic behavior increases the concentration of certain author characteristics.”

    [3.3] The author has invested a significant effort into reviewing related literature. However, what is missing is a review of literature that discusses the pros and cons of economic research being strongly focused on publishing in a small number of ‘elite’ journals, for instance literature that discusses the risks of marginalization of non-mainstream approaches to economic research and the risks of economic research being dominated by a small number of ‘elite’ universities. In the light of the empirical findings presented by the author, a review of this literature would be highly relevant.
    (***) Unfortunately, I could not find many papers that discuss the lack of heterodox research in top economics journals. However, I have added the following brief discussion on page 5.
    “The editors may not publish articles in certain subfields of economics because of their citation performance. For example, there are few replication experiments (Andrews & Kasy 2019) in the top economics journals. It is also known that heterodox papers are not published in the top economics journals (Earl & Peng 2012). This may be due to the fact that the heterodox articles do not receive many citations from mainstream articles (Lee 2012). ”

    [3.4] In addition, while I appreciate that it is not possible to comprehensively review all relevant literature, the selection of the studies included in the literature review sometimes feels a bit arbitrary. For instance, there is a significant body of work studying the relationship between gender and citation impact. The choice of the three studies cited by the author seems rather arbitrary, and two of the studies are quite old (from 2000 and 2012). It would be preferable to cite more recent studies.
    (***) I did the following improvements.
    a-) First, I have changed the gender part of the literature discussion. I excluded two studies that are old and replaced them with new ones. On page 4:
    “There is no consensus on whether the gender of authors is an important factor for citation performance. Nunkoo et al. (2019) find that men receive significantly more citations than women in the tourism field, whereas Hengel and Moon (2020) find that women receive significantly more citations than men in economics. Nielsen (2017) could not find any significant gender effects for management journals. Thelwall (2020) analyzes citation performance in 27 academic fields and shows that whether or not women have a higher citation performance depends on the academic field.”
    b-) I exclude older papers if there are more recent examples. Danell (2011), Kodrzycki & Yu (2006), and Skilton (2009) are excluded for this reason.
    c-) For all the older papers, I looked at the alternatives. I replaced Althouse et al. (2009) with Petersen et al. (2019), and Bosquet & Combes (2013) with Card & DellaVigna (2020).
    d-) I chose more recent papers when I expanded the literature. (please see comment 2.2)
    I still have some older papers especially about the bibliometrics under the economics field. I could not find any suitable replacements for these studies.

    [3.5] The author explains that gender information was obtained from pictures in internet biographies. This approach is increasingly considered unsatisfactory. Ideally gender information is obtained from self-reporting by the individuals that are being studied (e.g., self-reporting in someone’s CV) instead of guessing someone’s gender from a picture. This needs to be acknowledged as a limitation of the research.
    (***) I have collected the gender information from biographies from a subset of the papers. This analysis is explained on page 7 as follows.
    “We obtained gender information from pictures in the authors’ internet biographies. This is not an ideal method as errors can be made during the process. We independently collected gender information for randomly selected 245 (10 percent of the sample) authors without using picture information. Most biographies do not contain direct gender information so we mostly obtained gender information from gender pronouns. We could deduce the gender of 224 (91 percent of the random sample). The gender information collected through pictures did not contain any errors for these 224 authors. Since we could get the pictures of 100% of the authors, we used the gender obtained by the pictures in the analysis.”

    [3.6] The author uses the Shanghai Ranking to obtain a ranking of institutions in the field of economics. The use of the Shanghai Ranking needs a better motivation, especially because this ranking has attracted significant criticism. I wonder to what extent economists regard this ranking as a valid and reliable tool for identifying ‘top institutions’ and whether there are other rankings that could be used as an alternative.

    (***) I do not think that there is one single ranking that everyone can agree on. Nevertheless, I have looked into three other rankings that are used widely. THE and USNews rank economics and business subjects together. This leads to a ranking that would be unacceptable to many economists. For example, Princeton was not even ranked in one of these rankings and has been ranked as 17th in the other. I replaced Shanghai with QS rankings that rank the economics subject alone.

    [3.7] In the simulation model, the analysis of an opportunistic editor with limited information has an important limitation that needs to be acknowledged. In reality, an opportunistic editor with limited information would make predictions for new submissions to his/her journal using a model that is based on older submissions. However, the simulations presented by the author don’t make a distinction between old and new submissions. Predictions are made for articles using a model that is based on the same articles. This leads to the problem of overfitting. As a consequence of this problem, the increase in citation impact by 31.7% and 53.3% reported in Section 5 is an overestimation of the increase that would actually be realized in practice.
    (***) I changed the simulation part. In this version, the opportunistic editor uses the first seven years of data to predict the citation performance in the last three years. This part of the analysis is explained on page 14:
    “We consider two types of opportunistic editors. First, we consider the opportunistic editor who has perfect foresight, so that he exactly knows the number of citations that an article will receive before he accepts the article. Second, we consider the opportunistic editor who has limited information. We re-ran the General Linear Model with Logarithm presented earlier in the regression analysis section by restricting the sample for the articles published from 2008 to 2014. Then, we use the coefficients to compute the predicted citation performance for the articles published from the years 2015 to 2017. Therefore, the opportunistic editor uses the information by using the first seven years of our sample to form expectations for the last three years of our sample. Lastly, the opportunistic editor simply takes the average of the predicted performance of authors to find the predicted citation performance of the article. Both types of editors select half and a quarter of articles within each of the three years. ”

    End of Report.

    Author response by


    Cite this author response
  • pre-publication peer review (ROUND 1)
    Decision Letter
    2021/10/03

    03-Oct-2021

    Dear Dr. Yuret:

    Your manuscript QSS-2021-0052 entitled "Predicting the impact of American Economic Review articles by author characteristics", which you submitted to Quantitative Science Studies, has been reviewed. The comments of the reviewers are included at the bottom of this letter.

    Based on the comments of the reviewers as well as my own reading of your manuscript, my editorial decision is to invite you to prepare a major revision of your manuscript. I need to emphasize that revising your manuscript does not guarantee that your work will eventually be accepted for publication. This will depend on the outcome of the revision.

    To revise your manuscript, log into https://mc.manuscriptcentral.com/qss and enter your Author Center, where you will find your manuscript title listed under "Manuscripts with Decisions." Under "Actions," click on "Create a Revision." Your manuscript number has been appended to denote a revision.

    You may also click the below link to start the revision process (or continue the process if you have already started your revision) for your manuscript. If you use the below link you will not be required to login to ScholarOne Manuscripts.

    PLEASE NOTE: This is a two-step process. After clicking on the link, you will be directed to a webpage to confirm.

    https://mc.manuscriptcentral.com/qss?URL_MASK=ee8ccca54b5b4b8096fada95acca9ec1

    You will be unable to make your revisions on the originally submitted version of the manuscript. Instead, revise your manuscript using a word processing program and save it on your computer. Please also highlight the changes to your manuscript within the document by using the track changes mode in MS Word or by using bold or colored text.

    Once the revised manuscript is prepared, you can upload it and submit it through your Author Center.

    When submitting your revised manuscript, you will be able to respond to the comments made by the reviewers in the space provided. You can use this space to document any changes you make to the original manuscript. In order to expedite the processing of the revised manuscript, please be as specific as possible in your response to the reviewers.

    IMPORTANT: Your original files are available to you when you upload your revised manuscript. Please delete any redundant files before completing the submission.

    If possible, please try to submit your revised manuscript by 31-Jan-2022. Let me know if you need more time to revise your work.

    Once again, thank you for submitting your manuscript to Quantitative Science Studies and I look forward to receiving your revision.

    Best wishes,
    Dr. Ludo Waltman
    Editor, Quantitative Science Studies
    qss@issi-society.org

    Reviewers' Comments to Author:

    Reviewer: 1

    Comments to the Author
    General remarks

    The present study is concerned with author factors that incluence the citation impact of journal articles. Among these, primarily the academic background, i.e. institutional affiliation, is investigated. The study is limited to a single journal, albeit one of the most important ones in its field. This limitation is justified by the effort of data collection, which puts analysis of larger samples beyond the means of single authors or small groups. Moreover, this limitation is somewhat ameliorated by various comparisons to statistics for other journals taken from the literature and from a sample of lower ranked journals.

    I believe that the abstract should be more balanced to the general results and less to the simulation exercise results.

    The presented literature review is adequate in that it selectively reports on prior work of relevance without getting lost in details or excessively trying to review every single paper published in the rather large topic.

    The description of the data and the justification of the choice of variables is clear and thorough. However, the naming of the variables could be improved. For example, in my understanding, table 1 rows 5 to 7 are one variable and rows 8 to 10 are another variable. So each level of the variables gets one row. This leads to the problem that variables are not clearly separated and associated with their respective levels. In other words, table 1 and the regression results table (table 3) can easily be misunderstood because the impression is suggested that, e.g., "Affiliated Institution: North America" is one variable, "Affiliated Institution: Europe" is another variable, etc. These variables in both tables should be better labeled and grouped visually to prevent misunderstandings.

    Also, I have one issue the data section/regression model specification has left me unclear about. For co-authored papers, how are the variables handled when the values differ? For example, an article of two authors, one of which is from North America and one is from Rest of the World. Which covariate is used? Both? In fact, it is not clear what the level of analysis in the regression model is, authors, articles, or a combination.

    The use of regression analysis on log-transformed citation counts is appropriate for the data and research question.

    The results illustrate the strong association of academic background (affiliation/continent) and citation impact and the unimportance of other author factors such as gender. The simulation excercise also provides some interesting, if hypothetical, outcomes on how relatively easy it could be to increase journal citation statistics by simple selection strategies if editors are willing to sacrifice the total number of published papers. These results also show how the published work in the journal would degenerate to something like a monoculture in terms of institutional affiliations and empirical vs. theoretical contributions.

    In summary, the article is an interesting and overall well-written contribution to the literature on author factors influencing citation counts. The manuscript also shows that the author has a good command the relevant prior literature and can contextualize his results well with those of other studies. Nevertheless, the manuscript still has some minor flaws that should be worked out before acceptance can be recommended.

    Minor remarks
    in several places the authors uses "attained" where it should probably be "obtained" when describing how data was collected
    p. 6 the hyperlink to the Shanghai ranking page is already outdated and 404
    p. 9, l.~31 used "experimental", but perhaps "empirical" was meant, which was also used in the referred to table
    I suggest the author is more careful about using phrases which suggest causal relationships, e.g. p. 11 "Average citations that have been received by authors’ earlier top five journal publications positively affect the citation performance of their current AER article". This could be understood to mean the earlier citations cause the later citation while it would be more appropriate to state that the two variables are associated. Another example on the same page: "Yet, more publications enable authors to receive more citations."

    Reviewer: 2

    Comments to the Author
    Referee report on “Predicting the impact of American Economic Review articles by author characteristics" (QSS-2021-0052)
    The paper provides some evidence on the effect of author characteristics on the citation performance of articles published between 2008 and 2017 in the American Economic Review. The study finds that authors with greater past publication and citation performance and from top-ranked affiliation received more citations within the first two years of publication. In the second part of the study, the author conducted a simulation exercise to show an opportunistic editor whose goal is to improve the journal’s average citation performance might discriminate against authors with certain characteristics.
    Main comments:
    The discussion of the literature is somewhat limited to the factors driving citation behaviour, yet the more important argument (I think) in this paper is about the potential strategic editorial behaviour. I think the literature review can be improved by incorporating a discussion on this linking to other forms of editorial bias such as publication bias (favouring positive results).
    The paper should also discuss, at least theoretically, the role of the referees. While editors sometimes overrule referees’ recommended decisions, reviewers could exert a large influence on the publication outcome of a paper which may not depend on the quality of the paper. One may even be able to extend the analysis by examining this empirically as the list of reviewers is published in each AER volume.
    The citation outcome is quite short-term focus (2-year impact factor), per se, one could extend this to look at longer citation performance (e.g., take the first half of the sample (published before 2012) and look at their 5-year/10-year impact). This can increase the contribution of the paper, potentially to the literature on Sleeping Beauty (SB) in science (e.g., how would the results change by having an editor with long-term foresight?).
    The author should consider incorporating the topic of the paper (e.g., using JEL codes, microeconomics or macroeconomics) in the analysis as the citation dynamics could be driven by, whether it is not only theoretical/empirical in nature but also how hot the field is (e.g., maturity of a field). More importantly, this might correlate with author characteristics (e.g., authors from less prestigious universities may go for more novel topics).
    Following the point above, as a general journal in the field of economics, AER would also have an interest in maintaining the diversity of the topics of its article portfolio (other than focusing on its average citations), compared to a specialised field journal. This aspect could be further discussed in the paper.
    Given the potential problem posited by the author, I think the paper should include some suggestions or potential solutions to deal with such a problem. How should the journal maintain its editorial check and balance? Would a large editorial board help, particularly in the situation with imperfect information on article quality?
    One limitation that should be discussed is that the results are based only on accepted papers, and rejected submissions were not considered. An editor would likely learn about a rejected paper (that was published elsewhere) receiving a large number of citations – so an editor with limited information on the quality of the paper may not only look at past published articles in AER.
    The paper should also discuss any potential unobserved factors that correlate with an article’s quality (but observed to an editor), which would influence the outcome of the simulation exercise. Particularly, if such factors are specific to papers that are accepted for publication (and that the top-end of the quality distribution do compose of more authors from the higher-ranked authors), then the articles selected by an opportunistic editor may actually not differ that much from the actual selection, but obviously depends on what information is limited.

    Minor comment:
    The unit of analysis is on author-article level, which means that observations are not iid (repeated author who publish more than once and co-author papers), one should cluster the SE at the author as well as paper level to increase the robustness of the findings.
    It would be useful to show how the structure of the author characteristics changed given the opportunistically behaviour.

    Reviewer: 3

    Comments to the Author
    This paper presents an analysis of the association between the citation impact of articles published in the American Economic Review and characteristics of the authors of these articles. While I believe the paper has significant potential, it also has a number of important weaknesses that need to be addressed by the author.

    Most importantly, in the introduction section of the paper, no clear research question or research objective is presented. The author explains what is done in the paper, but not why it is done. Presenting a clear and well-motivated research question or research objective is essential.

    The author has invested a significant effort into reviewing related literature. However, what is missing is a review of literature that discusses the pros and cons of economic research being strongly focused on publishing in a small number of ‘elite’ journals, for instance literature that discusses the risks of marginalization of non-mainstream approaches to economic research and the risks of economic research being dominated by a small number of ‘elite’ universities. In the light of the empirical findings presented by the author, a review of this literature would be highly relevant.

    In addition, while I appreciate that it is not possible to comprehensively review all relevant literature, the selection of the studies included in the literature review sometimes feels a bit arbitrary. For instance, there is a significant body of work studying the relationship between gender and citation impact. The choice of the three studies cited by the author seems rather arbitrary, and two of the studies are quite old (from 2000 and 2012). It would be preferable to cite more recent studies.

    The author explains that gender information was obtained from pictures in internet biographies. This approach is increasingly considered unsatisfactory. Ideally gender information is obtained from self-reporting by the individuals that are being studied (e.g., self-reporting in someone’s CV) instead of guessing someone’s gender from a picture. This needs to be acknowledged as a limitation of the research.

    The author uses the Shanghai Ranking to obtain a ranking of institutions in the field of economics. The use of the Shanghai Ranking needs a better motivation, especially because this ranking has attracted significant criticism. I wonder to what extent economists regard this ranking as a valid and reliable tool for identifying ‘top institutions’ and whether there are other rankings that could be used as an alternative.

    In the simulation model, the analysis of an opportunistic editor with limited information has an important limitation that needs to be acknowledged. In reality, an opportunistic editor with limited information would make predictions for new submissions to his/her journal using a model that is based on older submissions. However, the simulations presented by the author don’t make a distinction between old and new submissions. Predictions are made for articles using a model that is based on the same articles. This leads to the problem of overfitting. As a consequence of this problem, the increase in citation impact by 31.7% and 53.3% reported in Section 5 is an overestimation of the increase that would actually be realized in practice.

    Decision letter by
    Cite this decision letter
    Reviewer report
    2021/10/03

    This paper presents an analysis of the association between the citation impact of articles published in the American Economic Review and characteristics of the authors of these articles. While I believe the paper has significant potential, it also has a number of important weaknesses that need to be addressed by the author.

    Most importantly, in the introduction section of the paper, no clear research question or research objective is presented. The author explains what is done in the paper, but not why it is done. Presenting a clear and well-motivated research question or research objective is essential.

    The author has invested a significant effort into reviewing related literature. However, what is missing is a review of literature that discusses the pros and cons of economic research being strongly focused on publishing in a small number of ‘elite’ journals, for instance literature that discusses the risks of marginalization of non-mainstream approaches to economic research and the risks of economic research being dominated by a small number of ‘elite’ universities. In the light of the empirical findings presented by the author, a review of this literature would be highly relevant.

    In addition, while I appreciate that it is not possible to comprehensively review all relevant literature, the selection of the studies included in the literature review sometimes feels a bit arbitrary. For instance, there is a significant body of work studying the relationship between gender and citation impact. The choice of the three studies cited by the author seems rather arbitrary, and two of the studies are quite old (from 2000 and 2012). It would be preferable to cite more recent studies.

    The author explains that gender information was obtained from pictures in internet biographies. This approach is increasingly considered unsatisfactory. Ideally gender information is obtained from self-reporting by the individuals that are being studied (e.g., self-reporting in someone’s CV) instead of guessing someone’s gender from a picture. This needs to be acknowledged as a limitation of the research.

    The author uses the Shanghai Ranking to obtain a ranking of institutions in the field of economics. The use of the Shanghai Ranking needs a better motivation, especially because this ranking has attracted significant criticism. I wonder to what extent economists regard this ranking as a valid and reliable tool for identifying ‘top institutions’ and whether there are other rankings that could be used as an alternative.

    In the simulation model, the analysis of an opportunistic editor with limited information has an important limitation that needs to be acknowledged. In reality, an opportunistic editor with limited information would make predictions for new submissions to his/her journal using a model that is based on older submissions. However, the simulations presented by the author don’t make a distinction between old and new submissions. Predictions are made for articles using a model that is based on the same articles. This leads to the problem of overfitting. As a consequence of this problem, the increase in citation impact by 31.7% and 53.3% reported in Section 5 is an overestimation of the increase that would actually be realized in practice.

    Reviewed by
    Cite this review
    Reviewer report
    2021/09/21

    Referee report on “Predicting the impact of American Economic Review articles by author characteristics" (QSS-2021-0052)
    The paper provides some evidence on the effect of author characteristics on the citation performance of articles published between 2008 and 2017 in the American Economic Review. The study finds that authors with greater past publication and citation performance and from top-ranked affiliation received more citations within the first two years of publication. In the second part of the study, the author conducted a simulation exercise to show an opportunistic editor whose goal is to improve the journal’s average citation performance might discriminate against authors with certain characteristics.
    Main comments:
    The discussion of the literature is somewhat limited to the factors driving citation behaviour, yet the more important argument (I think) in this paper is about the potential strategic editorial behaviour. I think the literature review can be improved by incorporating a discussion on this linking to other forms of editorial bias such as publication bias (favouring positive results).
    The paper should also discuss, at least theoretically, the role of the referees. While editors sometimes overrule referees’ recommended decisions, reviewers could exert a large influence on the publication outcome of a paper which may not depend on the quality of the paper. One may even be able to extend the analysis by examining this empirically as the list of reviewers is published in each AER volume.
    The citation outcome is quite short-term focus (2-year impact factor), per se, one could extend this to look at longer citation performance (e.g., take the first half of the sample (published before 2012) and look at their 5-year/10-year impact). This can increase the contribution of the paper, potentially to the literature on Sleeping Beauty (SB) in science (e.g., how would the results change by having an editor with long-term foresight?).
    The author should consider incorporating the topic of the paper (e.g., using JEL codes, microeconomics or macroeconomics) in the analysis as the citation dynamics could be driven by, whether it is not only theoretical/empirical in nature but also how hot the field is (e.g., maturity of a field). More importantly, this might correlate with author characteristics (e.g., authors from less prestigious universities may go for more novel topics).
    Following the point above, as a general journal in the field of economics, AER would also have an interest in maintaining the diversity of the topics of its article portfolio (other than focusing on its average citations), compared to a specialised field journal. This aspect could be further discussed in the paper.
    Given the potential problem posited by the author, I think the paper should include some suggestions or potential solutions to deal with such a problem. How should the journal maintain its editorial check and balance? Would a large editorial board help, particularly in the situation with imperfect information on article quality?
    One limitation that should be discussed is that the results are based only on accepted papers, and rejected submissions were not considered. An editor would likely learn about a rejected paper (that was published elsewhere) receiving a large number of citations – so an editor with limited information on the quality of the paper may not only look at past published articles in AER.
    The paper should also discuss any potential unobserved factors that correlate with an article’s quality (but observed to an editor), which would influence the outcome of the simulation exercise. Particularly, if such factors are specific to papers that are accepted for publication (and that the top-end of the quality distribution do compose of more authors from the higher-ranked authors), then the articles selected by an opportunistic editor may actually not differ that much from the actual selection, but obviously depends on what information is limited.

    Minor comment:
    The unit of analysis is on author-article level, which means that observations are not iid (repeated author who publish more than once and co-author papers), one should cluster the SE at the author as well as paper level to increase the robustness of the findings.
    It would be useful to show how the structure of the author characteristics changed given the opportunistically behaviour.

    Reviewed by
    Cite this review
    Reviewer report
    2021/08/17

    General remarks

    The present study is concerned with author factors that incluence the citation impact of journal articles. Among these, primarily the academic background, i.e. institutional affiliation, is investigated. The study is limited to a single journal, albeit one of the most important ones in its field. This limitation is justified by the effort of data collection, which puts analysis of larger samples beyond the means of single authors or small groups. Moreover, this limitation is somewhat ameliorated by various comparisons to statistics for other journals taken from the literature and from a sample of lower ranked journals.

    I believe that the abstract should be more balanced to the general results and less to the simulation exercise results.

    The presented literature review is adequate in that it selectively reports on prior work of relevance without getting lost in details or excessively trying to review every single paper published in the rather large topic.

    The description of the data and the justification of the choice of variables is clear and thorough. However, the naming of the variables could be improved. For example, in my understanding, table 1 rows 5 to 7 are one variable and rows 8 to 10 are another variable. So each level of the variables gets one row. This leads to the problem that variables are not clearly separated and associated with their respective levels. In other words, table 1 and the regression results table (table 3) can easily be misunderstood because the impression is suggested that, e.g., "Affiliated Institution: North America" is one variable, "Affiliated Institution: Europe" is another variable, etc. These variables in both tables should be better labeled and grouped visually to prevent misunderstandings.

    Also, I have one issue the data section/regression model specification has left me unclear about. For co-authored papers, how are the variables handled when the values differ? For example, an article of two authors, one of which is from North America and one is from Rest of the World. Which covariate is used? Both? In fact, it is not clear what the level of analysis in the regression model is, authors, articles, or a combination.

    The use of regression analysis on log-transformed citation counts is appropriate for the data and research question.

    The results illustrate the strong association of academic background (affiliation/continent) and citation impact and the unimportance of other author factors such as gender. The simulation excercise also provides some interesting, if hypothetical, outcomes on how relatively easy it could be to increase journal citation statistics by simple selection strategies if editors are willing to sacrifice the total number of published papers. These results also show how the published work in the journal would degenerate to something like a monoculture in terms of institutional affiliations and empirical vs. theoretical contributions.

    In summary, the article is an interesting and overall well-written contribution to the literature on author factors influencing citation counts. The manuscript also shows that the author has a good command the relevant prior literature and can contextualize his results well with those of other studies. Nevertheless, the manuscript still has some minor flaws that should be worked out before acceptance can be recommended.

    Minor remarks
    in several places the authors uses "attained" where it should probably be "obtained" when describing how data was collected
    p. 6 the hyperlink to the Shanghai ranking page is already outdated and 404
    p. 9, l.~31 used "experimental", but perhaps "empirical" was meant, which was also used in the referred to table
    I suggest the author is more careful about using phrases which suggest causal relationships, e.g. p. 11 "Average citations that have been received by authors’ earlier top five journal publications positively affect the citation performance of their current AER article". This could be understood to mean the earlier citations cause the later citation while it would be more appropriate to state that the two variables are associated. Another example on the same page: "Yet, more publications enable authors to receive more citations."

    Reviewed by
    Cite this review
All peer review content displayed here is covered by a Creative Commons CC BY 4.0 license.