Abstract

The citation impact of a scientific publication is usually seen as a one-dimensional concept. We introduce a multi-dimensional framework for characterizing the citation impact of a publication. In addition to the level of citation impact, quantified by the number of citations received by a publication, we also conceptualize and operationalize the depth and breadth and the dependence and independence of the citation impact of a publication. The proposed framework enables us to distinguish between publications that have a deep citation impact in a relatively narrow research area and publications that have a broad citation impact in a wider research area. It also allows us to make a distinction between publications that are strongly dependent on earlier work and publications that make a more independent scientific contribution. We use our multi-dimensional citation impact framework to report basic descriptive statistics on the citation impact of highly cited publications in all scientific disciplines. In addition, we present a detailed case study focusing on the field of scientometrics. The proposed citation impact framework provides a more in-depth understanding of the citation impact of a publication than a traditional one-dimensional perspective.


Authors

Bu Yi;  Waltman Ludo;  Huang Yong

Publons users who've claimed - I am an author
Contributors on Publons
  • 2 authors
  • 2 reviewers
  • pre-publication peer review (FINAL ROUND)
    Decision Letter
    2020/12/31

    31-Dec-2020

    Dear Dr. Bu:

    It is a pleasure to accept your manuscript entitled "A multi-dimensional framework for characterizing the citation impact of scientific publications" for publication in Quantitative Science Studies. Both referees have concluded that you had addressed their concerns on the manuscripts, and did not have any further comments.

    I would like to request you to prepare the final version of your manuscript using the checklist available at https://bit.ly/2QW3uV5. Please also sign the publication agreement, which can be downloaded from https://bit.ly/2QYuW4w. The final version of your manuscript, along with the completed checklist and the signed publication agreement, can be returned to qss@issi-society.org.

    Thank you for your contribution. On behalf of the Editors of Quantitative Science Studies, I look forward to your continued contributions to the journal.

    Best wishes,
    Prof. Vincent Larivière
    Editor, Quantitative Science Studies
    vincent.lariviere@umontreal.ca

    Decision letter by
    Cite this decision letter
    Author Response
    2020/11/23

    Response to Reviewers

    Response to Reviewer #1
    Thank you for your detailed comments.

    1. The authors propose the concept of measuring citation impact multi-dimensionally by introducing two indicators. This is a very interesting study that should be published in QSS. In a revision of the manuscript, the following points should be considered.
      Response: Thank you for your positive assessment of our work. Please see our response below.

    2. In this paper (https://www.mitpressjournals.org/doi/abs/10.1162/qss_a_00068?mobileUi=0), the dependency indicator proposed by the authors was empirically investigated. Since the results are central for the paper, the authors should discuss them in detail.
      Response: Thanks for providing this information. We have added some discussions on this paper in the “Related Research” section. We write:


    The idea of analyzing citation relations between publications that cite a focal publication has been explored in a number of earlier studies. Clough, Gollings, Loach, and Evans (2015) compared the number of citations given to a publication in a citation network with the number of citations given to the same publication in the transitive reduction of the citation network. According to Clough et al., the transitive reduction can be used to get ‘an indication that results in a paper were used across a wide number of fields’. Huang, Bu, Ding, and Lu (2018, 2020) analyzed so-called citing cascades, defined as the citation network of a focal publication and its citing publications. In particular, they studied citation relations between citing publications. The citation impact framework proposed in the current paper partly builds on the ideas explored by Huang et al.
    The notion of dependence introduced in our citation impact framework is closely related to the concepts of development and disruption proposed by Funk and Owen-Smith (2017) and used by Wu, Wang, and Evans (2019). Wu et al. investigated an indicator that provides a proxy of whether a publication tends to ‘disrupt’ or ‘develop’ science by taking into consideration the publication’s references and its citing publications, as well as the citations between all these publications. For a given focal publication, they defined ‘type i’ publications as those that cite the focal publication but not the references of the focal publication, ‘type j’ publications as those that cite both the focal publication and the references of the focal publication, and ‘type k’ publications as those that cite the references of the focal publication but not the focal publication itself. Based on content-level validation, expert interviews, and some other evaluations, the indicator adopted by Wu et al. showed a good performance in assessing the degree to which a publication ‘disrupts’ or ‘develops’ science. Yet, a follow-up study by Bornmann and Tekles (2019a) suggested that the length of the time window for calculating the disruptiveness of publications may affect the results. In addition, a case study by Bornmann and Tekles (2019b) questioned the ability of the disruptiveness indicator to identify disruptive publications in the journal Scientometrics. Furthermore, Bornmann, Devarakonda, Tekles, and Chacko (2020a, 2020b) compared the disruptiveness indicator with other related indicators, in particular those proposed by Wu and Yan (2019) and by an earlier version of the current paper. Bornmann et al. (2020a) argued that different indicators tend to represent similar dimensions.
    The notion of dependence introduced in the current paper is also related to the idea of originality proposed by Shibayama and Wang (2020). Both approaches consider the number of citations from the citing publications of a focal publication to its references.
    Building on the idea of citing cascades proposed by Huang et al. (2018), Mohapatra, Maiti, Bhatia, and Chakraborty (2019) introduced a method for pruning the citing cascade of a focal publication p. For each citing publication q of p, only the longest path between q and p is retained in the pruned network. Based on the pruned network, Mohapatra et al. defined several indicators, in particular depth (i.e., the length of the longest path between the focal publication and the leaf nodes in the network) and width (i.e., the maximum number of nodes at a given level in the network). They assumed that a publication has the most ‘influential’ impact when the values of depth and width are equal. This assumption lacks a clear conceptual foundation, but it was tested empirically using ‘Test of Time Awards’. Importantly, as will become clear in Section 5, the definitions of depth and breadth that we propose in the current paper are quite different from the definitions of depth and width introduced by Mohapatra et al.

    1. I miss a more extensive validation of the proposed indicators in the manuscript itself. In the current manuscript, only a case study is presented that is related to scientometrics. In this paper (https://www.nature.com/articles/s41586-019-0941-9), for example, the authors can find many possibilities of validation including comprehensive datasets.
      Response: Our focus is on introducing a coherent conceptual framework for the ideas of breadth, depth, dependence, and independence and on proposing a systematic approach for operationalizing these ideas. We empirically explore the resulting indicators, and we present a case study that offers a small-scale validation of these indicators. We acknowledge that there is room for a more extensive validation, but we do not consider this to be within the scope of our paper. In our revised paper, we explicitly mention the need for more extensive validation in the section on future research.

    2. Figures 1 and 2: Please illustrate clearer what the citing papers, the focal papers, and the cited references are.
      Response: Thanks. We now clearly explain what the citing papers, the focal paper, and the cited references are in the captions of Figures 1 and 2.

    3. Page 6, lines 3-5: The sentence is wrong. This paper (https://link.springer.com/article/10.1007/s11192-020-03406-8) does not deal with the influence of time on measuring disruption, but this paper (https://pdfs.semanticscholar.org/3a9f/f3f0e30a42156d7ae3584907fde150f48274.pdf). This paper (https://link.springer.com/article/10.1007%2Fs11192-019-03113-z) shows disruptive papers published in Scientometrics (measured by the initially proposed disruption indicator, DI, in https://www.nature.com/articles/s41586-019-0941-9). The case study shows that the DI does not seem to be able to identify disruptive papers published in Scientometrics; this follow-up paper (https://link.springer.com/article/10.1007%2Fs11192-020-03406-8) shows that the DI5 indicator - a variant of the DI indicator - seems to be better able to identify disruptive papers in Scientometrics.
      Response: Thank you for raising this important point. We have corrected the relevant sentences.

    4. Page 5, line 38: In my opinion, “related” is a too weak term. The indicators are very similar. The authors should write that their indicator is based on the earlier proposed DI indicator (https://www.nature.com/articles/s41586-019-0941-9).
      Response: We prefer not to use the expression ‘based on’, because we developed our ideas independently of the work done by others, so in that sense our ideas are not based on their work. To acknowledge the close connection between our ideas and those of others, we have replaced ‘related’ by ‘closely related’.

    5. Page 7, footnote 1: This is an important information which should be in the main text.
      Response: Thanks. This footnote has been moved to the main text.

    6. Page 9, line 3: “in field” should be “in the field”.
      Response: Corrected.

    7. Page 9: Please report the correlation coefficients. Furthermore, the authors report the top-10 list of papers for each indicator. In my opinion, an extensive discussion of these papers is essential in this study to receive an indication for the validity of the indicators: are the indicators able to identify the papers that they intend to identify? Thus, the top-10 papers should be presented in the main text, but not in the appendix. The complete bibliographic information should be given in the table (including title). Orient your discussion at this paper (https://link.springer.com/article/10.1007%2Fs11192-019-03113-z).
      Response: In the revised paper, we provide full bibliographic information for the top 10 publications. This makes it easier for the reader to interpret the results. We prefer to keep the table that contains the bibliographic information in the appendix. In our view the table is too big to be included in the main text. As suggested by the reviewer, we have added two tables in the appendix to report correlation coefficients. We have also added correlation coefficients to Figure 7. Furthermore, we have extended the discussion of the results obtained for the absolute indicators. In addition to the article by Egghe, we now also discuss in detail the article by Falagas and colleagues. In the revised paper, we also indicate more explicitly that the most substantial differences between the various perspectives provided by our indicators (i.e., breadth, depth, dependence, and independence) are obtained when a relative rather than an absolute viewpoint is taken. This is the reason why our analysis is more detailed for the relative indicators than for the absolute indicators.

    8. Pages 23 – 27: Formally cite the four papers which you discuss in the text. For example, it is not necessary to indicate papers in this complicated form: „Publication P1 is the article, co-authored by one of us“.
      Response: Thanks. This has been addressed.

    9. Page 26, line 41: “in field” should be “in the field”.
      Response: Corrected.

    10. Page 28: I miss a section with limitations of the new indicators. For example, for calculating the indicators, a considerable number of citations and/ or cited references is necessary. It does not make sense to calculate them for papers with only a few references and/ or citations. Thus, one might question to include the indicators into literature databases, since they are relevant (meaningful) for only a selected set of papers.
      Response: In the case of relative indicators, you are right that a considerable number of citations are necessary. However, in the case of absolute indicators, there is no need to have a large number of citations. We have in mind to use absolute indicators, not relative ones. However, this was not stated explicitly in the original paper. In the revised paper, this is stated explicitly.

    11. Section “future research”: Please outline why case studies should be done. What is the objective of doing further case studies? Please explain possible extensions in more detail. What do the authors expect from specific extensions? It is not clear how the indicators can be calculated on the aggregated level. Please exemplify this.
      Response: Thanks. More details have been offered at the end of this paper. We write:


    There are many directions for future research. First of all, additional case studies can be carried out to assess the usefulness and validity of our proposed multi-dimensional citation impact framework. Such case studies could also analyze how the proposed indicators change over time for individual publications, and how such changes relate to the accumulation of citations for a given focal publication. Also, the proposed framework can be extended in various ways, for instance by taking into account publication type (e.g., review articles), citation type (e.g., self-citations), and citation context (e.g., location in the full text of the citing publication). For instance, if we omit self-citations, how does this affect the values of the indicators? And how do the values of the indicators differ between review articles and regular articles?
    Ideas similar to the ones proposed in this paper can also be explored at aggregate levels rather than at the level of individual publications. For instance, based on the current framework, indicators of depth can be defined at the level of authors instead of publications. One approach could be to first determine the depth of each publication of an author and to then aggregate the outcomes from the publication level to the author level. Another approach could be to consider an author-author citation network and to determine the depth of an author based on this network.
    Finally, the distinction between cumulative research and more independent research can be studied in alternative ways. Research areas that are of a strongly cumulative nature for instance may be identified by searching for densely connected subnetworks in a citation network.

    Response to Reviewer #2

    Your suggestions have been very helpful to improve our paper. Please see our response below.

    1. This paper proposes several novel indicators: depth vs. breadth, independence vs dependence, based on to what extent the citing papers cite other citing or cited papers of a focal paper. This paper has a lot of things to like about, the indicators are novel and relevant, methodology is well and transparently documented, analyses and case studies are informative. However, I also have some suggestions.
      Response: Thanks. See our response below.

    2. First, the authors’ position is rather ambiguous regarding whether depth and breadth (and independence and dependence) are two different dimensions (position A) or two opposite sides of the same dimension (position B). Either conceptualization/position can be justified, but it seems that the authors are switching positions from time to time, which can cause confusion in understanding the paper and more importantly misinterpretation and misuse of the indicators for research evaluation. The authors did carefully discuss that they think depth and breadth are two separate dimensions and that they do not claim which one is “better.” So it seems that they subscribe to position A. However, other parts seem disagree.

    In terms of terminology, it makes perfect sense that “depth” and “breadth” are two different dimensions, but I cannot image how “dependence” and “independence” are two different dimensions.

    In terms of operationalization, the relative indicators (PCPs) are rather inconsistent with position A. TR and MR indicators are only for depth and dependence, but no equivalent indicators are developed for breadth or independence, seem incomplete if we follow position A.

    In terms of case study, the four selected cases categorize papers following position B. Following position A, the authors should report at least 2^4 cases.

    In summary, the choice between position A and B should be made more explict and consistent throughout the paper.
    Response: Apologies for the lack of clarity on this issue. The confusion is understandable because positions A and B are both applicable. Position A applies when an absolute perspective is taken (i.e., indicators scale with the number of citations of a publication), while position B applies when a relative perspective is taken (i.e., indicators are normalized for the number of citations of a publication). This was not explained in a sufficiently clear way in our original paper. In the revised paper, Sections 5.1 and 6.1 have been extended to provide a proper explanation of this issue. For instance, in Section 5.1 we have added the following explanation:


    To quantify the depth and breadth of the citation impact of a publication, we propose the six indicators summarized in Table 2. On the one hand, we distinguish between indicators of depth and indicators of breadth. On the other hand, we also make a distinction between absolute and relative indicators. Absolute indicators scale with the level of citation impact of a publication, while relative indicators are normalized for the level of citation impact. Relative indicators are defined only for publications that have received at least one citation (i.e., CP>0). From a relative point of view, depth and breadth are opposite concepts. A high depth implies a low breadth, and vice versa. Hence, when a relative perspective is taken, depth and breadth can be seen as two sides of the same coin. This is different when an absolute perspective is taken. From an absolute point of view, a publication may have both a high depth and a high breadth, or it may have both a low depth and a low breadth. This means that, from an absolute point of view, depth and breadth are conceptually distinct dimensions, even though they may be empirically correlated.

    1. Second, since these indicators are proposed for research evaluation, the authors need to give more examples and specific guidelines for how they can and should be used. The current discussion in the applications section seems to be like: We provide different indicators, feel free to choose the one that suit you best. This is prone to misuse. The authors should at least link these indicators to different scientific goals, and provide pointers for which indicators should be used for which purposes.
      Response: We are not proposing our indicators specifically for research evaluation. As explained in Section 8.2 in our paper, research evaluation is just one possible area in which our indicators may be useful. The focus of our paper is on introducing the concepts of (absolute and relative) depth, breadth, dependence, and independence and on showing how these concepts can be operationalized. Our ideas have not been developed as a direct solution to address practical problems in a specific area such as research evaluation. We see our conceptual framework and the associated indicators as a general tool that may be of use in various different areas, such as research evaluation, scientific literature search, and science of science research. To clarify this point, we have added the following paragraph in Section 8.2:


    We acknowledge that the above applications of the ideas presented in this paper require additional research. For specific applications, some of our proposed indicators may turn out to be more useful than others. The indicators may also require additional fine-tuning to optimize them for a specific use case.

    1. Third, on MR indicators, I understand that the numerator is the total number of links (not involving the focal publication) in Figure 1 and 2, and the denominator is the total number of nodes (except the focal publication), is that correct? I appreciate that the authors tried really hard to make it clear, but somehow the current description is rather difficult to comprehend. Presenting it using different terminologies (e.g., in network terms) might help. Furthermore, why not use the total number of possible links as the denominator? Wouldn’t that give a measure ranging between 0 and 1, which is more fitting for a “relative” indicator and more consistent with other relative indicators in the paper. In addition, this measure seems very much like the originality measure in:

    Shibayama, S., & Wang, J. (2020). Measuring originality in science. Scientometrics, 122(1), 409-427.
    Response: There are two MR indicators, namely MR[citing pub] and MR[cited pub]. It is not clear to which of the two indicators the reviewer refers. For both indicators, the denominator is the total number of publications that cite the focal publication (so not the total number of publications in the figure, because that may also include publications cited by the focal publication). The definition of the numerator is different for the two MR indicators. The numerator does not consider all links in the figure, but only links given by citing publications (either to other citing publications or to cited publications, depending on which of the two MR indicators is considered). We have carefully reread the way in which we explain the two MR indicators. Our impression is that the explanations are sufficiently clear. We do not see any obvious ways to improve them. If the reviewer has suggestions on how we can improve the explanations, we would very much welcome this. However, we prefer not to use network science terminology. We have avoided the use of this terminology in order to make sure that our paper is accessible also for scientometricians that do not have a background in network science.

    Using the total number of possible links in the denominator would result in different types of indicators that in our view do not match well with the concepts that we try to capture using our indicators.

    1. I suggest some further analyses:

    First, provide the correlation matrix between all indicators for the whole dataset as well as for the subset of bibliometric papers. It is important to discuss how these indicators are related to each other and what added information value each indicator brings.
    Response: As suggested by the reviewer, we have added two tables in the appendix to report correlation coefficients. We have also added correlation coefficients to Figure 7.
    5. Second, examine how the indicators involves over time and carefully provide guidelines for research evaluation in terms of the choice of citation time windows. I suspect that the breadth and independence is increasing over time, while depth and dependence is decreasing over time. After many years, the citation links between the current citing paper and old cited paper might get rather loose, and it is questionable whether such citation links are valuable for inferring depth/breadth or dependence/independence.
    Response: We are indeed also interested in this. However, adding the analysis proposed by the reviewer would substantially increase the length of our paper, while the paper is already quite lengthy. We therefore consider this to be outside the scope of our paper. We have mentioned this as a topic for future research.

    1. Third, discussions on the four example papers are very interesting. It provides detailed idiosyncratic explanations, but at the expense of generality. I would suggest to look into the papers with the highest and lowest scores for each dimension/indicator and try to find some general patterns.
      Response: In our revised paper, we provide a somewhat more extensive discussion of our results for the absolute indicators. We do not think there is room in the paper to further extend the discussion of the results. As an alternative, we have made the results openly available in a spreadsheet file uploaded to Zenodo. In this way, we enable interested readers to explore our results in more detail themselves.

    2. Some other points:

    Table 1 and figure should better be reported for the whole dataset rather than the subset of papers with at least 100 citations. Or at least, the sampling ratio should be reported.
    Response: Relative indicators provide meaningful results only for publications that have received a substantial number of citations. This is why our analysis focuses on publications that have received at least 100 citations. In our original paper, this was not properly explained. In the revised paper, we have added an explanation.

    1. Page 10. Paper A introduces an innovative new idea and has a deep impact, paper B introduces a new software tool and has a broad impact. I find this illustration very intriguing and plausible. However, do you want to make some general claims without any empirical evidence? It is also possible that a paper contributes a new idea, which is then applied in various different and separated fields, leading to a broad impact. another paper develops a very specialized tool which is only useful within the small specialty, leading to a deep impact.
      Response: We provide examples of scenarios that can be analyzed using our proposed framework for characterizing the citation impact of publications. One of these examples is the scenario of a software tool with a broad but not so deep citation impact. Although the results reported in Subsection 7.2 offer some support for this scenario (see the discussion of publication P1), we do not claim that the examples we provide are necessarily realistic. We agree with the reviewer that this requires further empirical research. In the future research section in our revised paper, we mention the need for additional research to assess the validity of our proposed framework.

    2. Page 13. “This seems to suggest that PSE research is of a stronger cumulative nature than MCS and SSH research.” What do you mean by “cumulative nature”? It seems to be about degree of specialization and separation between specialties to me.
      Response: In the revised paper, we introduce the word ‘cumulative’ in Section 1, when we also introduce the distinction between deep and broad citation impact.

    3. Page 26. “It is sometimes suggested that researchers tend to cite review articles instead of citing the underlying original works, but the high dependence of P3 shows that this is not the case for P3.” This is rather suboptimal, since you do have the data, why not show the overall distribution by document types?
      Response: This distinction between research articles and review articles in Web of Science is not reliable (see https://doi.org/10.1007/s11192-012-0738-1). Therefore, we prefer not to rely on this distinction in our analyses.



    Cite this author response
  • pre-publication peer review (ROUND 1)
    Decision Letter
    2020/08/28

    28-Aug-2020

    Dear Dr. Bu:

    Manuscript ID QSS-2020-0063 entitled "A multi-dimensional framework for characterizing the citation impact of scientific publications" which you submitted to Quantitative Science Studies, has been reviewed. The comments of the reviewers are included at the bottom of this letter. Both reviewers agree on the merit of your manuscript. However, they also suggest some revisions to be performed on your manuscript. I therefore invite you to respond to the reviewers' comments and revise your manuscript.

    To revise your manuscript, log into https://mc.manuscriptcentral.com/qss and enter your Author Center, where you will find your manuscript title listed under "Manuscripts with Decisions." Under "Actions," click on "Create a Revision." Your manuscript number has been appended to denote a revision.

    You may also click the below link to start the revision process (or continue the process if you have already started your revision) for your manuscript. If you use the below link you will not be required to login to ScholarOne Manuscripts.

    PLEASE NOTE: This is a two-step process. After clicking on the link, you will be directed to a webpage to confirm.

    https://mc.manuscriptcentral.com/qss?URL_MASK=07f2805676e145dc9fa998fe11228f83

    You will be unable to make your revisions on the originally submitted version of the manuscript. Instead, revise your manuscript using a word processing program and save it on your computer. Please also highlight the changes to your manuscript within the document by using the track changes mode in MS Word or by using bold or colored text.

    Once the revised manuscript is prepared, you can upload it and submit it through your Author Center.

    When submitting your revised manuscript, you will be able to respond to the comments made by the reviewers in the space provided. You can use this space to document any changes you make to the original manuscript. In order to expedite the processing of the revised manuscript, please be as specific as possible in your response to the reviewers.

    IMPORTANT: Your original files are available to you when you upload your revised manuscript. Please delete any redundant files before completing the submission.

    If possible, please try to submit your revised manuscript by 26-Dec-2020. Let me know if you need more time to revise your work.

    Once again, thank you for submitting your manuscript to Quantitative Science Studies and I look forward to receiving your revision.

    Best wishes,
    Prof. Vincent Larivière
    Associate Editor
    Quantitative Science Studies
    vincent.lariviere@umontreal.ca

    Reviewer: 1

    Comments to the Author
    The authors propose the concept of measuring citation impact multi-dimensionally by introducing two indicators. This is a very interesting study that should be published in QSS. In a revision of the manuscript, the following points should be considered:

    General comments:
    1) In this paper (https://www.mitpressjournals.org/doi/abs/10.1162/qss_a_00068?mobileUi=0), the dependency indicator proposed by the authors was empirically investigated. Since the results are central for the paper, the authors should discuss them in detail.
    2) I miss a more extensive validation of the proposed indicators in the manuscript itself. In the current manuscript, only a case study is presented that is related to scientometrics. In this paper (https://www.nature.com/articles/s41586-019-0941-9), for example, the authors can find many possibilities of validation including comprehensive datasets.

    Specific comments:
    3) Figures 1 and 2: Please illustrate clearer what the citing papers, the focal papers, and the cited references are.
    4) Page 6, lines 3-5: The sentence is wrong. This paper (https://link.springer.com/article/10.1007/s11192-020-03406-8) does not deal with the influence of time on measuring disruption, but this paper (https://pdfs.semanticscholar.org/3a9f/f3f0e30a42156d7ae3584907fde150f48274.pdf). This paper (https://link.springer.com/article/10.1007%2Fs11192-019-03113-z) shows disruptive papers published in Scientometrics (measured by the initially proposed disruption indicator, DI, in https://www.nature.com/articles/s41586-019-0941-9). The case study shows that the DI does not seem to be able to identify disruptive papers published in Scientometrics; this follow-up paper (https://link.springer.com/article/10.1007%2Fs11192-020-03406-8) shows that the DI5 indicator - a variant of the DI indicator - seems to be better able to identify disruptive papers in Scientometrics.
    5) Page 5, line 38: In my opinion, “related” is a too weak term. The indicators are very similar. The authors should write that their indicator is based on the earlier proposed DI indicator (https://www.nature.com/articles/s41586-019-0941-9).
    6) Page 7, footnote 1: This is an important information which should be in the main text.
    7) Page 9, line 3: “in field” should be “in the field”.
    8) Page 9: Please report the correlation coefficients. Furthermore, the authors report the top-10 list of papers for each indicator. In my opinion, an extensive discussion of these papers is essential in this study to receive an indication for the validity of the indicators: are the indicators able to identify the papers that they intend to identify? Thus, the top-10 papers should be presented in the main text, but not in the appendix. The complete bibliographic information should be given in the table (including title). Orient your discussion at this paper (https://link.springer.com/article/10.1007%2Fs11192-019-03113-z).
    9) Pages 23 – 27: Formally cite the four papers which you discuss in the text. For example, it is not necessary to indicate papers in this complicated form: „Publication P1 is the article, co-authored by one of us“.
    10) Page 26, line 41: “in field” should be “in the field”.
    11) Page 28: I miss a section with limitations of the new indicators. For example, for calculating the indicators, a considerable number of citations and/ or cited references is necessary. It does not make sense to calculate them for papers with only a few references and/ or citations. Thus, one might question to include the indicators into literature databases, since they are relevant (meaningful) for only a selected set of papers.
    12) Section “future research”: Please outline why case studies should be done. What is the objective of doing further case studies? Please explain possible extensions in more detail. What do the authors expect from specific extensions? It is not clear how the indicators can be calculated on the aggregated level. Please exemplify this.

    Reviewer: 2

    Comments to the Author
    This paper proposes several novel indicators: depth vs. breadth, independence vs dependence, based on to what extent the citing papers cite other citing or cited papers of a focal paper. This paper has a lot of things to like about, the indicators are novel and relevant, methodology is well and transparently documented, analyses and case studies are informative. However, I also have some suggestions.

    First, the authors’ position is rather ambiguous regarding whether depth and breadth (and independence and dependence) are two different dimensions (position A) or two opposite sides of the same dimension (position B). Either conceptualization/position can be justified, but it seems that the authors are switching positions from time to time, which can cause confusion in understanding the paper and more importantly misinterpretation and misuse of the indicators for research evaluation. The authors did carefully discuss that they think depth and breadth are two separate dimensions and that they do not claim which one is “better.” So it seems that they subscribe to position A. However, other parts seem disagree.

    In terms of terminology, it makes perfect sense that “depth” and “breadth” are two different dimensions, but I cannot image how “dependence” and “independence” are two different dimensions.

    In terms of operationalization, the relative indicators (PCPs) are rather inconsistent with position A. TR and MR indicators are only for depth and dependence, but no equivalent indicators are developed for breadth or independence, seem incomplete if we follow position A.

    In terms of case study, the four selected cases categorize papers following position B. Following position A, the authors should report at least 2^4 cases.

    In summary, the choice between position A and B should be made more explict and consistent throughout the paper.

    Second, since these indicators are proposed for research evaluation, the authors need to give more examples and specific guidelines for how they can and should be used. The current discussion in the applications section seems to be like: We provide different indicators, feel free to choose the one that suit you best. This is prone to misuse. The authors should at least link these indicators to different scientific goals, and provide pointers for which indicators should be used for which purposes.

    Third, on MR indicators, I understand that the numerator is the total number of links (not involving the focal publication) in Figure 1 and 2, and the denominator is the total number of nodes (except the focal publication), is that correct? I appreciate that the authors tried really hard to make it clear, but somehow the current description is rather difficult to comprehend. Presenting it using different terminologies (e.g., in network terms) might help. Furthermore, why not use the total number of possible links as the denominator? Wouldn’t that give a measure ranging between 0 and 1, which is more fitting for a “relative” indicator and more consistent with other relative indicators in the paper. In addition, this measure seems very much like the originality measure in:

    Shibayama, S., & Wang, J. (2020). Measuring originality in science. Scientometrics, 122(1), 409-427.

    I suggest some further analyses:

    First, provide the correlation matrix between all indicators for the whole dataset as well as for the subset of bibliometric papers. It is important to discuss how these indicators are related to each other and what added information value each indicator brings.

    Second, examine how the indicators involves over time and carefully provide guidelines for research evaluation in terms of the choice of citation time windows. I suspect that the breadth and independence is increasing over time, while depth and dependence is decreasing over time. After many years, the citation links between the current citing paper and old cited paper might get rather loose, and it is questionable whether such citation links are valuable for inferring depth/breadth or dependence/independence.

    Third, discussions on the four example papers are very interesting. It provides detailed idiosyncratic explanations, but at the expense of generality. I would suggest to look into the papers with the highest and lowest scores for each dimension/indicator and try to find some general patterns.

    Some other points:

    Table 1 and figure should better be reported for the whole dataset rather than the subset of papers with at least 100 citations. Or at least, the sampling ratio should be reported.

    Page 10. Paper A introduces an innovative new idea and has a deep impact, paper B introduces a new software tool and has a broad impact. I find this illustration very intriguing and plausible. However, do you want to make some general claims without any empirical evidence? It is also possible that a paper contributes a new idea, which is then applied in various different and separated fields, leading to a broad impact. another paper develops a very specialized tool which is only useful within the small specialty, leading to a deep impact.

    Page 13. “This seems to suggest that PSE research is of a stronger cumulative nature than MCS and SSH research.” What do you mean by “cumulative nature”? It seems to be about degree of specialization and separation between specialties to me.

    Page 26. “It is sometimes suggested that researchers tend to cite review articles instead of citing the underlying original works, but the high dependence of P3 shows that this is not the case for P3.” This is rather suboptimal, since you do have the data, why not show the overall distribution by document types?

    Decision letter by
    Cite this decision letter
    Reviewer report
    2020/08/28

    This paper proposes several novel indicators: depth vs. breadth, independence vs dependence, based on to what extent the citing papers cite other citing or cited papers of a focal paper. This paper has a lot of things to like about, the indicators are novel and relevant, methodology is well and transparently documented, analyses and case studies are informative. However, I also have some suggestions.

    First, the authors’ position is rather ambiguous regarding whether depth and breadth (and independence and dependence) are two different dimensions (position A) or two opposite sides of the same dimension (position B). Either conceptualization/position can be justified, but it seems that the authors are switching positions from time to time, which can cause confusion in understanding the paper and more importantly misinterpretation and misuse of the indicators for research evaluation. The authors did carefully discuss that they think depth and breadth are two separate dimensions and that they do not claim which one is “better.” So it seems that they subscribe to position A. However, other parts seem disagree.

    In terms of terminology, it makes perfect sense that “depth” and “breadth” are two different dimensions, but I cannot image how “dependence” and “independence” are two different dimensions.

    In terms of operationalization, the relative indicators (PCPs) are rather inconsistent with position A. TR and MR indicators are only for depth and dependence, but no equivalent indicators are developed for breadth or independence, seem incomplete if we follow position A.

    In terms of case study, the four selected cases categorize papers following position B. Following position A, the authors should report at least 2^4 cases.

    In summary, the choice between position A and B should be made more explict and consistent throughout the paper.

    Second, since these indicators are proposed for research evaluation, the authors need to give more examples and specific guidelines for how they can and should be used. The current discussion in the applications section seems to be like: We provide different indicators, feel free to choose the one that suit you best. This is prone to misuse. The authors should at least link these indicators to different scientific goals, and provide pointers for which indicators should be used for which purposes.

    Third, on MR indicators, I understand that the numerator is the total number of links (not involving the focal publication) in Figure 1 and 2, and the denominator is the total number of nodes (except the focal publication), is that correct? I appreciate that the authors tried really hard to make it clear, but somehow the current description is rather difficult to comprehend. Presenting it using different terminologies (e.g., in network terms) might help. Furthermore, why not use the total number of possible links as the denominator? Wouldn’t that give a measure ranging between 0 and 1, which is more fitting for a “relative” indicator and more consistent with other relative indicators in the paper. In addition, this measure seems very much like the originality measure in:

    Shibayama, S., & Wang, J. (2020). Measuring originality in science. Scientometrics, 122(1), 409-427.

    I suggest some further analyses:

    First, provide the correlation matrix between all indicators for the whole dataset as well as for the subset of bibliometric papers. It is important to discuss how these indicators are related to each other and what added information value each indicator brings.

    Second, examine how the indicators involves over time and carefully provide guidelines for research evaluation in terms of the choice of citation time windows. I suspect that the breadth and independence is increasing over time, while depth and dependence is decreasing over time. After many years, the citation links between the current citing paper and old cited paper might get rather loose, and it is questionable whether such citation links are valuable for inferring depth/breadth or dependence/independence.

    Third, discussions on the four example papers are very interesting. It provides detailed idiosyncratic explanations, but at the expense of generality. I would suggest to look into the papers with the highest and lowest scores for each dimension/indicator and try to find some general patterns.

    Some other points:

    Table 1 and figure should better be reported for the whole dataset rather than the subset of papers with at least 100 citations. Or at least, the sampling ratio should be reported.

    Page 10. Paper A introduces an innovative new idea and has a deep impact, paper B introduces a new software tool and has a broad impact. I find this illustration very intriguing and plausible. However, do you want to make some general claims without any empirical evidence? It is also possible that a paper contributes a new idea, which is then applied in various different and separated fields, leading to a broad impact. another paper develops a very specialized tool which is only useful within the small specialty, leading to a deep impact.

    Page 13. “This seems to suggest that PSE research is of a stronger cumulative nature than MCS and SSH research.” What do you mean by “cumulative nature”? It seems to be about degree of specialization and separation between specialties to me.

    Page 26. “It is sometimes suggested that researchers tend to cite review articles instead of citing the underlying original works, but the high dependence of P3 shows that this is not the case for P3.” This is rather suboptimal, since you do have the data, why not show the overall distribution by document types?

    Reviewed by
    Cite this review
    Reviewer report
    2020/08/12

    The authors propose the concept of measuring citation impact multi-dimensionally by introducing two indicators. This is a very interesting study that should be published in QSS. In a revision of the manuscript, the following points should be considered:

    General comments:
    1) In this paper (https://www.mitpressjournals.org/doi/abs/10.1162/qss_a_00068?mobileUi=0), the dependency indicator proposed by the authors was empirically investigated. Since the results are central for the paper, the authors should discuss them in detail.
    2) I miss a more extensive validation of the proposed indicators in the manuscript itself. In the current manuscript, only a case study is presented that is related to scientometrics. In this paper (https://www.nature.com/articles/s41586-019-0941-9), for example, the authors can find many possibilities of validation including comprehensive datasets.

    Specific comments:
    3) Figures 1 and 2: Please illustrate clearer what the citing papers, the focal papers, and the cited references are.
    4) Page 6, lines 3-5: The sentence is wrong. This paper (https://link.springer.com/article/10.1007/s11192-020-03406-8) does not deal with the influence of time on measuring disruption, but this paper (https://pdfs.semanticscholar.org/3a9f/f3f0e30a42156d7ae3584907fde150f48274.pdf). This paper (https://link.springer.com/article/10.1007%2Fs11192-019-03113-z) shows disruptive papers published in Scientometrics (measured by the initially proposed disruption indicator, DI, in https://www.nature.com/articles/s41586-019-0941-9). The case study shows that the DI does not seem to be able to identify disruptive papers published in Scientometrics; this follow-up paper (https://link.springer.com/article/10.1007%2Fs11192-020-03406-8) shows that the DI5 indicator - a variant of the DI indicator - seems to be better able to identify disruptive papers in Scientometrics.
    5) Page 5, line 38: In my opinion, “related” is a too weak term. The indicators are very similar. The authors should write that their indicator is based on the earlier proposed DI indicator (https://www.nature.com/articles/s41586-019-0941-9).
    6) Page 7, footnote 1: This is an important information which should be in the main text.
    7) Page 9, line 3: “in field” should be “in the field”.
    8) Page 9: Please report the correlation coefficients. Furthermore, the authors report the top-10 list of papers for each indicator. In my opinion, an extensive discussion of these papers is essential in this study to receive an indication for the validity of the indicators: are the indicators able to identify the papers that they intend to identify? Thus, the top-10 papers should be presented in the main text, but not in the appendix. The complete bibliographic information should be given in the table (including title). Orient your discussion at this paper (https://link.springer.com/article/10.1007%2Fs11192-019-03113-z).
    9) Pages 23 – 27: Formally cite the four papers which you discuss in the text. For example, it is not necessary to indicate papers in this complicated form: „Publication P1 is the article, co-authored by one of us“.
    10) Page 26, line 41: “in field” should be “in the field”.
    11) Page 28: I miss a section with limitations of the new indicators. For example, for calculating the indicators, a considerable number of citations and/ or cited references is necessary. It does not make sense to calculate them for papers with only a few references and/ or citations. Thus, one might question to include the indicators into literature databases, since they are relevant (meaningful) for only a selected set of papers.
    12) Section “future research”: Please outline why case studies should be done. What is the objective of doing further case studies? Please explain possible extensions in more detail. What do the authors expect from specific extensions? It is not clear how the indicators can be calculated on the aggregated level. Please exemplify this.

    Reviewed by
    Cite this review
All peer review content displayed here is covered by a Creative Commons CC BY 4.0 license.