Go to publons.com

Why would we want another way of evaluating scientific research?

This is the first in a series of blog posts from the Publons R&D team, who are looking to investigate and experiment with superior methods of scientific research evaluation. See the other posts in this series here.

Scientists evaluate the work of their peers via the peer-review process to ensure that the work is sound. A second layer of evaluation is added by publication itself. Over the long run the relative importance of a paper is assumed to be proportional to the number of citations it receives. The future citation count of a publication can be estimated more immediately by the impact factor [1] of the journal it is published in.

The impact factor of a journal for a given year (e.g. 2011) is calculated from all the citations and articles from the previous two years (2010 and 2009). The total number of citations is divided by the number of articles to give impact factors ranging from 0 to 52.28 [2].

It is important to remember that the reason for these metrics is because they allow us to (indirectly) measure the value of the contributions a researcher makes to their field.

You tend to optimize what you can measure. One of the drawbacks of these metrics is that they tend to encourage gratuitous citations. Among academics and journals there is an unwritten practice of self-citation, and preferential citation of newer papers (to show that these papers are at 'the cutting edge').

This latter type of citation is linked to the definition of impact: the impact of a paper is not solely related to the quality of the science contained within; it is also a factor of the relevance of the paper to contemporary work. The impact factor is thus a convoluted measure of both quality and context.

One way of establishing whether quality or context is the dominant effect is by examining how the average age of the papers cited by publications in a given journal (known as its 'citation half-life') changes with impact factor. The following schematic qualitatively illustrates the relationship one could expect if context dominates impact factor (red), or if quality dominates impact factor (green). If quality alone is the basis of citation, the impact factor scales with increased citation lifetime - higher quality papers will continue to be cited, even as they get older. If context is the primary basis of citation, the impact factor reflects progress with greater citation preference for newer papers.


These data are published every two years as part of the Journal Citation Reports (JCR). We took a look at the six physical sciences disciplines mentioned in the 2011 report [3]. The following figure shows the aggregate impact factor of journals within a particular subject area (e.g. physics) plotted against the cited half life of papers. The aggregate impact factor is the average for all journals within that subject category and the cited half life is the median age of the cited articles.


There is clearly a negative correlation between citation half-life and impact factor; papers lose context faster in fields where the average impact factor is high. Therefore the impact factor measures context more than quality.

In other words, the number of papers you publish in high impact factor journals are not a true judge of the quality of your work. Here at Publons, we think it is time for the academic community to find a better way of evaluating the quality and context of your research.

Stay tuned for details.


[1] The impact factor was developed by the Institute of Scientific Information (ISI).

[2] The Annual Review of Immunology had an impact factor of 52.28 in 2011.

[3] http://admin-apps.webofknowledge.com/JCR/static_html/notices/notices.htm

comments powered by Disqus

Subscribe to our mailing list

* indicates required