Content of review 1, reviewed on July 17, 2017

The authors of this paper analysed the publication output indexed in WoS for Portuguese research units (293 Research Units and 26 Associated laboratories) funded by the Portuguese Government. The results showed that “in general, units that had better peer ratings, and thus more funding, as well as an increased capacity to attract extra funding, were not necessarily those that ended up producing more excellent research”. This is not a novel result. Some authors already showed that some research units showed a higher scientific impact than some Associated laboratories (Cabral & Pechincha, 2013). However, Ramos and Sarrico (2013) did not cite their study. Both studies have one limitation. They have not used Scopus data base that has a much higher coverage of social sciences publications. This was recognized in a comparative scientometric assessment of the results of ERC funded projects that choosed the google scholar for the coverage of humanities literature (Robitaille et al., 2015). And also confirmed by Mongeon & Paul-Hus (2016). As a consequence Ramos and Sarrico (2013) excluded from the study the Associated laboratories (Ciências Sociais and the Centro de Estudos Sociais). Ramos and Sarrico (2013) also excluded two more Associated Laboratories (INESC—Lisboa, and Instituto de Telecomunicacões) for having a WoS coverage below 60%. Their explanation has to do with the fact that those two Alabs are centered on Electrical & Electronic Engineering, Telecommunications, and Computer Science for which conference papers are important but not indexed in WoS or indexed in WoS but not included by CWTS in the bibliometric study. It’s a pitty that the study did not analysed the cost/publication because when assessing research units with very different budget its important to compare this metric. This is important because the 26 ALabs received more than 50% of the Portuguese science budget while the remaining 293 research units receive the rest of the budget. And its specially important because in a context in which research teams with higher budgets can get higher publishing rates by publishing in journals like Plos One that have high article processing charges but also have a light peer review. Cabral, J. A. S., & Pechincha, P. (2013). Investigação U. Porto _Laboratórios Associados e Unidades de Iamp; D Indicadores bibliométricos CWTS _ Universidade de Leiden/FCT. Mongeon, P., & Paul-Hus, A. (2016). The journal coverage of Web of Science and Scopus: a comparative analysis. Scientometrics, 106(1), 213-228. Robitaille, J.P; Macaluso, B.; Pollitt, A.; Gunashekar,S.; Larivière, V. (2015) Comparative scientometric assessment of the results of ERC-funded projects. Bibliometric assessment report (D5). ERCEA/A1/2014/20 Under the Framework Contract: ‘Provision of Services in the Fields of Research Evaluation and Research Policy Analysis’ [No 2010/S 172-262618], Brussels

Source

    © 2017 the Reviewer (CC BY 4.0).

References

    Ana, R., S., S. C. 2016. Past performance does not guarantee future results: lessons from the evaluation of research units in Portugal. Research Evaluation.