Training the next generation of peer reviewers is essential to maintaining the quality and integrity of published research. That's why we developed the free online Publons Academy, which arms early-career researchers with the core competencies they need to review and provides a useful stepping stone to connect reviewers like Software Engineering Researcher, Melina Vidoni, with editors such as Markus Borg of the e-Informatica Software Engineering Journal.
In a recent Publons survey with ~12,000 researchers, 87 per cent of respondents said peer review training was either “extremely important” or “important” to ensure high quality of peer review. Despite this, over 75 per cent of these researchers had either never received peer review training or learned the skills themselves through self-selected reading.*
This lack of training can be detrimental to the research ecosystem as a whole. Peer review is an essential bulwark in maintaining accuracy and trust in scholarly communication. Researchers and the public alike rely on peer reviewers' critical analysis skills for knowing what research is sound and citable versus potentially unreproducible.
Furthermore, it’s integral to the professional development of all researchers. It helps scholars stay abreast of the latest research trends in their field, it improves their writing and analysis skills, and it helps them spot and avoid common flaws in their own manuscripts.
Peer reviewers can learn these skills “on the job” - but at what cost?
Many respondents in our survey admitted their first review experience came from co-authoring manuscripts with their supervisor, or by waiting for an editor to reach out to them after publication. Unfortunately, this is causing cracks in the peer review system. While early career researchers struggle to forge those critically important relationships with editors in their fields, editors, in turn, are struggling to fill their reviewer pools with talented, motivated, and available researchers.
Melina Vidoni, a postdoc at INGAR CONICET-UTN, in Santa Fe, and Teaching Assistant at Argentina’s Universidad Tecnológica Nacional, completed our free online Publons Academy in January this year. It was here she learned the fundamental aspects of peer review, including how to construct a peer review report, what’s involved in the process, and the common flaws to look out for.
We then introduced Melina to an editor of e-Informatica Software Engineering Journal, Markus Borg, of Lund University in Sweden. Markus helped put her skills into practice by inviting her to peer review for the journal.
Here’s how it went…
Publons: How would you describe your peer review experience before starting the Publons Academy?
Melina: Any knowledge I had about reviewing was only from the author perspective and obtained ad-hoc through my own submission experiences. I felt I lacked experience writing reviews so I asked my advisor for assistance. As a result, he co-reviewed a manuscript with me.
What drew you to our online peer review training course?
Melina: I hoped to learn the process, structure and implicit guidelines of academic reviews. I wanted to understand it not only as a reviewer, but also as a fellow aiming to publish scientific research. I valued the possibility of practising this during the course and having my reviews assessed by an expert with experience in reviewing.
What was the most valuable learning you took from the Publons Academy and why?
Melina: Learning about the different types of flaws in an article was really valuable. Knowing these flaws simplifies the process of assessing any manuscript, pointing directly to its strengths and weaknesses, and helping to decide which information to present to the authors, with which relevance and order. At the same time, this new knowledge of peer review has contributed to improving my own manuscripts as well: I can now think about the different types of article flaws from the perspective of a reviewer and author. Also, being critical of one’s own research leads to self-improvement.
“This new knowledge of peer review has contributed to improving my own manuscripts as well: I can now think about the different types of article flaws from the perspective of a reviewer and author.”
What were your initial thoughts about inviting a Publons Academy graduate to review for you?
Markus: I hadn’t heard about the Publons Academy before I was contacted. It sounded like a great idea and I was happy to give it a try. Reviewers are in high demand in software engineering.
Have you invited an early-career researcher for peer review before? How did that go?
Markus: Yes, I’ve invited early career researchers a few times before. I’ve only done it when I knew the junior reviewer’s expertise was perfect to assess the manuscript under submission, and only after also other senior researchers had agreed to review the same manuscript. The reviews I received were alright, especially regarding the technical details, but I suppose I expected higher quality reviews. On the other hand, a lot of practice is needed to become a great reviewer. Also, learning to navigate the body of related work takes years, and stepping back to consider the big picture is not easy for early career researchers. Still, I will surely continue to invite junior reviewers when I know that they are experts on the particular topic – especially if I’ve met them in person.
“Learning to navigate the body of related work takes years, and stepping back to consider the big picture is not easy for early career researchers.”
How did you find working as an editor with Publons Academy graduate Melina Vidoni when she reviewed for the e-informatica Software Engineering Journal?
Markus: I’ve got nothing but good things to say. I received a brilliant review within a few days – I’ve never received a high-quality review that fast before. Not only was the review well-written and highly useful in the decision process, I especially remember the effective structure of the review itself – I surely learned something from the review.
Melina, how did you find the peer review experience for e-informatica Software Engineering Journal? Did you feel up to the challenge?
Melina: The journal’s reviewing process was straightforward, and the template they provided fitted with what was taught and practiced in the Publons Academy. Using the guidelines offered in the Academy made the peer review process seamless, and also showed that the Academy is up-to-date, providing good knowledge to start as a reviewer. Using the knowledge I acquired from the Publons Academy was fundamental, as it changed my perspective in reviewing and gave me the ability to give back to the scientific community through peer-review.
“Using the knowledge I acquired from the Publons Academy was fundamental, as it changed my perspective in reviewing and gave me the ability to give back to the scientific community through peer-review.”
Do you approach reviewing differently now, and how so?
Melina: Yes, I do. I follow the reviewing structure taught in the course, checking that the manuscript provides new knowledge without any fatal flaws, and also assessing its integrity –i.e. how the problem presented in the Introduction is solved throughout the paper. Knowing defined guidelines on what to do and what not to do while reviewing increased my confidence, and helped me to create a personal, organised approach for reviewing.
“Knowing defined guidelines on what to do and what not to do while reviewing increased my confidence, and helped me to create a personal, organised approach for reviewing.”
Markus, Do you have any tips for early career researchers wanting to get into reviewing or improving their peer review skills?
Markus: Analogous to when you’re looking for a new job, let people know that you are available and willing to review papers. In software engineering, reviewing for workshops is the obvious entry point – and a very good learning experience. In general, both editors and conference/workshop organizers have a hard time finding the reviewers they need. There are so many papers around in our field. Furthermore, once you get into reviewing I would strongly recommend carefully reading other reviewers’ opinions about the papers. There is a lot to learn from comparing reviews. Perhaps you can even ask for feedback on your review?
Finally, there are many useful online resources on how to write good reviews. Many researchers have written blog posts on peer review. Consider advice also presented by researchers in other fields than your own – it can really bring some perspective! Software engineering is much inspired by the rigor in evidence-based medicine, and the qualitative research methods we use have been adapted from social science. Exploring the review recommendations in related areas can be both interesting and rewarding.
Thanks, Melina and Markus!
**The statistics noted in this blog are from Publons’ large-scale survey, the results of which were released as part of our Global State of Peer Review report.