Content of review 1, reviewed on July 10, 2013

Significance Comment

Novelty

The first thing to say is that publons is to be welcomed. It's part of an evolving community of ideas around revolutionising academic publishing that has been in the making for a long time. It's a pretty exciting time right now - less so for the incumbent BIg Publishers, more so for researchers in the area and a number of interesting startups and projects around reshaping the journal.

The second thing to say is that where we are in the academic publishing industry is due to a whole set of pressures that serve to empower the incumbents and disable the forces of change. While this paper correctly says that Open Access is not the solution, is it however one of the levers that are being pulled that will open the door to a whole wave of innovation - mostly centered on startups, and their investors, that see opportunities that Big Publishing can't see. Open Access is good - not for editors, reviewers, authors, instituions or funders since there is effectively no real change to the status quo - but because of the fact that it opens up what was previously hidden away in a process that has been the same for several centuries. Now the discussion is alive about who pays, and for what, and what the value delivered is; where the labour comes from, and who pays for that; and, crucially, where are the next innovations going to come from. Open Access will prove bad for Big Publishing however, because it signals the start of the end for the traditional journal.

Publons, as discussed in this paper, is about fixing the lack of transparency in academic publishing by changing peer review from a closed, hidden process to an open one. Just as crowdsourcing taps into the opinions of the crowd, and crowdfunding leverages the economic contributions of many, Publons aims to open up peer review. "Peer review is flawed" says this article, and it may be right - but it might also be that this is not the argument that is the most powerful to motivate change - or the one that will create change in a direction that will significantly improve the way academic publishing works.

There are lots of issues to examine here, but I'm going to start with what Andrew says is wrong with peer review:

  1. Only one or two experts determine the validity of a paper

  2. The review process takes too long

  3. Peer review is not transparent

Only one or two experts determine the validity of a paper. This might be true or it might not, depending on what journal you look at. For example, in the journal I run there may only be two or three peer reviews for a given paper, but each paper passes through the hands of an editor-in-chief, an editor, possibly an associate editor and all of these people input into the process of taking a submission and turning it into a published article. But to cast this as a criticism devalues 'expertise': in reality most papers that our editors see will be already known to them - they may have been presented at conferences, discussed at seminars, been presented at workshops - and in any case, an experienced editor will be aware of the major research themes, concentrations of expertise in research groups or projects or will have read other papers around the topic. Editors, and the reviewers they use, have lots of expertise and it's easy to discount it. The real issue here, I think, is leveraging the network to enhance the process. So, for example, we have the idea of reviewing reviewers, rating reviewers, and commenting on their output. which - as I understand it - it one of the central contributions of Publons. Easy to throw the reviewer out with the bathwater, though, and replace it with a version of peer review that weakens what can be an extremely effective process.

The review process takes too long. I agree. But while this might be about the seemingly pre-internet workflows that most of Big Publishing uses, it's probably more about the fact that folks are really busy. Reviewers who review in a timely fashion tend to get more reviews to do and they reach capacity. The process of peer review, to maintain a level of quality acceptable to authors, editors and their communities, requires committed, meticulous reviewing and that takes time. There are ways to speed that process up, and it will be interesting to see whether Publons is effective in doing that and creating workflows that work with the constraints.

Peer review is not transparent. It isn't. There certainly is a need to open up the process of discussion, commentary and opinion about articles - whether published or not - to the community. Publons does that by allowing academics to "comment on, or ask questions about, the papers they already rely on to do their research". I like this idea - that Publons can start to disrupt the academic publishing process not by publishing, but by starting to decouple and enhance a peer review process which has until now been tied closely to the journal itself.

This is perhaps the most interesting aspect of Publons - and perhaps the most difficult to enact since, as Andrew says, it requires a change to the incentival structure that currently exists for academics. Publons does this by providing a way to build reputation through the enhancement of their personal/professional profile that is generated for a paper they have authored. The profile contains standard per-paper or per-researcher metrics plus the rep. created by engaging in discussions about articles, projects and contributions to the field. Other sites - ResearchGate or academia.edu, for example - also offer the opportunity to build reputation - and perhaps the outcome will be that there is a super-reputation aggregator (a Klout for academics) that scans the various sites for key indicators of reputation and influence. However it pans out, the experience design of sites like Publons will be critical: ask too much, in the wrong way, and it's a turn-off.

Quality Comment

Validity

The authors say in conclusion:

"In this paper we have raised a hypothesis for how to change the incentive structures faced by academics but have largely ignored the three other major players in the ecosystem we hope to disrupt: journals, libraries, and funding agencies. While it may be possible to marginalize journals and libraries, our ideas for the future of research will go nowhere if funding agencies and universities continue to rely entirely on conventional publication records to allocate jobs and funding."

There are a lot of issues here that I might discuss, but I suspect that the major one is around what 'disruption' means. One way that the academic publishing industry might trajectory is that, like many other industries, small, agile publishing startups will enter the industry fulfilling the publisher role but who have correctly understood the disruptive power of the internet and built their offer around it: they will offer enhanced utility at a lower price. Another trajectory is that the industry is disrupted by what has been started by the OA movement and forces Big Publishing to change, and change quickly. Their market muscle will mean that everyone else is playing catchup. The other is that, as Publons might demonstrate, disruption will come from an innovation in one part of the process - it could be peer review or it could be around how journals are managed and led - but change is in the air.

Summary

Reviewer

Peter Thomas is founder of the Manifesto Group, Executive Director of the Leasing Foundation, Visiting Senior Fellow at The University of Melbourne, and Visiting Professor at Brunel University, London. He is editor-in- chief of the international research journals Personal and Ubiquitous Computing (PUC), and Communications in Mobile Computing (ComC), both from Springer, and the new Internet of Things research journal, RIOT, an independent, open access altmetrics-based journal. Peter’s current interests are in the use of mobile, social and open source technologies in higher education to deliver engaging student experiences, and in the use of reputation media to reshape academic publishing.

Expertise

Probably not for me to judge. Maybe Publons contributors will be able to tell me.

Source

    © 2013 the Reviewer (CC BY-SA 3.0).

References

    Preston, A., Johnston, D. 2011. The Future of Academic Research. Publons Demo Journal.