Content of review 1, reviewed on April 26, 2023

The topic of this review is interdisciplinary (evidence synthesis and open science), which is considered an ongoing interest for many in the evidence synthesis community. I appreciate both authors' interest in the topic and encourage them to continue studying and practising open research in the evidence synthesis context; however, I have to be critical of the manuscript as an independent peer, and I hope my criticism could constructively improve the quality of the manuscript without being taken as personal.

Expertise in the Field of the Review A review of such importance could have been co-authored by experts (researchers, practitioners, and policymakers) on both or at least one of these topics and submitted to the most relevant journal for visibility to the relevant community members. Lacking such expertise becomes clearer as one reads that the SR workflow is "phenomenally simple" in the abstract. I wondered if the authors considered or will consider inviting experts (from the list of cited references) to co-author the next version with them, as it happens that because of time constraints, some authors rush towards submission and make the main effort in the revised version.

Definition of Evidence Synthesis The authors need to present a definition of evidence synthesis and/or systematic review that fits their review's purpose. The use of components of systematic reviewing as part of evidence synthesis is very popular and a source of confusion. For example, if a team reports the keywords used for searching for their review, should we call it a systematic review? What if they only report a search strategy without following the other steps or don't conduct a risk of bias assessment?

Purpose The writing style could be more coherent for a review article. This manuscript could be perceived as the first version of a self-reflective work when someone new to the field studies papers and tries to write an introductory chapter for a dissertation or coursework. In the current format, the review serves a purpose other than scientific communications with the community through publishing a review in an academic journal. I hope the authors find an acceptable gap in the literature to fill by publishing this review.

Contribution to the Field This review does not add to the already known, documented, and updated knowledge in this field and does not summarise or enrich the existing knowledge. While the authors have selectively and briefly used some of the widely known literature (Tsafnat et al. 2014; Ross 2016; Wolfenden et al. 2016; ICASR and Vienna Principles), they have missed some of the most important literature in the field (Haddaway 2018; Metzendorf et al. 2022; Shokraneh et al. 2018; Shokraneh 2019; Uttley et al. 2023). It should also be noted how some previous experts have published their works as commentary, letters, and editorials, not necessarily as reviews.

Internal and External Facilitators of Open Science The authors have focused on highlighting the role of open science from an internal perspective, only focusing on what systematic reviewers can do without discussing the external barriers such as blockage of open science by publishers through copyright, open access article processing charge that stops unfunded or less-funded research from becoming visible, and publishing/trapping information/data in PDF format which is one of the worst enemies of openness. Other external barriers are the primary researchers not practising open science principles that, make the data for systematic reviews and automation unavailable.

Technical Comments SR workflow might look "phenomenally simple" from a non-expert point of view, but it's common knowledge in the expert community that no systematic review process could be called simple. MEDLINE and Embase are not search engines; however, they have search interfaces that may be called search engines for simplicity. These sources are technically known as bibliographic medical databases. PROSPERO is only one of the available platforms for registering systematic reviews, and it does not accept all types of reviews (or systematic reviews). PRISMA 2009 mandated reporting search strategy for only one database, contributing to waste and a barrier to openness and reproducibility. PRISMA 2020 and PRISMA-Search mandate reporting all search strategies for all sources. Challenges of systematic searching and possible solutions have already been reported, and the authors have cited them (MacFarlane et al. 2022). Selectively repeating them could lead to highlighting only some of the issues. searchRxiv is the optional open registry for reporting search strategies, not PROSPERO. It is possible to register a review in PROSPERO without a search strategy. The authors have cited the old strategy for Cochrane. In "Our next strategy: proposed new strategic framework to 2025", Cochrane will try to "improve access to Cochrane data for research users". While it is not Cochrane's data to hold, Cochrane shares data upon request and does not make it publicly and openly available in line with FAIR or CC. A problem with systematic reviews is that they are time-consuming and resource intensive; however, beginners see reviews as an easy way to have a publication in their resume. As a result, many start the review, but they can't finish it. Many conduct reviews to publish rather than answer a real-world question or problem. Some look at systematic review workflow as phenomenally simple so they can start and finish a review without funding (time, expertise, or resources). If they finish, the result usually suffers from low quality or answers a question with no or low priority, so it does not serve the science, policy, or practice. Some references have been cited in a rush and must be completed and corrected.

Automation as Solution The discussion on automation is a brief and uninformative summary of what has been presented systematically and coherently in previous works. The most important source of automation tools (Systematic Review Toolbox) has yet to be mentioned or cited. The authors have only cited access to the full text as the main problem for the identification stage, missing the aforementioned external barriers. Most importantly, the main ML help we have received so far is using ML for screening the search results. While progressing on other SR steps, ML/AI has a long way to go. Having a standard list of data points (metadata) for systematic reviews is almost impossible. Each systematic review is unique and requires setting up data extraction and entry forms with a different number of data points and metadata. Automation has little or no success in data extraction because of a lack of standards in reporting primary studies, using PDF rather than HTML as the primary format, and a lack of standards in analysing or reporting the analysed or measured data. Some publishers also hold copyright/ownership of data, now allowing unrestricted use. Many researchers find data sharing an extra task to their already overloaded schedule, especially if they have not received any training. If not enforced, they avoid sharing. It should be considered that any solutions today can only help to resolve future problems. Retrospective trapped science will remain in commercial publishers' prisons. The risk of bias assessment as part of SR depends on the definition of SR and field and the availability of reliable and valid tools for the included study designs. Automation can only help some reviews as human judgement is usually required. Paradoxically, instead of presenting the tools and sources compliant with open science principles, the authors listed the tools that do not follow such principles in the text and figure.

Final Words I appreciate the authors' interest in the field to study and summarise some important literature for this introductory piece. I see no addition to the existing knowledge and even supplying incomplete and misleading information that would confuse the readers. Since the authors advocate avoiding duplication, reducing waste, and open science, I'd request that the authors write the second version of the review in line with these three recommendations. This article requires at least 6 to 12 months of part-time work and the involvement of at least two experts in the authorship team to become an impactful work with new insights. I will gladly allocate another six hours to reading and commenting on the next version.

Source

    © 2023 the Reviewer (CC BY 4.0).

References

    Martinou, E., Angelidi, A. 2022. The role of open research in improving the standards of evidence synthesis: current challenges and potential solutions in systematic reviews. F1000Research, 11: 1435.