Content of review 1, reviewed on December 02, 2022

"The logical structure of experiments lays the foundation of a theory of reproducibility" is an exciting attempt at better grounding the notion of replication, as well as understanding the relation between openness, replication and reproducibility, via a formalisation of the logical structure of an idealised experiment. The authors tackle a crucial topic in the ongoing discussions on how to reform scientific institutions to do better science and they do a commendable job in presenting a narrative version of a necessarily technical formalisation.
I found the simulations persuasive and the interpretation sober, while incisive. So I recommend publication and congratulate the authors on their worthy contribution to the debate.

As customary, I have some comments to improve the manuscript, but mostly for clarity:

  • there is some uncertainty in the early paragraphs about replication and reproduction (e.g. in page 1 they're always both present, but on page 2 the objective is to "understand and develop a precise language to talk about results reproducibility" (where has replication ended? It takes a whole page to get back to it).

  • My previous understanding of the notion of reproducibility (vs replication) was: the ability to re-run the code on the same data and getting the same results. This possible misunderstanding could be misspelled from the start.

  • The discussion of openness to remediate for lack of reproducibility was a bit opaque to me. After reading the full paper, I gather it's on making information available to ensure the possibility of exact replications. However, my initial understanding was slightly misaligned. On the one hand, it was about open the data and the scripts to analyse them, which should ensure reproducibility strictu sensu (the ability to run the script and get the same plots and findings), barring of course some issues in the code. On the other hand, the authors mention malpractice, and I gathered that the "openness" solution would then be in pre-registration or open lab notebook of some sort (limiting the degrees of freedom, or highlighting and motivating deviations from the plan).
    I think the notion and purpose of openness could be made more clear from the start.

  • it could be useful to anticipate at the first formula that there is a reference table where all symbols are defined

  • in the replication literature power is a big topic, with replications being usually more powered than original experiments. The authors indirectly tackle this in their simulation, varying the sample size, but it might be more explicitly mentioned.

  • On page 25 the authors open to a discussion of robustness or heterogeneity of the phenomenon. I missed a mention of multi-site replications (which, depending on K, might not be exact, e.g. involving different languages).

Source

    © 2022 the Reviewer.

Content of review 2, reviewed on January 06, 2023

The authors have satisfactorily answered my comments and clarified potential misunderstandings. I look forward to see this paper published, and the discussion that it will engender.
Signed review: Riccardo Fusaroli

Source

    © 2023 the Reviewer.

References

    O., B. E., Berna, D., Bert, B. 2023. The logical structure of experiments lays the foundation for a theory of reproducibility. Royal Society Open Science.