Abstract

When reporting research findings, scientists document the steps they followed so that others can verify and build upon the research. When those steps have been described in sufficient detail that others can retrace the steps and obtain similar results, the research is said to be reproducible. Computers play a vital role in many research disciplines and present both opportunities and challenges for reproducibility. Computers can be programmed to execute analysis tasks, and those programs can be repeated and shared with others. The deterministic nature of most computer programs means that the same analysis tasks, applied to the same data, will often produce the same outputs. However, in practice, computational findings often cannot be reproduced because of complexities in how software is packaged, installed, and executed-and because of limitations associated with how scientists document analysis steps. Many tools and techniques are available to help overcome these challenges; here we describe seven such strategies. With a broad scientific audience in mind, we describe the strengths and limitations of each approach, as well as the circumstances under which each might be applied. No single strategy is sufficient for every scenario; thus we emphasize that it is often useful to combine approaches.

Authors - I am an author
Contributors on Publons
  • 2 reviewers
Followers on Publons
  • Full text of Stephen Eglen's review can be found here: (https://drive.google.com/open?id=0B0V9UazwxfgRdjI2Q3ZlYmR2a28)

    Level of interest
    Please indicate how interesting you found the manuscript:
    An article whose findings are important to those with closely related research interests

    Quality of written English
    Please indicate the quality of language in the manuscript:
    Acceptable

    Declaration of competing interests
    Please complete a declaration of competing interests, considering the following questions:
    Have you in the past five years received reimbursements, fees, funding, or salary from an organisation that may in any way gain or lose financially from the publication of this manuscript, either now or in the future?
    Do you hold any stocks or shares in an organisation that may in any way gain or lose financially from the publication of this manuscript, either now or in the future?
    Do you hold or are you currently applying for any patents relating to the content of the manuscript?
    Have you received reimbursements, fees, funding, or salary from an organization that holds or has applied for patents relating to the content of the manuscript?
    Do you have any other financial competing interests?
    Do you have any non-financial competing interests in relation to this paper?
    If you can answer no to all of the above, write 'I declare that I have no competing interests' below. If your reply is yes to any, please give details below.

    I declare that I have no competing interests.

    I agree to the open peer review policy of the journal. I understand that my name will be included on my report to the authors and, if the manuscript is accepted for publication, my named report including any attachments I upload will be posted on the website along with the authors' responses. I agree for my report to be made available under an Open Access Creative Commons CC-BY license (http://creativecommons.org/licenses/by/4.0/). I understand that any comments which I do not wish to be included in my named report can be included as confidential comments to the editors, which will not be published.
    I agree to the open peer review policy of the journal.

    Authors' response to reviews:

    ** In the point-by-point replies below, the reviewer’s comments are shown first. Our responses are prefixed by >> characters. **

    Reviewer #1 (Titus Brown)

    In this paper, Piccolo et al. do a nice (and I think comprehensive?) job of outlining strategies for computational reproducibility. The point is well made that science is increasingly dependent on computational reproducibility (and that in theory we should be able to do computational reproducibility easily and well) and hence we should explore effective approaches that are actually being used.

    I know of no other paper that covers this array of material, and this is a quite nice exposition that I would recommend to many. I can't evaluate how broadly it will appeal to a diverse audience but it seems very readable to me.

    >> We thank the reviewer for taking time to review our manuscript and for these positive comments!

    This paper is a nice complement to Ten Simple Rules..., http://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1003285, in that it is longer, more in depth, and provides more explicit recommendations. It is also more up to date.

    >> Thanks for this comment. Our goal, in part, was to extend beyond what the Ten Simple Rules… article covers. We cite that article multiple times within our manuscript.

    I reviewed this paper previously, and it has been updated in light of my (and other) reviewers' comments. I was positive about it then, and the revisions are good ;). One note is that the 'binder' service (http://mybinder.org) should be mentioned as a harbinger of the future. I have a blog post summary of it here: ivory.idyll.org/blog/2016-mybinder.html

    Signed,
    C. Titus Brown
    ctbrown@ucdavis.edu

    >> Thanks for this suggestion. We have updated the second sentence of the Discussion section. We mention Binder (and Everware) and describe them as potential “harbingers of the future.” (I hope it’s OK that we borrowed this phrase from Titus. )



    Reviewer #2 (Stephen Eglen)

    Summary

    This review provides an overview of the main tools and techniques used to ensure computational reproducibility of results. This review is a good fit to the readership of the Gigascience journal.

    >> We thank the reviewer for taking time to review this manuscript and for these positive comments!

    Overall I found the manuscript to be clearly written and would recommend it for publication, although I would like the authors to consider the following issues in a potential revision. In particular, I am most critical about the current set of figures. As a general comment, where possible, the material in the figures should be supported by online files (e.g. knitr, Docker) so that the reader can immediately get workable examples to examine, run and amend.

    >> We have addressed these comments below. We now provide executable files to complement Figures 1-4. We have included these in the supplementary material. We have also improved these materials based on the reviewer’s comments.

    Detailed comments

    p4, l2 (page 4, line 2): an extra citation might be (Gronenschild et al. 2012) which showed that even the choice of operating system or neuroimaging software version affects results.

    >> We have added this citation at the suggested location.

    p4, l18-20: as far as I am aware, journals do not encourage direct use of repositories like github or bitbucket because they offer no long-term storage. It is regarded much more appropriate to use permanent URLS (e.g. DOIs) to point to archived versions of software, such as zenodo. We have written elsewhere on this topic (Eglen et al. 2016).

    >> We have updated this text so that it no longer mentions GitHub and Bitbucket as examples of storage locations for archived versions of code. Now we mention Zenodo and figshare as services that provide permanent DOIs.

    p6, l10: in addition to the other topics of reproducibility and education, it would be worth
    mentioning education/training to encourage users to adopt reproducible practices (Wilson 2016).

    >> We have added a reference to Wilson 2016 and have mentioned “education about reproducibility” as a topic that is covered elsewhere.

    p6, l15: minor point, but perhaps worth numbering the seven sections that describe the seven
    approaches reviewed, starting here.

    >> We have numbered these sections as an aid to the reader.

    p8, l17: “Make can be configured”; if you are referring to the “-j N” switch, then it is simpler to say that “Make can automatically identify. . . ” as there is no extra configuration needed.

    >> We have made this change to the text. Thanks for pointing it out.

    p11 l9: reference 53 at the end of the sentence is not needed, as you refer to it at the start of the
    sentence.

    >> We have removed this citation at the end of the sentence.

    p13, l3-9: you should probably mention http://mybinder.org in this section. It provides a transparent method for interacting with Jupyter documents over the web (Rosenberg and Horn 2016).

    >> Thanks for this suggestion. We have updated the second sentence of the Discussion section. We mention Binder (and Everware, a similar service) and cite the paper by Rosenberg and Horn.

    p14, l2: I disagree slightly here with the view re: long-running jobs. knitr at least can cache
    intermediate computations transparently which helps enormously.

    >> We have modified this paragraph to indicate that long-running notebooks can be executed at the command line.

    p14, l8: Do you have examples of Dexy in use within this field? Asking this also in a more general way, it might be worth making a table listing examples explicitly of each of the seven approaches, so that they are easy to find.

    >> We haven’t found any substantive examples where Dexy is used in a biology-focused study. We have included it more as an indicator that alternative literate programming environments may be useful in cases where Juptyer and knitr are not suitable.

    >> Although it may be useful to add a table that lists examples where each of the seven methods is used, we have cited various such studies throughout the text. We hope these references will provide useful examples, but we wish to avoid putting too much emphasis on any individual study.

    p18, l18: I do not see why the VMs are “black boxes”. Surely to create the VM all the relevant
    code must be provided, so that you can at least examine what is done, or extend the analysis (as mentioned on l20). Can you clarify what you mean here.

    >> We have rewritten this paragraph to clarify our points. Our goal was to emphasize the importance of providing easy access to scripts and code that are used within a virtual machine for an analysis. Although it would be possible for other scientists to examine the full contents of a virtual machine, it is less convenient (and perhaps less transparent). So we encourage the idea of storing these scripts and code in a public repository, separate from the virtual machine. C. Titus Brown (reviewer #1) has also emphasized this point on his blog, which we cite: http://ivory.idyll.org/blog/vms-considered-harmful.html.

    p22, l1: as well as capturing software in Docker, we have used it recently to capture the entire
    environment to write our research papers in knitr (https://hub.docker.com/r/sje30/eglen2015/
    and https://hub.docker.com/r/sje30/waverepo/ ).

    >> Thanks for pointing to these. We now cite these papers as examples of enabling others to execute analyses described in research papers.

    p22, l14-19: I found the section about other operating systems (Windows/mac) rather clumsy.
    From the user’s perspective, the modern docker toolbox seems to work smoothly enough (at least on macs) that the details at the end of p22 seem irrelevant. It might instead be worth mentioning that docker builds can be automatically triggered, e.g. upon new commits to github.

    >> Thanks for this suggestion. Indeed, the Docker toolbox has been updated recently so that it is much more convenient to install and execute Docker on non-Linux machines. We have simplified the text accordingly. We have also clarified that the overhead of using a virtual machine to execute Docker containers on Windows or Mac operating systems offsets some of the performance benefits of containers.

    p 23, l6: “Scientific advancement requires trust.” I think trust is the wrong word here.
    e.g. The Royal Society’s motto ‘Nullius in verba’ is taken to mean ‘take nobody’s word for it’
    (https://royalsociety.org/about-us/history/). Rather, what these tools do is promote transparency
    to reduce the barriers for others to repeat prior work. With this in mind, I’d suggest the authors
    re-read this first paragraph of the discussion to see whether they think “trust” is what they are
    promoting here.

    >> Thanks for this insight. We have revised the first paragraph of the Discussion section. It no longer uses the word “trust” but instead focuses on the ideas of explicitly documenting research steps and being transparent about the way those steps are shared.

    Figures

    Figure 1: is this really needed in such a review? I don’t think it adds anything.

    >> After considering this feedback, we agree that this figure is not needed, and we have removed it from the manuscript.

    Figure 2: I think should be provided as a text file so that people can run it for themselves.

    >> We now provide a text version of this script as a supplementary file.

    Figure 3: Text file definitely needed so people can run it for themselves. But also I’d consider
    making the targets a bit more specific. All of the targets in this example Makefile are PHONY and perhaps it could be rewritten in the more canonical Makefile style? The way it is currently written, it is hard to see how this differs from a shell script. (Each time make all is run, won’t the files be downloaded again?)

    >> Thanks for pointing this out. We have modified the example Makefile so that it follows a more conventional style. Now it will only download the files (and execute the other steps) one time (if the steps executed successfully). In addition, we now provide a text version as a supplementary file.

    Figure 4: can you show something that is a bit more bio-relevant, e.g. some genomic analysis?

    >> We have updated this figure so that it shows a plot of simulated gene-expression data.

    Figure 5: as figure 4. Can you design Figure 4 and 5 carefully to highlight the differences between knitr and jupyter?

    >> We have updated this figure so it also shows a plot simulated-gene expression data. We are unsure which differences between knitr and Jupyter that the reviewer would like us to point out. However, we believe the updated figures show a more distinct difference between the two.

    Figure 6-8: I would suggest you drop these. They show the notion of stacks and containers, but
    most bioscientists won’t get much value from them.

    >> We respect the reviewer’s point of view; however, given feedback from scientists who have read our preprint, we feel that some scientists will benefit from these figures. Thus we prefer to retain them.

    References

    Eglen, Stephen, Ben Marwick, Yaroslav Halchenko, Michael Hanke, Shoaib Sufi, Padraig Gleeson, R Angus Silver, et al. 2016. “Towards Standard Practices for Sharing Computer Code and Programs in Neuroscience.” BioRxiv. doi:10.1101/045104.

    Gronenschild, Ed H B M, Petra Habets, Heidi I L Jacobs, Ron Mengelers, Nico Rozendaal, Jim van Os, and Machteld Marcelis. 2012. “The Effects of FreeSurfer Version, Workstation Type, and Macintosh Operating System Version on Anatomical Volume and Cortical Thickness Measurements.” PLoS One 7 (6): e38234. doi:10.1371/journal.pone.0038234.

    Rosenberg, David M, and Charles C Horn. 2016. “Neurophysiological Analytics for All! Free OpenSource Software Tools for Documenting, Analyzing, Visualizing, and Sharing Using Electronic Notebooks.” J. Neurophysiol., 20~apr, jn.00137.2016. doi:10.1152/jn.00137.2016.

    Wilson, Greg. 2016. “Software Carpentry: Lessons Learned.” F1000Res. 3 (28~jan).
    doi:10.12688/f1000research.3-62.v2.


    Published in
    Reviewed by
    Ongoing discussion
  • A review of "Tools and techniques for computational reproducibility"

    In this paper, Piccolo et al. do a nice (and I think comprehensive?)
    job of outlining strategies for computational reproducibility. The
    point is well made that science is increasingly dependent on
    computational reproducibility (and that in theory we should be able to
    do computational reproducibility easily and well) and hence we should
    explore effective approaches that are actually being used.

    I know of no other paper that covers this array of material, and this
    is a quite nice exposition that I would recommend to many. I can't
    evaluate how broadly it will appeal to a diverse audience but it seems
    very readable to me.

    This paper is a nice complement to Ten Simple Rules...,
    http://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1003285,
    in that it is longer, more in depth, and provides more explicit
    recommendations. It is also more up to date.

    I reviewed this paper previously, and it has been updated in light of
    my (and other) reviewers' comments. I was positive about it then,
    and the revisions are good ;).

    One note is that the 'binder' service (http://mybinder.org) should be
    mentioned as a harbinger of the future. I have a blog post summary of
    it here --

    ivory.idyll.org/blog/2016-mybinder.html

    Signed,

    C. Titus Brown
    ctbrown@ucdavis.edu

    Level of interest
    Please indicate how interesting you found the manuscript:
    An article of importance in its field

    Quality of written English
    Please indicate the quality of language in the manuscript:
    Acceptable

    Declaration of competing interests
    Please complete a declaration of competing interests, considering the following questions:
    Have you in the past five years received reimbursements, fees, funding, or salary from an organisation that may in any way gain or lose financially from the publication of this manuscript, either now or in the future?
    Do you hold any stocks or shares in an organisation that may in any way gain or lose financially from the publication of this manuscript, either now or in the future?
    Do you hold or are you currently applying for any patents relating to the content of the manuscript?
    Have you received reimbursements, fees, funding, or salary from an organization that holds or has applied for patents relating to the content of the manuscript?
    Do you have any other financial competing interests?
    Do you have any non-financial competing interests in relation to this paper?
    If you can answer no to all of the above, write 'I declare that I have no competing interests' below. If your reply is yes to any, please give details below.

    I declare that I have no competing interests.

    I agree to the open peer review policy of the journal. I understand that my name will be included on my report to the authors and, if the manuscript is accepted for publication, my named report including any attachments I upload will be posted on the website along with the authors' responses. I agree for my report to be made available under an Open Access Creative Commons CC-BY license (http://creativecommons.org/licenses/by/4.0/). I understand that any comments which I do not wish to be included in my named report can be included as confidential comments to the editors, which will not be published.
    I agree to the open peer review policy of the journal.

    Authors' response to reviews:

    ** In the point-by-point replies below, the reviewer’s comments are shown first. Our responses are prefixed by >> characters. **

    Reviewer #1 (Titus Brown)

    In this paper, Piccolo et al. do a nice (and I think comprehensive?) job of outlining strategies for computational reproducibility. The point is well made that science is increasingly dependent on computational reproducibility (and that in theory we should be able to do computational reproducibility easily and well) and hence we should explore effective approaches that are actually being used.

    I know of no other paper that covers this array of material, and this is a quite nice exposition that I would recommend to many. I can't evaluate how broadly it will appeal to a diverse audience but it seems very readable to me.

    >> We thank the reviewer for taking time to review our manuscript and for these positive comments!

    This paper is a nice complement to Ten Simple Rules..., http://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1003285, in that it is longer, more in depth, and provides more explicit recommendations. It is also more up to date.

    >> Thanks for this comment. Our goal, in part, was to extend beyond what the Ten Simple Rules… article covers. We cite that article multiple times within our manuscript.

    I reviewed this paper previously, and it has been updated in light of my (and other) reviewers' comments. I was positive about it then, and the revisions are good ;). One note is that the 'binder' service (http://mybinder.org) should be mentioned as a harbinger of the future. I have a blog post summary of it here: ivory.idyll.org/blog/2016-mybinder.html

    Signed,
    C. Titus Brown
    ctbrown@ucdavis.edu

    >> Thanks for this suggestion. We have updated the second sentence of the Discussion section. We mention Binder (and Everware) and describe them as potential “harbingers of the future.” (I hope it’s OK that we borrowed this phrase from Titus. )



    Reviewer #2 (Stephen Eglen)

    Summary

    This review provides an overview of the main tools and techniques used to ensure computational reproducibility of results. This review is a good fit to the readership of the Gigascience journal.

    >> We thank the reviewer for taking time to review this manuscript and for these positive comments!

    Overall I found the manuscript to be clearly written and would recommend it for publication, although I would like the authors to consider the following issues in a potential revision. In particular, I am most critical about the current set of figures. As a general comment, where possible, the material in the figures should be supported by online files (e.g. knitr, Docker) so that the reader can immediately get workable examples to examine, run and amend.

    >> We have addressed these comments below. We now provide executable files to complement Figures 1-4. We have included these in the supplementary material. We have also improved these materials based on the reviewer’s comments.

    Detailed comments

    p4, l2 (page 4, line 2): an extra citation might be (Gronenschild et al. 2012) which showed that even the choice of operating system or neuroimaging software version affects results.

    >> We have added this citation at the suggested location.

    p4, l18-20: as far as I am aware, journals do not encourage direct use of repositories like github or bitbucket because they offer no long-term storage. It is regarded much more appropriate to use permanent URLS (e.g. DOIs) to point to archived versions of software, such as zenodo. We have written elsewhere on this topic (Eglen et al. 2016).

    >> We have updated this text so that it no longer mentions GitHub and Bitbucket as examples of storage locations for archived versions of code. Now we mention Zenodo and figshare as services that provide permanent DOIs.

    p6, l10: in addition to the other topics of reproducibility and education, it would be worth
    mentioning education/training to encourage users to adopt reproducible practices (Wilson 2016).

    >> We have added a reference to Wilson 2016 and have mentioned “education about reproducibility” as a topic that is covered elsewhere.

    p6, l15: minor point, but perhaps worth numbering the seven sections that describe the seven
    approaches reviewed, starting here.

    >> We have numbered these sections as an aid to the reader.

    p8, l17: “Make can be configured”; if you are referring to the “-j N” switch, then it is simpler to say that “Make can automatically identify. . . ” as there is no extra configuration needed.

    >> We have made this change to the text. Thanks for pointing it out.

    p11 l9: reference 53 at the end of the sentence is not needed, as you refer to it at the start of the
    sentence.

    >> We have removed this citation at the end of the sentence.

    p13, l3-9: you should probably mention http://mybinder.org in this section. It provides a transparent method for interacting with Jupyter documents over the web (Rosenberg and Horn 2016).

    >> Thanks for this suggestion. We have updated the second sentence of the Discussion section. We mention Binder (and Everware, a similar service) and cite the paper by Rosenberg and Horn.

    p14, l2: I disagree slightly here with the view re: long-running jobs. knitr at least can cache
    intermediate computations transparently which helps enormously.

    >> We have modified this paragraph to indicate that long-running notebooks can be executed at the command line.

    p14, l8: Do you have examples of Dexy in use within this field? Asking this also in a more general way, it might be worth making a table listing examples explicitly of each of the seven approaches, so that they are easy to find.

    >> We haven’t found any substantive examples where Dexy is used in a biology-focused study. We have included it more as an indicator that alternative literate programming environments may be useful in cases where Juptyer and knitr are not suitable.

    >> Although it may be useful to add a table that lists examples where each of the seven methods is used, we have cited various such studies throughout the text. We hope these references will provide useful examples, but we wish to avoid putting too much emphasis on any individual study.

    p18, l18: I do not see why the VMs are “black boxes”. Surely to create the VM all the relevant
    code must be provided, so that you can at least examine what is done, or extend the analysis (as mentioned on l20). Can you clarify what you mean here.

    >> We have rewritten this paragraph to clarify our points. Our goal was to emphasize the importance of providing easy access to scripts and code that are used within a virtual machine for an analysis. Although it would be possible for other scientists to examine the full contents of a virtual machine, it is less convenient (and perhaps less transparent). So we encourage the idea of storing these scripts and code in a public repository, separate from the virtual machine. C. Titus Brown (reviewer #1) has also emphasized this point on his blog, which we cite: http://ivory.idyll.org/blog/vms-considered-harmful.html.

    p22, l1: as well as capturing software in Docker, we have used it recently to capture the entire
    environment to write our research papers in knitr (https://hub.docker.com/r/sje30/eglen2015/
    and https://hub.docker.com/r/sje30/waverepo/ ).

    >> Thanks for pointing to these. We now cite these papers as examples of enabling others to execute analyses described in research papers.

    p22, l14-19: I found the section about other operating systems (Windows/mac) rather clumsy.
    From the user’s perspective, the modern docker toolbox seems to work smoothly enough (at least on macs) that the details at the end of p22 seem irrelevant. It might instead be worth mentioning that docker builds can be automatically triggered, e.g. upon new commits to github.

    >> Thanks for this suggestion. Indeed, the Docker toolbox has been updated recently so that it is much more convenient to install and execute Docker on non-Linux machines. We have simplified the text accordingly. We have also clarified that the overhead of using a virtual machine to execute Docker containers on Windows or Mac operating systems offsets some of the performance benefits of containers.

    p 23, l6: “Scientific advancement requires trust.” I think trust is the wrong word here.
    e.g. The Royal Society’s motto ‘Nullius in verba’ is taken to mean ‘take nobody’s word for it’
    (https://royalsociety.org/about-us/history/). Rather, what these tools do is promote transparency
    to reduce the barriers for others to repeat prior work. With this in mind, I’d suggest the authors
    re-read this first paragraph of the discussion to see whether they think “trust” is what they are
    promoting here.

    >> Thanks for this insight. We have revised the first paragraph of the Discussion section. It no longer uses the word “trust” but instead focuses on the ideas of explicitly documenting research steps and being transparent about the way those steps are shared.

    Figures

    Figure 1: is this really needed in such a review? I don’t think it adds anything.

    >> After considering this feedback, we agree that this figure is not needed, and we have removed it from the manuscript.

    Figure 2: I think should be provided as a text file so that people can run it for themselves.

    >> We now provide a text version of this script as a supplementary file.

    Figure 3: Text file definitely needed so people can run it for themselves. But also I’d consider
    making the targets a bit more specific. All of the targets in this example Makefile are PHONY and perhaps it could be rewritten in the more canonical Makefile style? The way it is currently written, it is hard to see how this differs from a shell script. (Each time make all is run, won’t the files be downloaded again?)

    >> Thanks for pointing this out. We have modified the example Makefile so that it follows a more conventional style. Now it will only download the files (and execute the other steps) one time (if the steps executed successfully). In addition, we now provide a text version as a supplementary file.

    Figure 4: can you show something that is a bit more bio-relevant, e.g. some genomic analysis?

    >> We have updated this figure so that it shows a plot of simulated gene-expression data.

    Figure 5: as figure 4. Can you design Figure 4 and 5 carefully to highlight the differences between knitr and jupyter?

    >> We have updated this figure so it also shows a plot simulated-gene expression data. We are unsure which differences between knitr and Jupyter that the reviewer would like us to point out. However, we believe the updated figures show a more distinct difference between the two.

    Figure 6-8: I would suggest you drop these. They show the notion of stacks and containers, but
    most bioscientists won’t get much value from them.

    >> We respect the reviewer’s point of view; however, given feedback from scientists who have read our preprint, we feel that some scientists will benefit from these figures. Thus we prefer to retain them.

    References

    Eglen, Stephen, Ben Marwick, Yaroslav Halchenko, Michael Hanke, Shoaib Sufi, Padraig Gleeson, R Angus Silver, et al. 2016. “Towards Standard Practices for Sharing Computer Code and Programs in Neuroscience.” BioRxiv. doi:10.1101/045104.

    Gronenschild, Ed H B M, Petra Habets, Heidi I L Jacobs, Ron Mengelers, Nico Rozendaal, Jim van Os, and Machteld Marcelis. 2012. “The Effects of FreeSurfer Version, Workstation Type, and Macintosh Operating System Version on Anatomical Volume and Cortical Thickness Measurements.” PLoS One 7 (6): e38234. doi:10.1371/journal.pone.0038234.

    Rosenberg, David M, and Charles C Horn. 2016. “Neurophysiological Analytics for All! Free OpenSource Software Tools for Documenting, Analyzing, Visualizing, and Sharing Using Electronic Notebooks.” J. Neurophysiol., 20~apr, jn.00137.2016. doi:10.1152/jn.00137.2016.

    Wilson, Greg. 2016. “Software Carpentry: Lessons Learned.” F1000Res. 3 (28~jan).
    doi:10.12688/f1000research.3-62.v2.


    Published in
    Reviewed by
    Ongoing discussion