Review badges
2 pre-pub reviews
3 post-pub reviews
Abstract

Here we present NeuroVault — a web based repository that allows researchers to store, share, visualize, and decode statistical maps of the human brain. NeuroVault is easy to use and employs modern web technologies to provide informative visualization of data without the need to install additional software. In addition, it leverages the power of the Neurosynth database to provide cognitive decoding of deposited maps. The data are exposed through a public REST API enabling other services and tools to take advantage of it. NeuroVault is a new resource for researchers interested in conducting meta- and coactivation analyses.

Authors

Krzysztof J. Gorgolewski;  Gael Varoquaux;  Gabriel Rivera;  Yannick Schwartz;  Satrajit S. Ghosh;  Camille Maumet;  Vanessa V. Sochat;  Thomas E. Nichols;  Russell A. Poldrack;  Jean-Baptiste Poline;  Tal Yarkoni;  Daniel S. Margulies

Publons users who've claimed - I am an author
Contributors on Publons
  • 2 authors
  • 5 reviewers
Followers on Publons
Metrics
Publons score (from 1 score)
10.0
Altmetric
Web of Science Core Collection Citations
  • A difficulty in reviewing an article of this type is knowing whether one is really reviewing the article or the "product" that it describes. So I'll say something about each briefly...

    The article made sense, even to this non-neuroscientist. However, I would have liked to see more information about the relationship between NeuroVault and Neurosynth (and the other databases): how they are positioned relative to each other, why one would choose one over other, how NeuroVault "leverages the power of the Neurosynth database," etc. The article appears to assume some familiarity with these; a reference is provided for Neurosynth, but not for Brainmap or Brainspell. It would also have been nice to have a figure showing an example of the API, for the software geeks among us. Finally, if I may be permitted a curmudgeonly point, the article would benefit from a little editing by a native speaker of English; the various grammatical errors do not make for easy reading.

    I logged on to NeuroVault and explored the "Add new collection" process. The look of the site is clean and modern, and performance is fast. I was slightly surprised at how many of the description fields that accompany a collection upload are in free text format. This will cause problems over time due to typos, different interpretations of terms, etc. In some cases, even if the number of possibilities for a field is such that it cannot be changed into a dropdown list, it might be advantageous to provide a partial list with an "Other..." option, that causes the free-text box to appear. In my experience of comparable systems in a non-academic setting, archiving of the data tends to be assigned to the intern, who has little incentive to be able to retrieve it three years hence, so it may be worth "idiot-proofing" the system even at the expense of rapidity of use.

    The concept of sharing massive amounts of data in this way is a great example of what can be done for relatively little expense (as long as someone is crazy enough to put all those hours into writing the software!). My slight concern would be keeping the management and maintenance structures going as people change jobs, change institutions, and even perhaps lose interest (!). The amount of effort that people are prepared to put into creating free/open-source software is almost unlimited; their enthusiasm for getting out of bed at 4am on a Sunday to repair the SQL indexes again because of a still-not-fixed bug, not so much. Hopefully, if NeuroVault becomes sufficiently successful to need that level of support, appropriate funding will be available to pay for professional management.

    Ongoing discussion (1 comment - click to toggle)
    • Krzysztof Jacek Gorgolewski | 5 years, 8 months ago

      Thank you for your review. We have incorporated your feedback and uploaded a new version of the manuscript: http://biorxiv.org/content/early/2015/03/12/010348

      "A difficulty in reviewing an article of this type is knowing whether one is really reviewing the article or the "product" that it describes. So I'll say something about each briefly... The article made sense, even to this non-neuroscientist. However, I would have liked to see more information about the relationship between NeuroVault and Neurosynth (and the other databases): how they are positioned relative to each other, why one would choose one over another, how NeuroVault "leverages the power of the Neurosynth database," etc. The article appears to assume some familiarity with these; a reference is provided for Neurosynth, but not for Brainmap or Brainspell. It would also have been nice to have a figure showing an example of the API, for the software geeks among us. Finally, if I may be permitted a curmudgeonly point, the article would benefit from a little editing by a native speaker of English; the various grammatical errors do not make for easy reading."

      We have expanded the discussion of the position of NeuroVault in the ecosystem of other databases and services. We have also added the missing references and brief descriptions of Neurosynth, Brainmap, and Brainspell.

      "I logged on to NeuroVault and explored the "Add new collection" process. The look of the site is clean and modern, and performance is fast. I was slightly surprised at how many of the description fields that accompany a collection upload are in free text format. This will cause problems over time due to typos, different interpretations of terms, etc. In some cases, even if the number of possibilities for a field is such that it cannot be changed into a dropdown list, it might be advantageous to provide a partial list with an "Other..." option, that causes the free-text box to appear. In my experience of comparable systems in a non-academic setting, archiving of the data tends to be assigned to the intern, who has little incentive to be able to retrieve it three years hence, so it may be worth "idiot-proofing" the system even at the expense of rapidity of use."

      Figuring out what are all of the possible options for many of these fields is a difficult task. Essentially it required defining an ontology for the terms used. Following your advice we made progress on limiting some of the free form fields. One of the fields we added is the modality field with a list of possible MR and other modalities that were used during the experiment. The possible values on that list were obtained through an online discussion with members of the INCF Neuroimaging Data Sharing Task Force. Another field we added was the cognitive paradigm used during the scan. Again we restricted the answers to a preselected list. This time the list was taken from existing ontology - Cognitive Atlas - consisting of hundreds of task descriptions.

      "The concept of sharing massive amounts of data in this way is a great example of what can be done for relatively little expense (as long as someone is crazy enough to put all those hours into writing the software!). My slight concern would be keeping the management and maintenance structures going as people change jobs, change institutions, and even perhaps lose interest (!). The amount of effort that people are prepared to put into creating free/open-source software is almost unlimited; their enthusiasm for getting out of bed at 4am on a Sunday to repair the SQL indexes again because of a still-not-fixed bug, not so much. Hopefully, if NeuroVault becomes sufficiently successful to need that level of support, appropriate funding will be available to pay for professional management."

      This is a very good point. Developing open source desktop applications is easier than web applications. Even though modern web technologies make maintenance relatively easy it still requires some time commitment. We are trying to minimize this cost, but at the end of the day to keep the platform going we will require some sort of medium to long term support. We hope that if the community will see the advantage of having such platform resources for keeping it will become available.

  • Gorgolewski and colleagues here officially introduce and principle-of-proof check a new kind of neuroimaging platform called NeuroVault. This is a web-provided service to upload and share different types of 3D nifti volumes. This is not limited to volumes of Z/T values from fMRI and PET analyses, but includes also VBM, diffusion MRI, as well as source-reconstructed EEG/MEG data. In comparison to conventional nifti-data-sharing websites, one might want to stress the explicit support for sharing ROI (region of interest) maps, maps from neurological lesion studies, label maps (e.g., from connectivity-based parcellation investigations), and posterior probability maps (from Bayesian statistics). This might help to leverage Open Data by operation-system-independent sharing via a web-interface used by researchers in neuroscience. More generally, such approach could improve the outreach to an interested public.

    However, NeuroVault attemps to go beyond simply making data accessible. This is enabled by its seamless integration into a vibrant Open Source ecosystem, including for instance Papaya (Research Imaging Institute, San Antonio, USA), Neurosynth (Tal Yarkoni), and pycortex (Gallant lab, Berkeley, USA). An identical dataset can thus be viewed at different threshold options, rendered onto different brain templates, and personalized by different cosmetics. The maps can also be submitted to automatic functional decoding by assignment to cognitive principles. It is an intriguing possibility that future neuroimaging paper publications might be complemented by links to such interactive access to the discussed results. Further, it might be an ideal platform for quantitative cross-study comparison and synthesis. Even more so, the bundled access to results from diverging imaging techniques will be important for future large-scale projects of cross-modality fusion. The accumulation of neuroscientific finding is helped by the possible extension to novel types of machine-readable meta-data. The only downside I see is the currently missing integration with neuroanatomy that could by realized by quantitative comparison to probabilistic cytoarchitectonic maps. More generally, NeuroVault being a social-coding project on Github might foretell a rapid evolution of this initiative.

    Massive, clean, well-annotated, and easily accessible data will be a critical prerequisite for statistical learning of human functional brain architecture in health and disease. In this regard, NeuroVault has the potential to push imaging neuroscience to the next level of publication practices, inter-laboratory collaboration, as well as the structured discovery and propagation of neurobiological knowledge.

    Danilo Bzdok

    Reviewed by
    Ongoing discussion (1 comment - click to toggle)
    • Krzysztof Jacek Gorgolewski | 5 years, 8 months ago

      We thank the reviewer for his kind words. We have incorporated the feedback and uploaded a new version of the manuscript: http://biorxiv.org/content/early/2015/03/12/010348.

      We have build a preliminary support for storing and accessing atlases (so far limited to deterministic atlases). We are also testing a feature that would allow users to compare different statistical maps. We will expand this feature to include atlases (both probabilistic and deterministic), but some groundwork with regards of statistical comparison needs to be performed first. We completely agree that giving users the ability to compare patterns found in their maps to probabilistic atlases is an exciting feature and we hope to implement it in the near future.

  • This paper presents a description of Neurovault a repository for archiving neuroimaging data that I think will facilitate our understanding of the brain, and cognition, by improving the detail available to perform meta analyses. This is a worthy endeavour and one I am fully in support of. However, I have a general point about the meta analysis technique which I believe applies to Neurovault. A general problem with meta analysis is that the data is restricted to what has already been done, meaning that these techniques cannot really uncover fundamentally new discoveries. This is because the data is self selecting (e.g. those studies that have already been performed). This problem is exacerbated in the case of Neurovault because people also have to choose to upload their data. Thus Neurovault could possibly lead to a biased sample of the available data. A solution would be to ask funding bodies to require the authors to submit their data to neurovault as a condition of the grant.

    Ongoing discussion (1 comment - click to toggle)
    • Krzysztof Jacek Gorgolewski | 5 years, 8 months ago

      We thank the reviewer for his comments. We have incorporated thr feedback and uploaded a new version of the manuscript: http://biorxiv.org/content/early/2015/03/12/010348

      It is of course inevitable that a meta-analysis is built from studies that are already existing, and we don’t intend to claim that NeuroVault can replace the development of novel concepts and tools for investigating brain function. We also agree that because of the voluntary nature of the data submission we are facing a risk of a biased sample (we have added paragraph discussing this issue in the manuscript). We have added a paragraph discussing this issue to the manuscript. However we do not agree that using NeuroVault data cannot lead to new discoveries. As we have shown in our proof of concept analysis increased statistical power due to the use of unthresholded maps can uncover neural representations that would have been missed when looking at results from a single study or an analysis of coordinate data from multiple studies. We also envisage that using statistical maps from many studies can lead to better understanding of specificity of different neuronal representations, and may further aid in the development of novel experiments.

All peer review content displayed here is covered by a Creative Commons CC BY 4.0 license.