Introduction
It is broadly acknoweldged that the brain is a complex system. How then can we make progress towards understanding the brain and towards curing neurological and mental health disorders? There are many different approaches to study the brain, but I will here focus on data-driven research, the approach that starts with large, high-dimensional datasets – often an integration of datasets collected across multiple times and places, not neccesarily anticipating their integration – and seeks to tame their complexity by means of statistical methodology, computer software, and sophisticated ways of displaying and intepreting data. Though certainly not the only approach to study the brain, this approach has already been highly fruitful. Suffice it to refer to the oeuvre’s of 2024 OHBM Glass Brain Awardee Vince Calhoun and 2024 OHBM Early Career Investigator Awardee Bratislav Misic, as well as the extensive body of research that has been spurred through the work of 2024 Open Science Awardee Adriana Di Martino, as three shining examples that show the breadth and depth of insight and utility that this approach plumbs.
How can we advance data-driven research in human brain mapping? Though this is is not a new approach (for a historical perspective, see Donoho (2017)1), in recent years data-driven research has been increasingly linked with machine learning methods, with some recent calls to advance data-driven research focused in particular on the use of Artificial Intelligence (AI) models.2,3 Indeed, such models have had remarkable success in some fields of science, including the achievements in predicting 3D protein structure that were awarded the 2024 Nobel Prize in Chemistry.4,5 It is, therefore, not far-fetched to hope that these approaches may yield important advances in brain mapping as well. But this is certainly not the full scope of data-driven research going forward.
Frictionless reproducibility
In a thought-provoking essay, published in January 2024 in the Harvard Data Science Review that was titled “Data Science at the Singularity”6 (DSS, henceforth), Stanford statistician David Donoho introduced the concept of Frictionless Reproducibility (FR) as a key accelerator of “computation-based research”. Donoho – one of the most influential data scientists in the world today – observed the remarkable progress in empirical machine learning (EML) in the last decades – from AlexNet to DeepSeek. In observing some of the unique characteristic of EML as a field, Donoho points out that this progress has been undergirded by three characteristics that could be replicated in other fields as well. These are, slightly paraphrasing Donoho’s definitions: [FR1] Large and widely shared data resources; [FR2] lowered barriers to re-execution of research software; and [FR3] platforms and social structures around specific competitive challenges that have allowed measurable and well-incentivized incremental progress. He argues that it is these factors – sociotechnical in nature – rather than some threshold of a so-called “Singularity” that have allowed EML to advance so rapidly. Donoho explains that in a field where all three of these components are adopted, researchers can expect to be able to directly and rapidly examine the results of other researchers, and to build upon them with minimal effort. Certainly, for many of us in the human brain mapping research community the vision of rapid progress towards insights, and the ability to reproduce and build upon previous results is very compelling. In what follows, I will, therefore, evaluate the progress towards FR in human brain mapping, as of 2024/2025, critically assessing what more can be done, and where this vision may fall short for our field. By the nature of this missive – aimed to be timely and immediate – it is not meant to be a comprehensive literature review of data-driven research in neuroimaging, and I will acknowledge that it probably misses relevant and important issues that I am either unaware of, or which are not at the focus of my attention as I write about them now. I look forward to be enlightened by the conversations that will hopefully emerge from this document’s necessary shortcomings.
FR1: Open data
There has been incredible progress to advance FR1 in human neuroscience. A confluence of forces has contributed to a cambrian explosion in the availability and interoperability of open data resources in the last couple of decades: the establishment of platforms and standards for data sharing, of large consortium datasets, the formulation of institutional mandates, and the growth of a culture of data sharing and data reuse. Even the very first pioneering efforts to advance data sharing in human brain mapping, more than 20 years ago, were already heavily motivated by reference to data-driven research approaches,7 and the growth in availability and utility of these datasets will be apparent to anyone who attends the annual meeting. But – only somewhat acknowledged by Donoho in DSS – albeit very well-recognized by our research community is the challenge inherent in the fact that raw shared neuroimaging data is not directly actionable. That is because many processing steps intervene between the raw measurement and the inferences about brain structure and function that drive scientific understanding. Many of these steps are relatively standard and well-understood: for example, adjusting the data to correct for the effects of various artifacts. But some other are much more complex: the fitting of computational models to the data, quality assurance and annotation of experimental and clinical data, and so on. The gap is highlighted by David Baker, who’s data-driven research on protein structure was recognized with the 2024 Nobel Prize, in a recent MIT Tech Review article, where he said that ‘If there were many databases as good as the PDB [the protein structure database on which much of the protein design work that Baker received his prize for is based], I would say, yes, this [prize] probably is just the first of many, but it is kind of a unique database in biology,’‘’8 [1]. Developing shared neuroimaging datasets as a basis for FR, therefore, poses some interesting technical and conceptual challenges for the future. There are probably multiple paths towards this vision. For example, many datasets and consortia now share image-derived phenotypes (IDPs) based on standard processing pipelines. This certainly lowers the barriers to FR with these datasets, and there are myriad examples of what can be done with this approach. But what are the tradeoffs? Some concerns about the applicability of FR1 to neuroimaging are represented in one of the several commentaries that were published in HDSR alongside Donoho’s essay. This commentary, written by neuroimaging pioneer Brian Wandell [2], points out that a field entirely “awoken” to the potential of FR1 may lose sight of the importance and challenge of generating new experimental data. Even as we – as a field – have turned our attention towards data-driven research, we do also need to continue to pay attention and reward the creators of novel experimental methods, and new experimental data resources to answer the many important questions that will not be answered only by the measurements that are currently available. In stark contrast to the structural biology community that has solved the protein-folding problem, we do not currently have the full scope of measurements that we will need in order to provide satisfying answers to the challenges that I opened with. We are not nearly done developing the instruments that will provide the last word in measuring the living human brain. Nevertheless, I think that a fruitful synthesis of these opposing views does exist in the hope that data-driven research will – by exposing and acknowledging the limits of its inferences – spur and motivate new experimental efforts. There is plenty of room for data scientists and experimentalists to collaborate on advancing new measurement techniques, as well as the ways in which they can be shared and used widely. I find an encouraging sign in the clear progress within and between large consortium studies. For example, the remarkable technical advances that were made as part of the Human Connectome Project, or novel quantitative MRI measurements in the Health Brain and Child Development (HBCD) dataset that are an exciting sign of iterative improvement, expanding the scope of possible investigations with these datasets. For OHBM as a community, and particularly for those of us engaged in data-driven research, it is important that we remain curious about and continue to celebrate development of new measurement methods, and experimental studies that can only be done in small samples, so that the community spans the full range of research practice.
FR2: Code sharing and re-execution
The development of open-source software for science in the past two decades has also been nothing short of remarkable. In 2012, Jake Vanderplas, at the time an astronomy graduate student at the University of Washington, and a contributor to many open-source software projects wrote a blog post titled “Why Python is the Last Language You’ll Have To Learn”.9 In his post, Vanderplas pointed out that one of the great strengths of the Python ecosystem for software is that it not only poses a lower barrier to entry as a user, but maybe even more importantly, that with the transition of many of these tools to open development on the GitHub platform, the barrier to become a contributor to these software projects has also been lowered substantially. But one might wonder whether these effects translate to the more niche projects within a field such as neuroimaging, which target a much smaller audience, and do not have the same cross-over appeal to practitioners in other sectors (the projects that Vanderplas analyzed already had significant uptake in industry as he was writing). Following Vanderplas, in Figure 1 I have plotted the (approximate) cumulative number of contributors to a range of open-source software projects that focus on human brain mapping. In place of more thorough analysis of these data, I have added some idiosyncratic historical milestones to the plot, which may or may not explain some of the trends that are observed. For those of us who are involved in these communities, these plots represent myriad personal interactions, professional and scientific milestones, and the challenges of ongoing maintenance of projects on which we and others build our science.
The addition of new contributors is, of course, just one metric to assess the growth of a project. Nevertheless, it is one that I find compelling, because it directly demonstrates the way in which different individuals build on each other’s work. To me, that is one of the most compelling promises of FR as a driver of coherent scientific advances. It would be interesting to ask what are the characteristics – technical, social, and topical – that facilitate growth of a developer community, and to ask what are the costs and benefits of such growth. For example, as I understand it, both AFNI and MRTRIX are primarily developed in compiled languages (C/C++), which require substantially more knowledge to write, constituting a higher barrier to entry for students and researchers without the necessary engineering background. These technical choices have their merits of course, in robustness, portability, and performance. Similarly, a project such as nibabel, though clearly very broadly useful, may pose a challenge for new contributors because of the technical complexity of its domain – the reading and writing of neuroimaging data formats. Not exactly the material of a PhD thesis, even if it is absolutely required for so many. Overall, the picture that appears represents the fact that many of these projects do have a steady influx of new contributors, suggesting that the tools will continue to adapt to new scientific and technological developments. This bodes well for the software elements of FR in human brain mapping. But as pointed out by Donoho, and of significance to the broader discussion of FR in human brain mapping, open-source software by itself is not sufficient for FR2. This is because, as everyone who has tried it knows, openly available software and access to open data is only one of the steps required to build on previous work. Cloud platforms for neuroscience, such as BrainLife,11 CBrain,12 and EBrains,13 and other technologies such as Neurodocker and Neurodesk14 complement open-source software by making its deployment more straightforward for researchers. The Jupyter ecosystem, used by many of the aformentioned neuro-centric platforms, and also highlighted by Donoho, also offer compelling interdisciplinary solutions for deployment of data and code on cloud-based systems.
FR3: Common challenges
As Donoho points out, FR1 + FR2 constitute what is commonly known as “Open science”. Our community has embraced many characteristics of open science, with bursts of exuberant energy at the annual meeting devoted to the activities of the Open Science SIG and the Open Science Room, as well as the recognition of replication efforts and the awardees of the annual Open Science Award. These activities – together with the Brainhack hackathons that are organized as a pre-conference event – are highly fruitful, and have provided a basis for many ongoing international collaborations, such as those that are described in Figure 1.
The adoption of FR3 – platforms and datasets for common quantitative evaluation of incremental progress on specific datasets and questions – is less evident. Donoho refers to these as examples of the “common task framework”. In this framework, a community of practitioners direct their attention to a common task, with well-defined metrics to define success and ways to publicly broadcast the results to the community. Paradigmatic examples include the ImageNet challenge and Kaggle competitions, which have both helped advance EML in various ways.
There are areas of human brain mapping in which the common task framework is indeed already rather regularly employed. For example, the Algonauts challenge15,16 was designed as a machine learning competition focused on the prediction of brain responses to natural images. Similarly, the fiber-cup phantom17 and online tractometer system18 are used to evaluate diffusion-based tractography methods, and several other challenges have been organized in this field. Both of these case have benefitted from the overlap that they share with fields in which the common task framework is more commonly employed: computer vision and biomedical engineering. And in both cases, common tasks and standard evaluation approaches continue to mark progress. Ultimately, one of the benefits of the common task framework for science is that it incentivizes and rewards incremental progress. This is because, within this framework, any measurable advance relative to the state of the art is celebrated. This is in stark contrast to the use of “incremental” as a mild derogatory by journal reviewers and editors to motivate the rejection of articles that have contributed “merely an increment”. As Donoho points out, together with FR1 and FR2, these incremental improvements made by individual researchers can ultimately add up to substantial and rapid advances made by the community as a whole. It is not hard to imagine how this framework could be expanded to many areas of human brain mapping. In particular, there are many technical aspects of human brain mapping – automated approaches to quality control, denoising algorithms – to name just a couple, which could benefit from a community-based incremental approach. Advances made in this manner would redound to the benefit of many studies that will follow.
On the other hand, there are of course many scientific questions and discoveries that do not easily lend themselves to a common task framing. Even within some of the areas in which it has been used, there are signficant questions to ask about the validity of these benchmarks. For example, many of the benchmarks in diffusion-based tractography score their results based on their ability to closely match results that are obtained with histological and invasive tract-tracing methods. While a convergence of results between these different measurements is a positive indication, it is not clear that one has a preferred status as ground truth. This is because each of these methods has its own set of limitations. Rather, using diffusion-based tractography there may be different methods that answer different questions, and the best method to use may depend on characteristics of the data, on the scientific question that is being asked, and many other criteria. In this case, declaring a “winning method” may be counter-productive to broad scientific inquiry.
How do we advance FR in human brain mapping?
It should come as no surprise, given the context in which I am writing the present article, that I think that one of the major accelerators of progress towards FR in human brain mapping is education. In particular, I will continue to advocate, even in this age of automated code generation – maybe even more-so in this age of automated code generation! – that researchers should learn how to program. This is not because I think that every researcher needs to become a software engineer, or that we all need to know how to set up the foundational systems that facilitate FR. Rather, I continue to consider programming an important tool for creative computational thinking. I believe that – especially in data-driven research, but not only – programming is an important tool in scientific creativity. Borrowing an analogy from the late Steve Jobs, programming is the equivalent of a bicycle for the mind [3]. In contrast to a train, you might not get as fast to where you are going, but you can take a less beaten path and get to places that no one has reached before. This means that it facilitates creative and generative approaches to data analysis, even in a world of full re-executable research pipelines.
Fortunately, I think that the barriers to learn how to program are becoming lower, and there are more opportunities for students in our field to become productive and even creative users of programming. I strongly believe that the open-source software projects highlighted in Figure 1 constitute communities of practice within which aspiring brain mappers can learn from and with other practitioners through peripheral contributions to the software that wouldn’t even occupy too much of their bandwidth. And, given their high potential to contribute to FR, we should carefully consider how we motivate and support even broader involvement in these communities and how we generate broader institutional support for the efforts required to keep this crucial intellectual infrastructure thriving.19
Perhaps non-intuitively, even while widespread use of models and services that generate code based on natural language prompts will make programming easier and easier, their use will also increase the need for deep programming expertise. In particular, we will require more training in crucial practices for software quality control: software profiling, software testing and code review. This is because software that is written by these systems in the absence of critical human supervision is likely to generate erroneous conclusions for the foreseeable future. That said, tools for programming with these models are among the most rapid frontiers of development, and new paradigms for thinking about software are likely to arise, which will guide how we engage in creative and rigorous computational thinking – no “vibe coding” [4] please! Rigor and computational creativity – rather than programming per se – should continue to be our objective, even as new technologies are developed to interact with computational instruments [5].
Secondly, we need to develop a critical stance on the role of machine learning methods in science. This includes both development of clarity on the technical considerations required for trustworthy scientific machine learning,20 as well as consideration of the epistemological alignment between machine learning and the goals of an investigation.21 Whether data-driven research that embodies FR will produce advances in our understanding of the brain, and will translate into solutions to health challenges, depends to a large extent on whether this research will generate durable human insight. Insights that matter cannot arise solely by a mechanical set of operations on data and, almost by necessity, require human interpretive action. Frictionless reproducibility should be complemented with the friction of thinking, discussing, debating and understanding. It is possible that some of the creativity in coming years will be devoted to making the results of machine learning methods more understandable to humans. Interestingly, in an oroborous of scientific happenstance, even though a neuroscientist may not be able to understand a microprocessor22 (but why would we?), it turns out that understanding the recent wave of artificial neural network models may rely on scientific logic that resembles neuroscience – maybe even more than electrical engineering. The cutting edge of the field of mechanistic interpretability work that aims to uncover the ways that these models operate has a distinct empirical neuroscience/psychology flavor to them.23 It is hard to read this work and deny that it is steeped in creative ingenuity. Meanwhile, the hope is that new and more refined – and more energy-efficient! – models may arise from further consideration of neurobiological details.24 These recent developments suggest the potential for a virtuous cycle of dialogue between these areas of research that will lead to mutual improvements.
Finally, one of the distinct benefits of the habits that FR engenders are the way in which other researchers easily become collaborators in a shared endeavour. This will make research more enjoyable for many of us, surfacing what is often too implicit in academic scientific culture: that we are all in this together. These habits also tend to crumble international borders [6]. Particuarly in these times of political peril to our mission as scientists, and to international collaboration more broadly, human brain mapping could set an unabashed agenda of collaboration that transcends national borders. One may hope that our adoption and adaptation of the elements of FR to our field’s unique needs will serve not only to accelerate our science, but also our work as a global community.
Acknowledgements
Thanks to Franco Pestilli and Jason Yeatman for reading earlier versions of this piece and providing helpful feedback. Thanks also to Na’ama Rokem for helpful conversations about this article.
Funding Sources
AR’s work was funded by National Institutes of Health grants MH121868, MH121867, R25MH112480, R01AG060942, U19AG066567, and R01EB027585, as well as by National Science Foundation grants 1934292, and 2334483. Open-source software development was also supported by the Chan Zuckerberg Initiative’s Essential Open Source Software for Science program, the Alfred P. Sloan Foundation and the Gordon & Betty Moore Foundation.
Conflicts of Interest
The author declares no conflicts of interest.
It is worth noting that even many of the data available in PDB may be considered limited. For example, because they record the crystal structure of proteins, which is a static configuration not neccesarily well-representing all biological conformations of these molecules.
Full disclosure: I was a postdoc in Brian’s lab 2011-2015.
He said it best, of course: https://www.youtube.com/watch?v=ob_GX50Za6c
A term coined by Andrej Karpathy here: https://x.com/karpathy/status/1886192184808149383
As a side note, we should also be concerned about the environmental impact of these systems, particularly as necessary coordinated political action on climate change is increasingly likely to be derailed by greed and nihilism. Sometimes I feel like we are living in a morbid Borgesesque parable in which a civilization cooks itself in pursuit of self-knowledge.
though some platforms have sadly been all-too-effective in enforcing some international boundaries. For example, the Google and Gitlab code-sharing platforms that have limited access for collaborators in Cuba