A brain scan picture, with coloured patches over a grey brain, looks like a direct view of thinking. It is the end of a long chain: a physical signal, a statistical test run across the whole brain, and many choices about how to analyse the data. Each link can be done well or badly, and the picture alone does not show which.
What the scanner measures
The National Institute of Biomedical Imaging and Bioengineering (NIBIB) describes functional MRI (fMRI) as a specialized MRI used “to observe brain structures and determine which areas of the brain “activate” (consume more oxygen) during various cognitive tasks.”
The signal behind that description was reported by Ogawa and colleagues in 1990. They showed that deoxyhemoglobin, the form of haemoglobin that has given up its oxygen, is “a naturally occurring contrast agent” for MRI, giving images whose contrast reflects “the blood oxygen level.” They named it blood oxygenation level-dependent (BOLD) contrast. A 2011 overview by Glover describes BOLD fMRI as depicting “changes in deoxyhemoglobin concentration consequent to task-induced or spontaneous modulation of neural metabolism.”
So the scanner follows blood oxygen, which changes with the brain’s activity; it does not record nerve cells firing. In a 2008 review, Logothetis wrote that “fundamental questions concerning the interpretation of fMRI data abound, as the conclusions drawn often ignore the actual limitations of the methodology.”
Testing the whole brain at once
A brain scan is divided into many small volumes, and a study typically tests each one, or each cluster of neighbouring ones, for a change. Testing so many places at once makes false positives likely unless the statistics correct for it.
Eklund, Nichols and Knutsson (2016) checked those corrections. They used resting-state scans from 499 healthy people, in which there was no real task to detect, to run 3 million group analyses with the common software packages SPM, FSL and AFNI. “In theory, we should find 5% false positives (for a significance threshold of 5%), but instead we found that the most common software packages for fMRI analysis (SPM, FSL, AFNI) can result in false-positive rates of up to 70%.” The packages’ standard (parametric) methods were conservative when each small volume was tested on its own, but invalid for cluster-based tests, which, the authors’ results suggest, was mainly because the way the data were correlated across space did not follow the assumed Gaussian (bell-curve) shape. A nonparametric permutation test gave the expected rates for both.
The paper’s original summary said the results questioned “the validity of some 40,000 fMRI studies.” In a published correction later in 2016, the authors replaced that with “a number of fMRI studies,” narrowed “neuroimaging results” to “weakly significant neuroimaging results,” and rewrote a second sentence in the paper’s discussion so that it no longer mentioned 40,000 studies. They stated that the errors “do not affect the conclusions of the article.”
Same scans, different answers
In the Neuroimaging Analysis Replication and Prediction Study, Botvinik-Nezer and colleagues (2020) asked “70 independent teams to analyse the same dataset, testing the same 9 ex-ante hypotheses.” “[N]o two teams chose identical workflows to analyse the data,” and this flexibility “resulted in sizeable variation in the results of hypothesis tests, even for teams whose statistical maps were highly correlated at intermediate stages of the analysis pipeline.”
The picture was not only one of disagreement: a meta-analysis that combined information across teams “yielded a significant consensus in activated regions.” Researchers in the field, asked to predict the results in prediction markets, overestimated how likely significant findings were, even those with direct knowledge of the dataset. The authors emphasize the importance of “validating and sharing complex analysis workflows,” and the need for “performing and reporting multiple analyses of the same data.”
How many people a study needs
Some brain scan studies ask how differences between people’s brains relate to differences in their behaviour or traits. Marek and colleagues (2022) call these brain-wide association studies (BWAS). Using three large datasets with “a total sample size of around 50,000 individuals,” they found that such associations “were smaller than previously thought, resulting in statistically underpowered studies, inflated effect sizes and replication failures at typical sample sizes,” where the median neuroimaging study had “about 25” participants. As samples “grew into the thousands, replication rates began to improve.” They contrast these with approaches that have larger effects, such as lesion, intervention and within-person studies, and say it is BWAS that need thousands of participants.
How large a sample multivariate methods need was debated in a 2023 exchange in Nature. Spisak, Bingel and Wager (2023) reported that a multivariate method, which combines many brain measures, can give replicable effects with samples substantially smaller than thousands “in some cases”: in one high-quality dataset, it gave replicable effects “with sample sizes of 75–500 for 5 of 6 phenotypes tested.” In reply, Tervo-Clemmens and colleagues (2023) repeated 64 multivariate analyses with the same methods in a larger, more diverse dataset and found that “31% replicated at n = 1,000, dropping to 14% at n = 500 and none at n = 75.” Spisak and colleagues wrote that they agree “large samples are needed when conducting univariate BWAS,” which test brain measures one at a time. Tervo-Clemmens and colleagues argued that, “regardless of the method,” results from small samples need confirming “in large, diverse, independent samples.”
INSTAR’s research interest
INSTAR’s interest is in the questions a careful reader can ask before relying on a brain scan study. Is the claim about activity within each person, or about differences between people? How many people were scanned? How was the whole brain tested, and was the correction for many tests one that has been checked? Were the analysis choices fixed in advance, shared, or tried in more than one way?
These questions connect to the neuroscience program page, which discusses open neuroimaging data and brain-wide association studies. The brief on psychology replication asks similar questions about a single study, and the page on how this site checks a fact describes the checks applied to the sources cited here.