University researcher takes part in major study driving improvement in scientific methodology

The scientific process involves many steps, from developing a theory and creating hypotheses to collecting and analysing data.

Each of these steps can potentially affect the final conclusions, but to what extent? Specifically, will different researchers reach different conclusions based on the same data and hypotheses?

Now a major study published in the journal Nature has sought answers to these questions in the context of brain imaging research, but which can be applied much more widely.

Lecturer in Psychology Dr Nadège Bault was part of a team of almost 200 researchers who took part in the Neuroimaging Analysis, Replication and Prediction Study (NARPS), investigating how variable the findings of brain imaging research are as a result of researchers’ choices about how to analyse the data. The project was led by Dr Rotem Botvinik-Nezer from Dartmouth College and Dr Tom Schonberg from Tel Aviv University, along with Dr Russell Poldrack from Stanford University.

The results of NARPS show for the first time that there is considerable variance when the same complex neuroimaging dataset is analysed with different ‘analysis pipelines’ to test the same hypotheses.

No two teams chose identical workflows to analyse the data, leading to a wide range of different conclusions. The number of teams reporting a statistically significant outcome for each hypothesis varied substantially; for five of the hypotheses there was substantial disagreement, with between 20 and 40 per cent of the analysis teams reporting a statistically significant result. In addition, researchers in the field were over-optimistic regarding the likelihood of significant results, even if they had analysed the data themselves.

Despite this, a meta-analysis performed on teams’ intermediate results showed convergence for most hypotheses. This suggests the last steps of the analysis workflow, which are the most susceptible to human bias and interpretation, account for a substantial part of the variability in conclusions and therefore the human factor might be more to blame than the technique itself.

Dr Bault, who is Co-Lead for the MRI (Magnetic Resonance Imaging) Lab in the University’s new Brain Research & Imaging Centre (BRIC) said:

“This exciting international project highlighted the strong drive of many researchers to improve the quality of their work. This can only be achieved by cooperating in setting new standards. One aspect is the systematic sharing of data, but this on its own is not enough.

“What this study shows is that peer-validation of complex analysis workflows by making analysis code and intermediate results available is important as well. We are implementing these good practices in the BRIC MRI lab.”

The findings will be of interest to members of the neuroimaging research community, and those in every other field with complex analysis procedures, where researchers have to make many choices about how to analyse the data.

NARPS found that analytic flexibility can have substantial effects on scientific conclusions. However, the finding that underlying statistical results were relatively consistent across groups, and meta-analyses led to more convergence, suggests ways to improve research, highlighting the importance of validating and sharing complex analysis workflows, and the need for multiple analyses of the same data.

The fact that almost 200 individuals were each willing to put tens or hundreds of hours into such a critical self-assessment demonstrates the commitment of scientists in this field to assessing and improving the quality of data analyses.

Brain Research & Imaging Centre

The Brain Research & Imaging Centre (BRIC), the most advanced multi-modal brain imaging facility in the South West, will provide the sea-change to enhance the quality of our research in human neuroscience.

With seven cutting-edge human research laboratories, BRIC will include an MRI suite with the most advanced 3-Tesla scanner in the region. It will critically advance our enquiry toward the most advanced brain research, improved radiological diagnostics and better patient care.

Find out more about the facility

“A diverse range of internationally excellent researchers in the field of neuroscience underpins the University of Plymouth’s strong track record in brain research. We ask important questions that draw upon expertise from the single molecule to the whole human, explore the basic mechanisms of cognition and behaviour and apply advanced computational methods to improve our understanding of the brain in health and disease.

“I am confident that with the new BRIC facility, we will realise our aim to translate this world-leading research into outcomes that improve the lives of everyone, locally, nationally and internationally.”

Stephen Hall
Director of the Brain Research & Imaging Centre
Professor of Human Neuroimaging, University of Plymouth

Enhancing research through BRIC 

The MRI Laboratory in the BRIC facility will provide an environment for investigating theories with cutting-edge neuroimaging methodologies, made possible by a state-of-the-art 3-Tesla Siemens Prisma MR scanner, which will be the most advanced scanner in the region.

Getty

Research in the School of Psychology

Plymouth is a centre of excellence in psychological research. In the last Research Excellence Framework assessment over 80% of our research outputs were rated as either international (3*) or world-leading (4*) quality. This puts us in the top 20 nationally on this measure, and above institutions such as Bristol, UCL, Manchester, Southampton, Bournemouth, and Portsmouth.

We have a thriving PhD community, with around 80 doctoral students, purpose-built research facilities.

Learn more about the research in the School of Psychology

Faculty of Health

Exceptional clinical and academic learning, social engagement and research in medicine, dentistry, nursing, psychology and health professions.

#FirstChoiceForHealth

Discover more about the Faculty of Health