A University of Plymouth researcher has contributed to a large-scale international study which has highlighted that scientific conclusions can shift dramatically depending on who conducts the analysis.
The research, part of a series of articles published in the journal Nature, asked a team of 457 independent analysts from around the world to reassess data from 100 previously published studies across the social and behavioural sciences.
They included
Dr Nadège Bault , Lecturer in Cognitive Neuroscience within the University’s
School of Psychology , who was invited to become involved in the study having previously been involved in a similar assessment of statistical analyses in the field of neuroimaging.
All the analysts received the same dataset and the same key research question, but were given freedom in how to conduct the analysis based on their informed judgment.
Although most of the reassessments broadly supported the main claims of the original studies, effect sizes, statistical estimates, and levels of uncertainty often differed meaningfully, with the analysts reaching the same conclusion as the original authors in just one third of cases.
Importantly, the researchers say, these discrepancies were not due to a lack of expertise as experienced researchers with strong statistical backgrounds were just as likely to arrive at divergent results as others.
At the same time, observational studies proved less robust than experimental ones, suggesting that more complex data structures allow greater analytical flexibility – and thus greater uncertainty.
The project set out to strengthen the credibility of studies in the social and behavioural sciences and, its leads say, delivers a clear message that scientific objectivity does not lie in identifying a single “true” analysis, but in making the space of plausible alternatives transparent – both in research reports and in communication with the broader scientific community.
Dr Bault, who works within the
Centre for Therapeutic Ultrasound (CENTUS) and is Head of Operations for the
Brain Research & Imaging Centre (BRIC) become involved in the study as she wanted to explore whether findings were reproducible when variability in collecting data was set aside.
Dr Nadège Bault
