ºÚÁÏÍø

A new study published in Nature, "," finds that scientific conclusions can shift dramatically depending on who conducts the analysis.  

More than 450 independent researchers from around the world conducted over 500 re-analyses of datasets from one hundred previously published studies in the social and behavioural sciences. All analysts received the same data and the same central research question, but they were free to carry out the analysis based on their own expert judgment. 

The study revealed that scientific conclusions can change substantially depending on who performs the analysis. 

Discrepancies are not due to lack of expertise 

Over the past decade, the social and behavioural sciences have undergone substantial reforms aimed at making research more transparent, rigorous and reliable. One important question, however, is receiving increasing attention: to what extent do research findings depend on the specific way in which data are analysed? 

‘Research often involves choosing a single analytic path, where one dataset is typically analysed by one researcher or one team. But there are also other reasonable ways to analyse the same data,’ says Aalto University Assistant Professor Christoph Huber. 

While peer review assesses methodological acceptability, it rarely reveals what results might have emerged under alternative, yet equally defensible, statistical decisions. 

‘Sometimes researchers disagree about how certain types of data should be analysed in the first place. At other times, the phenomena themselves do not permit a single, unequivocal interpretation,’ says Enrico Glerean. 

‘I personally reanalysed an experimental study and reached the same results. This may be because experimental data allow less room for analysis-dependent, open-ended conclusions,’ Huber notes. 

We need humans more than machines 

One way to increase the reliability of research findings is a so-called multiverse and multi-analysts analysis approach, in which the same dataset is analysed by different teams using different methodological choices, rather than relying on a single team and a single set of analytic decisions. 

‘Multiverse and multi-analysts analyses can help show which results are robust regardless of the methods used. This also means that the social side of science becomes even more important: researchers should discuss different approaches and challenge one another,’ Glerean says. 

Glerean also notes that popular generative AI tools are exerting an ever-greater influence on everything, and it is often trained on vast amounts of old material. This means it also draws on findings that have been shown to be irreproducible, misleading or simply wrong. 

‘Science advances faster, and it self-corrects better than large language models trained on everything that was ever written. We will still need humans more than machines when it comes to understanding what counts as the best explanation of a phenomenon,’ Glerean says.

Original news article:

Nature: