Challenges in irreproducible research

The solutions adopted by the high-energy physics community to foster reproducible research are examples of best practices that could be embraced more widely. This first experience suggests that reproducibility requires going beyond openness.

Instead of arguing about whether results hold up, let’s push to provide enough information for others to repeat the experiments, says Philip Stark.

Nature authors say a reproducibility checklist is a step in the right direction, but more needs to be done.

I was hired to ferret out errors and establish routines that promote rigorous research, says Catherine Winchester.

Replicating our work took four years and 100,000 worms but brought surprising discoveries, explain Gordon J. Lithgow, Monica Driscoll and Patrick Phillips.

A survey of Nature readers revealed a high level of concern about the problem of irreproducible results. Researchers, funders and journals need to work together to make research more reliable.

Survey sheds light on the ‘crisis’ rocking research.

Mistakes in peer-reviewed papers are easy to find but hard to fix, report David B. Allison and colleagues.

It may not be sexy, but quality assurance is becoming a crucial part of lab life.

Humans are remarkably good at self-deception. But growing concern about reproducibility is driving many researchers to seek ways to fight their own worst instincts.

P values, the 'gold standard' of statistical validity, are not as reliable as many scientists assume.

Research needs an authoritative forum to hash out collective problems, argue C. K. Gunsalus, Marcia K. McNutt and colleagues.

Software that uncovers suspicious papers will do little for a community that does not confront organized research fraud, says Jennifer Byrne.

As the imaging technique produces ever-sharper protein structures, researchers are racing to develop tools to assess how accurate they are.

Collaboration across institutes can train students in open, team science, which better prepares them for challenges to come, says Katherine Button.

Researchers replicated 62% of social-behaviour findings published in Science and Nature — a result matched almost exactly by a prediction market.

Online technologies make it easy to share precise experimental protocols — and doing so is essential to modern science, says Lenny Teytelman.

A simple software toolset can help to ease the pain of reproducing computational analyses.

Instead of arguing about whether results hold up, let’s push to provide enough information for others to repeat the experiments, says Philip Stark.

I was hired to ferret out errors and establish routines that promote rigorous research, says Catherine Winchester.

Research needs an authoritative forum to hash out collective problems, argue C. K. Gunsalus, Marcia K. McNutt and colleagues.

Software that uncovers suspicious papers will do little for a community that does not confront organized research fraud, says Jennifer Byrne.

A multi-laboratory study finds that single-molecule FRET is a reproducible and reliable approach for determining accurate distances in dye-labeled DNA duplexes.

Customize the experiment for the setting instead of adjusting the setting to fit a classical design.

Many factors can skew the results of a widely used amplification technique for microbiome analysis, but researchers are finding strategies for getting at the truth.

An extensive evaluation of differential expression methods applied to single-cell expression data, using uniformly processed public data in the new conquer resource.

The publishing system builds in resistance to replication. Paul Gertler, Sebastian Galiani and Mauricio Romero surveyed economics journals to find out how to fix it.

Replication is not enough. Marcus R. Munafò and George Davey Smith state the case for triangulation.

Using the ORI of plasmids used in enhancer assays as the sole core promoter and inhibiting the interferon I response triggered by plasmid transfection greatly reduces false positive and negative results in single-candidate and massively parallel enhancer assays and enables genome-wide enhancer screens.

Creating a culture of replication takes prizes, grants and magnanimity — as well as publications.

As debate rumbles on about how and how much poor statistics is to blame for poor reproducibility, Nature asked influential statisticians to recommend one change to improve science. The common theme? The problem is not our maths, but ourselves.

Chemical probes and screening libraries can easily get mixed up or messed up, causing misleading results for unwary biologists.

China has a lucrative market for fake research reagents. Some scientists are fighting back.

Efforts to reduce irreproducibility in research must also tackle the temptation to cheat, argue Donald S. Kornfeld and Sandra L. Titus.

Survey sheds light on the ‘crisis’ rocking research.

It may not be sexy, but quality assurance is becoming a crucial part of lab life.

Humans are remarkably good at self-deception. But growing concern about reproducibility is driving many researchers to seek ways to fight their own worst instincts.

More fields should, like particle physics, adopt blind analysis to thwart bias, urge Robert MacCoun and Saul Perlmutter.

Crowdsourcing research can balance discussions, validate findings and better inform policy, say Raphael Silberzahn and Eric L. Uhlmann.

Antibodies are the workhorses of biological experiments, but they are littering the field with false findings. A few evangelists are pushing for change.

Ridding science of shoddy statistics will require scrutiny of every step, not merely the last one, say Jeffrey T. Leek and Roger D. Peng.

Naivety about promiscuous, assay-duping molecules is polluting the literature and wasting resources, warn Jonathan Baell and Michael A. Walters.

Notices should make obvious whether a withdrawal of research is the result of misconduct or a genuine mistake, says Daniele Fanelli.

Environmental factors lie behind many irreproducible rodent experiments.

A bill to make data for environmental regulation more transparent reveals the fuzzy boundary between science and ideology, argues Daniel Sarewitz.

Largest replication study to date casts doubt on many published positive results.

Initiative trying to validate 50 cancer papers finds difficulty in accessing original study data.

Research Resource Identifier (RRID) aims to clean up poorly referenced data.

Crowd-sourced effort raises nuanced questions about what counts as replication.

'Premature' rules for preclinical research need more flexibility and greater community involvement, say scientific society leaders.

Study on decision-making stokes controversy over power of distracted mind.

Start-up firms say robotics and software that autonomously record every detail of an experiment can transform the efficiency and reliability of research.

First report from Reproducibility Initiative highlights importance of publishing details of experimental methods.

Consensus on reporting principles aims to improve quality control in biomedical research and encourage public trust in science.

Papers in Nature journals should make computer code accessible where possible.

P values, the 'gold standard' of statistical validity, are not as reliable as many scientists assume.

Researchers and funding agencies need to put a premium on ensuring that results are reproducible, argues Jonathan F. Russell.

A new mechanism for independently replicating research findings is one of several changes required to improve the quality of the biomedical literature.

Reproducibility Initiative aims to speed up preclinical research.

Scientists and journals must work together to ensure that eye-catching artefacts are not trumpeted as genomic insights, says Daniel MacArthur.

Biologists must realize the pitfalls of work on massive amounts of data.

In the wake of high-profile controversies, psychologists are facing up to problems with replication.

Experimental biologists, their reviewers and their publishers must grasp basic statistics, urges David L. Vaux, or sloppy science will continue to grow.

Many of the studies that use animals to model human diseases are too small and too prone to bias to be trusted, says Malcolm Macleod.

Bayer halts nearly two-thirds of its target-validation projects because in-house experimental findings fail to match up with published literature claims, finds a first-of-a-kind analysis on data irreproducibility.

A multi-laboratory study finds that single-molecule FRET is a reproducible and reliable approach for determining accurate distances in dye-labeled DNA duplexes.

Camerer et al. carried out replications of 21 Science and Nature social science experiments, successfully replicating 13 out of 21 (62%). Effect sizes of replications were about half of the size of the originals.

Customize the experiment for the setting instead of adjusting the setting to fit a classical design.

The results of in vitro and in vivo screens to identify genes that are essential for the survival of a type of brain cancer show almost no overlap, underlining the need for caution when interpreting in vitro studies. See Letter p355.

Many factors can skew the results of a widely used amplification technique for microbiome analysis, but researchers are finding strategies for getting at the truth.

An extensive evaluation of differential expression methods applied to single-cell expression data, using uniformly processed public data in the new conquer resource.

Updates from the Human Microbiome Project analyse the largest known body-wide metagenomic profile of human microbiome personalization.

Leading voices in the reproducibility landscape call for the adoption of measures to optimize key elements of the scientific process.

Using the ORI of plasmids used in enhancer assays as the sole core promoter and inhibiting the interferon I response triggered by plasmid transfection greatly reduces false positive and negative results in single-candidate and massively parallel enhancer assays and enables genome-wide enhancer screens.

The finding that acute and chronic manipulations of the same neural circuit can produce different behavioural outcomes poses new questions about how best to analyse these circuits. See Article p.358

Quality control of cell lines used in biomedical research is essential to ensure reproducibility. Although cell line authentication has been widely recommended for many years, misidentification, including cross-contamination, remains a serious problem. We outline a multi-stakeholder, incremental approach and policy-related recommendations to facilitate change in the culture of cell line authentication.

The reliability and reproducibility of science are under scrutiny. However, a major cause of this lack of repeatability is not being considered: the wide sample-to-sample variability in the P value. We explain why P is fickle to discourage the ill-informed practice of interpreting analyses based predominantly on this statistic.

'Irreproducibility' is symptomatic of a broader challenge in measurement in biomedical research. From the US National Institute of Standards and Technology (NIST) perspective of rigorous metrology, reproducibility is only one aspect of establishing confidence in measurements. Appropriate controls, reference materials, statistics and informatics are required for a robust measurement process. Research is required to establish these tools for biological measurements, which will lead to greater confidence in research results.

Low-powered studies lead to overestimates of effect size and low reproducibility of results. In this Analysis article, Munafò and colleagues show that the average statistical power of studies in the neurosciences is very low, discuss ethical implications of low-powered studies and provide recommendations to improve research practices.

A wealth of microarray gene expression data and a growing volume of RNA sequencing data are now available in public databases. The authors look at how these data are being used and discuss considerations for how such data should be analysed and deposited and how data reuse could be improved.

Deficiencies in methods reporting in animal experimentation lead to difficulties in reproducing experiments; the authors propose a set of reporting standards to improve scientific communication and study design.

There are many different methods and tools available for the analysis of next-generation sequencing data. The challenges towards applying these analysis tools in a transparent and reproducible manner are presented, and a way forward for analysing these data in life sciences research is discussed.

Scientific reproducibility now very often depends on the computational method being available to duplicate, so here it is argued that all source code should be freely available.

Batch effects can lead to incorrect biological conclusions but are not widely considered. The authors show that batch effects are relevant to a range of high-throughput 'omics' data sets and are crucial to address. They also explain how batch effects can be mitigated.

C. Glenn Begley and Lee M. Ellis propose how methods, publications and incentives must change if patients are to benefit.

Too many sloppy mistakes are creeping into scientific papers. Lab heads must look more rigorously at the data — and at themselves.