How to Better Understand Reproducibility in Science

Updated April 11th, 2022

Reproducibility and replicability are essential to scientific data integrity. By repeating research or studies, the scientific community confirms the validity of scientific discovery. However, it’s difficult to reproduce results for various reasons, and you’ll find out why.

This article explains reproducibility (and replicability) in science and the confusing picture reproducibility paints when it comes to scientific research. Furthermore, it explains the new reproducibility crisis and what it means for science. In the end, we present you with an excellent solution for collaboration and scientific integrity - Orvium.

What is Reproducibility in Science?

Reproducibility means obtaining consistent results using the same data as the original study. Replicability means obtaining consistent results using new data or new computational results to answer the same scientific question. Reproducibility and replicability are sometimes used interchangeably in science.

Reproducibility is essential to science because it allows for more thorough research while replicability confirms our results. Many studies and experiments exist, leading to many different variables, unknowns, and things out of your control or that you cannot guarantee. But the one thing you can guarantee is that your work is reproducible. One way to do this is by being transparent.

Why is Transparency Important When it Comes to Reproducing Studies?

Due to the nature of science, it’s challenging to know if something can be reproduced because many factors can change, and results may not be correct or remain correct in time. But we know for a fact that science can only progress if there is collaboration among fellow researchers. That includes:

• Sharing results (both positive and negative)
• Providing data documentation and reporting, especially raw and unmodified data
• Being transparent about the methods and tools used to reach a consensus.

Part of that collaboration includes giving other researchers the ability to reproduce or replicate research results. This becomes difficult for others if the original study lacks correct and updated information, there is a misinterpretation of results, or no negative results are shared.

One could say that transparency and negative results sharing go hand in hand. We know from our Open Science and Negative Results Sharing article that by sharing the negative results obtained through reproducing or replicating another researcher’s work, they’re inspiring new directions for future work and contributing to the Open Science community.

Negative results don’t mean that a researcher is unskilled or that their work should be disregarded. In fact, negative results may help prove trustworthiness, because it shows that the data and methodology are authentic. And if negative results sharing was largely encouraged throughout the scientific community, more scientists or researchers would be more inclined to attempt reproduction or replication.

The Confusing Picture Reproducibility Paints

Confidence in all scientific fields is important and should be treated as such. The stakes are even higher when the results impact people’s health decisions, inform policy, or affect future scientific studies. However, the fear of non-reproducibility is widespread across all disciplines.

Further uncertainty occurs when trying to establish if a result is close enough to be called reproducible. Scientists, policymakers, funding agencies, etc., have found that results have low reproducibility for certain fields, such as psychology, forensics, and epidemiology. Meanwhile, researchers reported failure to reproduce results in biology, chemistry, medicine, physics, engineering, and earth and environmental sciences, despite believing that reproducibility of findings is crucial for the advancement of science and correct results sharing.

The ability to reproduce results is difficult because of multiple reasons:

• the exact value of a complex measurement may not be identical to another lab’s result
• the use of specialized techniques is different, given the complex nature of many of the studies performed
• many scientific studies rely on unique events or observations of one-time events (earthquakes, hurricanes, epidemics, climate studies, etc.), making running a controlled experiment near impossible
• metadata issues and not enough researchers who share the code and software used to assure reproducible and reusable research.

The New Reproducibility Crisis, and What it Means for Science

The reproducibility crisis (also known as the replicability crisis) is an ongoing methodological crisis based on the idea that many scientific studies or results are either difficult or impossible to reproduce. Thus, the validity, credibility, and the ability to build upon former theories or other data is called into question. This is deemed a crisis because the ability to reproduce results is an essential part of the scientific process.

However, instead of focusing on this “crisis” many scientists talk about, let’s focus on establishing confidence in the quality of scientific data

Instead of focusing on this so-called “crisis” many scientists talk about, let’s focus instead on establishing confidence in the quality of scientific data. This becomes crucial as our ability to store, mine, and transfer large amounts of data continues to increase. Unfortunately, there are currently few tools that assess the quality of data. An excellent example of a research center that uses partially automated data quality assessment tools is the National Institute of Standards and Technology (NIST) Thermodynamics Research Center.

Studies that involve complex experimental systems are challenging to replicate, especially for researchers who are not experts in that particular field. The Research Data Alliance is exploring a unique solution to facilitate the collecting, sharing, and reporting of details for tools and protocols. This could significantly improve the tracing of data back to a particular tool and its calibration information.

In turn, this can improve the overall quality of research data and allow data and software sharing in and across disciplines to be a strong motivator for replicability. A researcher or scientist who wants to replicate or reproduce a study or experiment will be able to do so without being an expert in that field. Ultimately, the adoption of a universal framework for data reporting would be extremely beneficial for science.

Let’s Improve Outcomes

The ability to replicate or reproduce research results will depend on and be limited by the reliability of the assumptions, data, and software on which the conclusions are based. Additionally, better systems must be in place to establish confidence in results that everyone can understand. Only in this way will we finally be able to replicate or reproduce studies successfully.

Orvium focuses on collaboration among colleagues and allows researchers, reviewers, and publishers to manage your research on an open platform. Being a cloud solution, there is no limit to what you can do with your data. Curious to know more about how we improve scientific publishing, collaboration, and transparency? Check out our platform and discover what matters to you.