Similarity-Based Assessment of Computational Reproducibility in Jupyter Notebooks CCS: Software and its engineering Reproducibility CCS: Information systems Data provenance CCS: Applied computing Computational notebooks CCS: Computing methodologies Verification and validation
A S M Shahadat Hossain, Colin Brown, David Koop, Tanu Malik · 2025
Read Similarity-Based Assessment of Computational Reproducibility in Jupyter Notebooks CCS: Software and its engineering Reproducibility CCS: Information systems Data provenance CCS: Applied computing Computational notebooks CCS: Computing methodologies Verification and validation by A S M Shahadat Hossain, Colin Brown, David Koop, Tanu Malik (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-23645.txt.
Introduction
Re-running a Jupyter notebook does not always give the same answer, thanks to randomness, library updates, and shifting machines. A S M Shahadat Hossain and 3 co-authors propose the Similarity-based Reproducibility Index, a score for how close a rerun comes. Their methods compare outputs type by type, using similarity measures built for different kinds of Python objects. Each output cell gets a number between zero and one, plus qualitative notes.
A case study applies the index to a set of notebooks. It shows how the per-type similarity scores can turn a vague sense of reproducibility into a measured one, helping teams judge whether shared computational work can be trusted.
Catalog record
| CREATOR | A S M Shahadat Hossain, Colin Brown, David Koop, Tanu Malik |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 46 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-23645
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Similarity-Based Assessment of Computational Reproducibility in Jupyter Notebooks CCS: Software and its engineering Reproducibility CCS: Information systems Data provenance CCS: Applied computing Computational notebooks CCS: Computing methodologies Verification and validation A S M Shahadat Hossain, Colin Brown, David Koop, Tanu Malik Abstract . Computational reproducibility refers to obtaining consistent results when rerunning an experiment. Jupyter Notebook, a web-based computational notebook application, facilitates running, publishing, and sharing computational experiments along with their results. However, rerunning a Jupyter Notebook may not always generate identical results due to various factors, such as randomness, changes in library versions, or variations in the computational environment. This paper introduces the Similarity-based Reproducibility Index (SRI) – a metric for assessing the reproducibility of results in Jupyter Notebooks. SRI employs novel methods developed based on similarity metrics specific to different types of Python objects to compare rerun outputs against original outputs. For every cell generating an output in a rerun notebook, SRI reports a quantitative score in the range [0, 1] as well as some qualitative insights to assess reproducibility. The paper also includes a case study in which the proposed metric is applied to a set of Jupyter Notebooks, demonstrating how various similarity metrics can be leveraged to quantify computational reproducibility. 1. Introduction In recent years, computational reproducibility has become one of the major concerns among various research communities. It has gained increasing attention since the National Academies of Sciences, Engineering, and Medicine published the report titled “Reproducibility and Replicability in Science” ( National Academies of Sciences, Engineering, and Medicine 2019 ) in 2019. Because better reproducibility can help save energy, financial cost, and time to reproduce an outcome, it is not only computer science but also other domains such as geoscience ( Steventon et al. 2022 ) , management science ( Fišar et al. 2024 ) , and social sciences ( Moody et al. 2022 ) that are aware of the need to form a reproducibility culture. Ideally, rerunning a scientific experiment is expected to generate outcomes identical to the results observed in the original experiment. However, practically, this expectation does not always hold due to various factors such as environmental catalysts, changes in methods of recording the results, or even changes in the measuring units. A computational experiment is also not much different from this manner. Rerunning the same code in the same computational environment, even with the same data, may generate results that are different from the original ones. These changes in results come in various forms. Sometimes, these changes are subtle and are even difficult to perceive, however, sometimes they can make a significant difference in the result interpretation. To understand how such changes impact the reproducibility of results, it is crucial to have a standard metric to measure the reproducibility that goes beyond the simple binary judgment of two results being completely “identical” or “different”. Jupyter Notebook is a popular web-based application that offers interactive environments for computational experiments. It allows users to divide the codes into smaller execution blocks called cells to run them separately and save the outputs. Jupyter Notebooks allow users to run and rerun the cells in any order. Though these types of flexibility facilitates the rerunnability of codes, they do not guarantee that the results will be reproduced. The results may rather change in rerun notebooks for various reasons, such as randomness, different computing environments, or changed library versions. There are a few comprehensive works ( Pimentel et al. 2019 ; Pimentel et al. 2021 ; Wang et al. 2021a ; Samuel and Mietchen 2024 ; Ahmad et al. 2022 ) on the reproducibility of Jupyter Notebooks. However, a standard metric to measure the reproducibility is still absent. In this paper, we propose a new metric called Similarity-based Reproducibility Index (SRI) to assess the reproducibility of results in Jupyter Notebooks. While the existing practices of evaluating reproducibility solely depend on a deterministic match, SRI is a combination of both quantitative and qualitative evaluation of reproducibility. It is based on the assumption that a higher similarity between two results makes them more reproducible, even if they are not identical. The quantitative evaluation of SRI leverages similarity metrics to compare two notebook cell outputs to generate a reproducibility score in the range [0, 1]. On the other hand, the qualitative evaluation makes the metric more comprehensive by providing useful insight regarding various data definition aspects of the outputs. We first applied SRI on the cell outputs extracted from notebooks and their rerun versions. After that, we accumulated all the cell-wise SRI results of the entire notebook into a notebook-wise SRI. We dumped the notebook-wise SRI results as JSON (JavaScript Object Notation) structures for better operability. Finally, we performed a case study on 15 notebooks where we generated the JSON structures for each notebook and visualized them using a web browser to cross-check the evaluation results with the notebook cell outputs and analyze the performance of the proposed metric. The rest of the paper is organized as follows: section 2 defines necessary terminologies with examples, section 3 presents the methodology of the proposed metric, and section 4 applies the metric on Jupyter Notebooks to analyze its performance. Finally, section 5 discusses related work, followed by the conclusion in section 6 . 2. Definitions 2.1. Cells in Jupyter Notebooks A cell is the primary building block and modular unit of a notebook. It facilitates modularity by allowing users to break down a computational process into discrete, manageable steps that can be executed independently or in sequence. There are three types of cells in Jupyter Notebooks: • Code cells : Being the most prominent type of cell in notebooks, a code cell is used to write code. Code cells automatically support smart code formatting, such as syntax highlighting and indentations. When this type of cell is executed, the code written in it is sent to the kernel responsible for running the code. If the code generates any results, then the result is displayed as the output of the respective code cell. • Markdown cells : This type of cell facilitates documentation of the code. The notebook author can use markdown cells to describe the code or the computational processes using natural texts, rich text, and even markup languages. The cells support text formatting using HTML and standard LaTeX notations. • Raw cells : This type of cell is the least commonly used one in notebooks. Raw cells are used to write raw codes for the notebook to be rendered or converted into different formats, such as HTML or LaTeX. Raw cells are not evaluated by notebooks. Figure 1 presents an example of a code cell and a markdown cell. Figure 1. Code Cell and Markdown Cell in a Jupyter Notebook. 2.2. Types of Code Cell Outputs Jupyter Notebook supports various types of outputs. Among them, the major types of outputs observed as the code cell outputs are as follows: Figure 2. Different Types of Code Cell Outputs in a Jupyter Notebook (a) execute_result (b) stream (c) display_data • stream outputs : This type of outputs are outputs typically displayed using the print statement. The most common MIME types under this category of outputs are text/plain and text/html . • display_data outputs :