A Study of Scientific Computational Notebook Quality CCS: Software and its engineering Context specific languages CCS: Applied computing Physical sciences and engineering

Shun Kashiwa, Ayla Kurdak, Savitha Ravi, Ridhi Srikanth, Angel Thakur, Sonia Chandra, Jonathan Truong, Michael Coblenz · 2026

Read A Study of Scientific Computational Notebook Quality CCS: Software and its engineering Context specific languages CCS: Applied computing Physical sciences and engineering by Shun Kashiwa, Ayla Kurdak, Savitha Ravi, Ridhi Srikanth, Angel Thakur, Sonia Chandra, Jonathan Truong, Michael Coblenz (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-22726.txt.

Introduction

Kashiwa and co-authors study the quality of scientific computing notebooks. They built a corpus of 518 code repositories from Code Availability notes in 2024 Nature papers. Quality was judged on reproducibility, readability, and reuse. Of 19 notebooks the team tried to run, only two reproduced. Missing data files and dependency issues caused most failures. Code duplication was common, with 326 clone classes of at least ten lines. Mutation analysis also showed tangled state changes. The authors call for better tools to support sound scientific software.

Catalog record

CREATORShun Kashiwa, Ayla Kurdak, Savitha Ravi, Ridhi Srikanth, Angel Thakur, Sonia Chandra, Jonathan Truong, Michael Coblenz
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE66484
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.22726

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

A Study of Scientific Computational Notebook Quality CCS: Software and its engineering Context specific languages CCS: Applied computing Physical sciences and engineering Shun Kashiwa, Ayla Kurdak, Savitha Ravi, Ridhi Srikanth, Angel Thakur, Sonia Chandra, Jonathan Truong, Michael Coblenz Abstract . The quality of scientific code is a critical concern for the research community. Poorly written code can result in irreproducible results, incorrect findings, and slower scientific progress. In this study, we evaluate scientific code quality across three dimensions: reproducibility, readability, and reusability. We curated a corpus of 518 code repositories by analyzing Code Availability statements from all 1 , 239 1,239 Nature publications in 2024. To assess code quality, we employed multiple methods, including manual attempts to reproduce Jupyter notebooks, documentation reviews, and analyses of code clones and mutation patterns. Our results reveal major challenges in scientific code quality. Of the 19 notebooks we attempted to execute, only two were reproducible, primarily due to missing data files and dependency issues. Code duplication was also common, with 326 clone classes of at least 10 lines and three instances found among 637 637 of the 1 , 510 1,510 notebooks in our corpus. These duplications frequently involved tasks such as visualization, data processing, and statistical analysis. Moreover, our mutation analysis showed that scientific notebooks often exhibit tangled state changes, complicating comprehension and reasoning. The prevalence of these issues—unreproducible code, widespread duplication, and tangled state management—underscores the need for improved tools and abstractions to help science build reproducible, readable and reusable software. 1. Introduction Software plays an increasingly important role in research ( Johanson and Hasselbring 2018 ) , but quality and correctness concerns ( Trisovic et al. 2022 ; Eisty et al. 2025 ; Johanson and Hasselbring 2018 ) and the low reproducibility rate of scientific code ( Trisovic et al. 2022 ; Tiwari et al. 2021 ; Samuel and Mietchen 2024 ) threaten the validity of research findings. Motivated by these concerns, there has been a growing movement towards open science, which aims to make research more transparent, accessible, and collaborative ( Vicente-Saez and Martinez-Fuentes 2018 ) , in part by improving the availability of research materials, including code ( Grant et al. 2025 ) . Jupyter, a popular environment for data analysis that combines Markdown, code, and outputs, has been lauded for supporting open science by enabling reproducible workflows, promoting shareability, and unifying code and narrative (“one study—one document”) ( Beg et al. 2021 ; Randles et al. 2017 ; Granger and Pérez 2021 ) . However, Jupyter notebooks have many usability and code quality issues ( Singer 2020 ; Pimentel et al. 2019 ; Wang et al. 2020b ; Grotov et al. 2022 ) . A study of over 800,000 general-purpose notebooks found that only 24% were fully executable, and a mere 4% produced the same results ( Pimentel et al. 2019 ) . This is troubling for scientific pursuits, where reproducibility is paramount, and may threaten the validity of findings produced by computational notebooks. Though scientific and traditional software development have distinct contexts and goals ( Kelly 2015 ; Maćkowiak et al. 2018 ) , we can still leverage knowledge from traditional software engineering to raise scientific code quality and correctness ( Heaton and Carver 2015 ; Storer 2017 ; Arvanitou et al. 2021 ) . In what ways do real-world scientific Jupyter notebooks meet or fail to meet the goals of open science? To answer this, we collected a corpus of 518 code repositories, extracted from all 1,239 Nature publications in 2024. We choose Nature because it (1) spans disciplines, so it enables examining code across many fields, each of which may have their own community standards; (2) is prestigious, so we can expect a high quality of research and associated materials; and (3) requires a Code Availability statement in any paper where custom code was central to the results of the study. To clarify what it means for a notebook to “meet the goals of open science,” we drew on the Transparency and Openness Promotion (TOP) Guidelines ( Grant et al. 2025 ) , the FAIR principles of research software ( Barker et al. 2022 ) , and Nature ’s reporting standards ( Nature Portfolio [n. d.] ) to identify three relevant software qualities: (1) Reproducibility: Another researcher should be able to run the code and produce the same results. (2) Readability: Another researcher should be able to understand what the code does and see that it aligns with the claims of the paper. (3) Reusability: Another researcher should be able to adapt and build upon the code. Notably, reproducibility, readability, and reusability align with three of the code quality attributes that scientists care about when writing Jupyter notebooks ( Huang et al. 2025 ) , suggesting that these qualities matter to scientists throughout the process of writing code, and not just as community standards that final code artifacts should meet. Our primary motivation for this work is to uncover limitations in existing computational notebooks with respect to scientific work and identify opportunities for future programming systems that better support scientists’ needs. Therefore, we devised analyses to understand how specific aspects of Jupyter’s design may impact reproducibility, readability, and reusability. To contextualize our findings and identify unique characteristics of scientific code, we compared scientific notebooks to general-purpose software when feasible. To assess reproducibility, we manually ran 19 Jupyter notebooks, following the procedures a domain scientist trying to run the code would likely use: reading documentation, downloading datasets, and troubleshooting errors. Only two notebooks were fully reproducible, and 11 of the remaining 17 failed to run due to missing data files. Next, we analyzed our scientific notebooks along three dimensions, chosen for their unique interaction with Jupyter features and their impact on readability and reusability: documentation, code clones, and variable mutation. Documentation Jupyter notebooks are literate programming environments ( Granger and Pérez 2021 ; Knuth 1984 ) that allow end-users to interleave Markdown and code cells into a single computational narrative. The ability to communicate scientific results is part of what makes Jupyter popular among scientists ( Granger and Pérez 2021 ) . By categorizing the content of Markdown cells, we found that documentation was primarily used for section headers or to describe code behavior, but rarely added rich scientific or narrative content. Interpretation of results, discussion of reasoning, or added background knowledge each appeared in fewer than 10% of Markdown cells. We further observed that although some notebooks were extensively documented, 29% contained no Markdown, and scientific notebooks contained less documentation than general Github notebooks overall. Reader-facing demonstration notebooks were generally better documented than author-facing findings notebooks. Code clones Modularity and abstraction promote readability and reusability by enabling independent understanding and reuse of components ( Parnas 1972 ) . In contrast, Jupyter notebooks organize code into cells that share a single global scope and offer no abstraction, encouraging frequent copy-paste actions ( Huang et al. 2025 ) . Furthermore, functions cannot span multiple cells, which limits their explorability, since subdividing code allows users to inspect intermediate values and provides fine-grained control over organization and execution ( Rawn and Chasins 2025 ) . This limitation may discourage users from abstracting code into functions. Additionally, notebook code cannot

Read the complete text