Are the Majority of Public Computational Notebooks Pathologically Non-Executable?

Tien Nguyen, Waris Gill, Muhammad Ali Gulzar · 2025

Read Are the Majority of Public Computational Notebooks Pathologically Non-Executable? by Tien Nguyen, Waris Gill, Muhammad Ali Gulzar (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-04184.txt.

Introduction

Tien Nguyen, Waris Gill, and Muhammad Ali Gulzar ask in two thousand twenty-five whether most public coding notebooks are truly beyond repair. Notebooks are the standard workspace for data science, but earlier studies claimed about three quarters of shared ones cannot run. The authors argue the usual test, a full error-free run, is too strict and brands many usable notebooks as dead.

They split broken notebooks into restorable and truly pathological ones, then fixed surface issues such as missing modules and absent data. In a set of over forty-two thousand popular notebooks, only about twenty-one percent were pathological. Language-model tools fully restored a slice of the rest, and partial fixes raised how many cells ran cleanly by large margins. The findings suggest public notebooks are healthier than assumed.

Catalog record

CREATORTien Nguyen, Waris Gill, Muhammad Ali Gulzar
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE56 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-04184

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Are the Majority of Public Computational Notebooks Pathologically Non-Executable? Tien Nguyen, Waris Gill, Muhammad Ali Gulzar Abstract Computational notebooks are the de facto platform for exploratory data science, offering an interactive programming environment where users can create, modify, and execute code cells in any sequence. However, this flexibility often introduces code quality issues, with prior studies showing that approximately 76% of public notebooks are non-executable, raising significant concerns about reusability. We argue that the traditional notion of executability—requiring a notebook to run fully and without error —is overly rigid, misclassifying many notebooks and overestimating their non-executability. This paper investigates pathological executability issues in public notebooks under varying notions and degrees of executability. Notebooks, by construction, are incrementally and interactively executed, where each cell execution advances logic toward the notebook’s goal. Even partially improving executability can improve code comprehension and offer a pathway for dynamic analyses. With this insight, we first categorize notebooks into potentially restorable and pathological non-executable notebooks and then measure how removing misconfiguration and superficial execution issues in notebooks can improve their executability (i.e., additional cells executed without error). For instance, we use a Large Language Model (LLM) to generate synthetic input data to restore non-executable notebooks with “FileNotFound” errors. In a dataset of 42,546 popular public notebooks, containing 34,659 non-executable notebooks, only 21.3% are truly pathologically non-executable. For restorable notebooks, LLM-based methods fully restore 5.4% of previously non-executable notebooks. Among the partially restored, it improves the notebooks’ executability by 40.5% and 28% by installing the correct modules and generating synthetic data. These findings challenge prior assumptions, suggesting that notebooks have higher executability than previously reported, many of which offer valuable partial execution, and that their executability should be evaluated within the interactive notebook paradigm rather than through traditional software executability standards. I Introduction Computational notebooks are the preferred tool for data analytics due to their portability and interactive capabilities. They enable easy sharing across various contexts and are often made publicly accessible, fostering reuse and adoption [ 1 , 2 ] . In notebooks, developers write code and explanatory text in cells that can be run individually, displaying results instantly. An unexpected consequence of this interactive programming is non-reproducibility—more than 75% of the public notebooks are previously reported to be non-executable, let alone reproduce the promised output [ 3 ] . Such a low level of executability is alarming as it diminishes the accessibility of public codebases and potentially causes code quality issues during reuse [ 4 ] . Previous investigations have studied the executability and reproducibility issue of notebooks and reasoned about the presence of such issues [ 3 , 5 , 6 , 7 , 8 , 9 ] . However, the findings of these studies have several limitations. Problem. Previous studies categorize notebooks into (1) non-executable if they do not execute end-to-end in their current form or (2) executable if they execute successfully without any interference. There are two limitations to this categorization. First, a seemingly non-executable notebook may technically be executable but lack the appropriate execution environments (input file, packages, or execution order). Second, the binary notion of executability is overly rigid and stringent. Notebooks, by construction, are executed incrementally and interactively, similar to REPL (read-eval-print loop [ 10 ] ), where each cell execution logically progresses the notebook. Even if a notebook is non-executable, its partial execution improves code comprehension during reuse. Furthermore, prior works report inconsistent findings on notebook non-executability (76% [ 3 ] , 72.6% [ 6 ] , 82.6% [ 9 ] , and 47% [ 7 ] ) due to differences in the notebooks datasets analyzed. For instance, despite the large number of notebooks studied by prior work [ 3 ] , most are unmanaged trial notebooks, often for personal, experimental projects rather than for reuse. Therefore, such findings do not generalize to popular notebooks (i.e., notebooks actively reused), demanding a fresh perspective on the non-executability of notebooks. We hypothesize that the count of executable notebooks is higher than previously reported, and even pathologically non-executable notebooks are partially executable, offering valuable code reusability, code comprehension, and code repair opportunities. Contributions. In this paper, we take a fresh perspective on notebook executability by investigating the reusability of public notebooks. We address the gaps in existing knowledge on notebook executability by making the following contributions. First, we introduce new notions and degrees of executability by separating misconfiguration from non-executability. To that end, we define pathologically non-executable notebooks that suffer from unresolvable fatal syntactic or runtime errors (such as bad indentation or attribute error), and restorable notebooks that are non-executable due to superficial misconfiguration errors (e.g., missing input, library, or execution order). Restorable notebooks are fully executable in a suitable environment (e.g., the original author’s local environment). Second, we introduce fine-grained metrics of executability that quantify the reuse potential of non-executable notebooks. For non-executable notebooks that are otherwise deemed unusable, we measure their partial executability based on the proportion of successfully executed cells, an aspect that has never been studied. Third, we conduct this investigation on actively reused and adopted notebooks. We follow the practice of using GitHub stars as a popularity metric [ 11 , 12 ] to rank notebooks during data collection. Doing so mitigates any bias from unmanaged, rarely used public notebooks. Lastly, we demonstrate the effectiveness of LLMs in addressing misconfiguration issues (e.g., by generating synthetic input data) in restorable notebooks. We realize the aforementioned contributions by building an automated measurement framework for notebooks. Each notebook is evaluated in a sandboxed environment with all required execution environments as stated in the parent repository. We leverage a combination of static and dynamic error checking to extract the causes of non-executability in notebooks. Static analysis on notebook code cells identifies errors such as structural, syntax, and variable undefined errors. Dynamic analysis identifies runtime errors such as missing libraries, packages, and type errors. This stage returns a categorized list of pathologically non-executable notebooks and restorable notebooks and their respective list of errors. LLMs are improving code generation and understanding [ 13 , 14 ] . To investigate the reusability of restorable notebooks, we utilize open-source LLM Llama-3 for error-driven notebook restoration. For example, error reports indicating missing modules or input files are combined with code cells to prompt the LLM to identify the correct module names for installation or generate synthetic input data, respectively. We iteratively restore and error-check the notebook until it is fully executable or an unfixable error is found. Study Results. We collect a dataset of the most popular 42,546 public notebooks from a stratified sample of 4,456 GitHub repositories with four or more stars in 2024. Using non-executability criteria from prior work, we found that over 81.5% of those notebooks are non-executable if executed as is, which is consistent with the findings of prior studies [ 5 , 6 ] . We investigate the top two reasons for non-executability: lack of appropriate input files and missing modules, packages, or libraries. Such notebooks are potentially restorable. Surprisingly, we find that only 21.3% of all notebooks are pathologically non-executable due to unrestorable executability issues, and the rest are either executable or potentially executable given a suitable execution environment, i.e., 78.7% of non-executable notebooks can potentially be restored. This

Read the complete text