JunoBench: A Benchmark Dataset of Crashes in Python Machine Learning Jupyter Notebooks CCS: Software and its engineering Software testing and debugging
Yiran Wang, José Antonio Hernández López, Ulf Nilsson, Dániel Varró · 2025
Read JunoBench: A Benchmark Dataset of Crashes in Python Machine Learning Jupyter Notebooks CCS: Software and its engineering Software testing and debugging by Yiran Wang, José Antonio Hernández López, Ulf Nilsson, Dániel Varró (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-18013.txt.
Introduction
Yiran Wang and three co-authors collect real crashes from machine learning notebooks into the first benchmark of its kind. Notebooks are the main sketchpad for ML work, yet few debugging tools target them, partly for lack of test data. JunoBench gathers over a hundred curated crashes from public Kaggle notebooks, each with a verified fix.
The set spans major libraries and includes notebook-only faults like running cells out of order. Every crash reproduces in one shared environment, with labels and plain-language notes on each fault. Debugging researchers gain realistic material for building tools that find, explain, and repair notebook errors.
Catalog record
| CREATOR | Yiran Wang, José Antonio Hernández López, Ulf Nilsson, Dániel Varró |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 23 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-18013
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
JunoBench: A Benchmark Dataset of Crashes in Python Machine Learning Jupyter Notebooks CCS: Software and its engineering Software testing and debugging Yiran Wang, José Antonio Hernández López, Ulf Nilsson, Dániel Varró Abstract . Jupyter notebooks are widely used for machine learning (ML) prototyping. Yet few debugging tools are designed for ML code in notebooks, potentially due to the lack of benchmarks. We introduce JunoBench, the first benchmark dataset of real-world crashes in Python-based ML notebooks. JunoBench has 111 curated and reproducible crashes from public Kaggle notebooks, each paired with a verifiable fix, ranging over popular ML libraries, including TensorFlow/Keras, PyTorch, Scikit-learn, Pandas, and NumPy, as well as notebook-specific out-of-order execution issue. To support reproducibility and ease of use, JunoBench offers a unified execution environment where all crashes can be reliably reproduced. By providing realistic crashes and their resolutions, JunoBench facilitates bug detection, localization, diagnosis, and repair tailored to the interactive and iterative nature of notebook-based ML development. 1. Introduction Machine learning (ML) and deep learning (DL) are increasingly used in modern software programs, with Jupyter notebooks emerging as a dominant development environment for ML prototyping ( NASA high-end computing capability 2025 ; Grotov et al. 2022 ) . Notebooks offer a flexible, interactive interface that supports incremental coding, intermediate feedback, and out-of-order execution. These features make notebooks well suited for exploratory ML development, but also introduce unique debugging challenges ( Grotov et al. 2022 ) . Even worse, the use of complex ML libraries, such as TensorFlow and PyTorch, make ML prototyping extra error prone ( Islam et al. 2019 ) . Existing tools offer limited support for identifying ML bugs, especially in the notebook environment ( De Santana et al. 2024 ) . One critical missing component is benchmarks. Benchmarks are crucial for advancing automatic debugging research by providing standardized datasets of real-world bugs for reproducible evaluation ( v. Kistowski et al. 2015 ) . While prior work ( Morovati et al. 2023 ; Liang 2022 ; Kim et al. 2021 ; Zhang et al. 2018 ; Wardat et al. 2021 ) focuses on ML bugs in Python scripts ( .py files), they do not account for the unique semantics of Jupyter notebooks. Furthermore, existing ML bug benchmarks ( e.g., ( Morovati et al. 2023 ) ) mix bug symptoms such as bad performance, incorrect functionality, and crashes. Crashes which cause code execution to terminate, are particularly valuable for benchmarking in the context of notebooks, as they are clearly observable from error outputs in .ipynb files. To fill this gap, we present JunoBench , the first benchmark of crashes in real-world Python ML notebooks. Based on our previous empirical study ( Wang et al. 2025c ) , JunoBench supports the evaluation of debugging tools for notebook-based ML development by providing: • 111 curated, reproducible real-world crashes from Python ML notebooks, each paired with a verifiable fix; • Broad, balanced coverage of major ML libraries ( e.g., TensorFlow/Keras, PyTorch, Scikit-learn, Pandas, NumPy, Matplotlib, Seaborn) and Notebook-specific failures (out-of-order execution); • Ground truth crash diagnosis labels for developing and evaluating crash identification tools. • A unified execution environment for consistent reproduction and evaluation. JunoBench is available on Hugging Face ( Wang et al. 2025a ) , and all artifacts used to construct the benchmark are provided on GitHub ( Wang et al. 2025b ) . 2. Methodology This section outlines the methodology (see Figure 1 ), including data collection, benchmark construction, and how we ensure reproducibility and usability of the benchmark. A plot showing the benchmark construction process. Figure 1 . Overview of the benchmark construction process. A plot showing the benchmark construction process. 2.1. Data collection Data sources The crashing notebooks in JunoBench were drawn from the notebook dataset in our previous empirical study of public Python ML notebook crashes ( Wang et al. 2025c ) where we studied 64,031 crashing notebooks and manually labeled 746 crashes with detailed annotations, including crash type , root cause , and the ML library used at the point of failure (referred to as the library cause ). The original notebook pool comprises notebooks collected from GitHub and Kaggle. This paper focuses specifically on Kaggle notebooks (2,869 notebooks containing 3,875 crashes, of which 356 are labeled) due to their greater support for reproducibility. For instance, Kaggle notebooks typically reference input datasets that are publicly accessible. Crash filtering We began with the 356 labeled crashes, and selected crashes that 1) occurred when using ML libraries (based on the library cause label), or 2) were due to notebook-specific out-of-order execution (the root cause label). Then we excluded crashes whose root causes were not reproducible or fixable within the notebook context, such as 1) insufficient resource (hardware-dependent), 2) incompatible library versions (requiring additional complex environment setups), 3) bugs inside third-party libraries. Finally, 174 candidate crashes remained for the benchmark construction process. Resampling The filtered labeled crashes are unevenly distributed across ML libraries. To balance coverage, we resampled 500 additional crashes from the unlabeled Kaggle pool, selecting notebooks using at least one of the ML libraries used in the labeled set ( i.e., TensorFlow/Keras, PyTorch, Scikit-learn, Pandas, NumPy, Matplotlib, Seaborn, Statsmodels, Torchvision, LightGBM, and OpenCV). Due to the non-linear execution of notebooks, it is often difficult to trace the origin of variables in a target code cell and identify the actual ML libraries involved without manual inspection. Therefore, we implemented a tool to automatically estimate the library cause for each crash. The tool assigns scores to ML libraries based on their presence in the crashing code cell and its error traceback, giving higher weight to those appearing in the crashing line. The tool achieved an accuracy of 80% when evaluated against labeled data. It was used as a guide to prioritize which crashes to manually reproduce and fix first for benchmark inclusion. 2.2. Benchmark inclusion Input dataset retrieval To reproduce each crash, we first retrieved the necessary input datasets used by the notebooks. We used the KGTorrent ( Quaranta et al. 2021 ) database and the Meta Kaggle ( Risdal and Bozsolik 2022 ) dataset to obtain dataset titles, downloading links, and licenses. We then attempted to download the datasets using the downloading links. Since we focus on crashes instead of performance issues, the use of realistic or full training data is not relevant.Hence, for datasets over 100MB, we checked if downsampling preserved the crash behavior without altering code logic. For instance, a 2.5GB image dataset was reduced to 94MB by retaining just 4% of images per class while still reproducing the crash. Manual reproducing and fixing In our previous study, the dataset was organized by crashes, with each entry representing a distinct crash rather than a notebook. Consequently, multiple crashes could originate from the same notebook. In constructing this benchmark, we adopted a different design in which each instance is represented as an independent notebook containing a single, isolated, reproducible crash. This ensures that each failure scenario is self-contained, simplifying evaluation and supporting consistent comparisons of automated debugging methods. For each notebook, we keep the original notebook and create two modified versions. One is a reproduced version in which we apply only the minimal necessary code changes ( e.g., paths or training settings). The other is a