Large-scale Evaluation of Notebook Checkpointing with AI Agents
Hanxi Fang, Supawit Chockchowwat, Hari Sundaram, Yongjoo Park · 2025
Read Large-scale Evaluation of Notebook Checkpointing with AI Agents by Hanxi Fang, Supawit Chockchowwat, Hari Sundaram, Yongjoo Park (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-01377.txt.
Introduction
Hanxi Fang, Supawit Chockchowwat, Hari Sundaram, and Yongjoo Park report a 2025 study of saving work steps in data notebooks. Analysts often save mid-way results so they can go back or branch off. Past studies of that saving habit used small groups of human users. The team instead used AI agents to act out many data tasks. That let them run far more cases than a human study could.
The test covered more than a thousand paths of work and close to three thousand blocks of code. Saving mid-way results did boost how much the agents got done. It cut wasted reruns of code and cut clutter from spare copies of data and code. The authors say the agent method can mimic real users well. That opens a cheaper way to test tools built for data work at large scale.
Catalog record
| CREATOR | Hanxi Fang, Supawit Chockchowwat, Hari Sundaram, Yongjoo Park |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 39 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-01377
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Large-scale Evaluation of Notebook Checkpointing with AI Agents Hanxi Fang, Supawit Chockchowwat, Hari Sundaram, Yongjoo Park Abstract . Saving, or checkpointing, intermediate results during interactive data exploration can potentially boost user productivity. However, existing studies on this topic are limited, as they primarily rely on small-scale experiments with human participants—a fundamental constraint of human subject studies. To address this limitation, we employ AI agents to simulate a large number of complex data exploration scenarios, including revisiting past states and branching into new exploration paths. This strategy enables us to accurately assess the impact of checkpointing while closely mimicking the behavior of real-world data practitioners. Our evaluation results, involving more than 1,000 exploration paths and 2,848 executed code blocks, show that a checkpointing framework for computational notebooks can indeed enhance productivity by minimizing unnecessary code re-executions and redundant variables/code. Conference: CHI Conference on Human Factors in Computing Systems; April 26-May 1, 2025; Yokohama, Japan CHI Conference on Human Factors in Computing Systems (CHI ’25), April 26-May 1, 2025, Yokohama, Japan DOI: 10.1145/3706598.3714141 ISBN: 979-8-4007-1394-1/25/04 Conference: Extended Abstracts of the CHI Conference on Human Factors in Computing Systems; April 26-May 1, 2025; Yokohama, Japan Extended Abstracts of the CHI Conference on Human Factors in Computing Systems (CHI EA ’25), April 26-May 1, 2025, Yokohama, Japan DOI: 10.1145/3706599.3719928 ISBN: 979-8-4007-1395-8/2025/04 CCS: Human-centered computing Usability testing CCS: Human-centered computing User studies CCS: Human-centered computing Interactive systems and tools CCS: Information systems Version management † † cc-license: by \myfigure 1. Introduction Checkpointing computational notebooks can improve user productivity ( Fang et al., 2025 ) . Specifically, notebook systems like Jupyter ( Jupyter, 2023 ) , Colab ( Google, [n.d.] ) , R Markdown ( Baumer and Udwin, 2015 ) let data scientists run a code block one at a time for analyzing tabular data, training machine learning models, visualizing results, etc. By checkpointing intermediate code/data, users can undo undesirable executions, explore alternative hypotheses, and restore from crashes. A recent work ( Fang et al., 2025 ) proposes a checkpointing interface for computational notebooks and demonstrates its productivity benefits. Unfortunately, the existing evaluation is limited to employing a handful of human subjects working on a fixed set of tasks. While those participants are allowed to freely explore, the study remains constrained by the number of participants, the variety of tasks, and the duration of the study. We recognize that these limitations are inherent to human subject studies: the limitations are difficult to overcome without adopting an entirely different approach. In this work, we tackle them by devising an AI agent-based strategy. Our observation is that AI agents can generate high-quality code akin to that produced by real data scientists, and exhibit iterative refinement behaviors that closely resemble the testing, debugging, and code improvement processes typically employed by human practitioners. Specifically, we employ a pre-trained AI agent (i.e., ChatGPT 4o) to simulate real-world data scientists who explore data with and without a notebook checkpointing tool. For each of the scenarios, the only independent variable is the assistance offered by the checkpointing tool; however, the actual variables residing in sessions and also elapsed times vary significantly. This “late-breaking work”, as a sequel to the recent research paper ( Fang et al., 2025 ) , expands the current state-of-the-art in two significant aspects. First, we systematically evaluate the effectiveness of the new code+data space checkpointing framework with hundreds of data exploration scenarios. This work is the first that evaluates data science checkpointing frameworks ( Li et al., 2023 ; Li et al., 2024 ; Manne et al., 2022 ; Wang et al., 2022 ) in such a large scale. Second, we reason why our agent-based approach is valid, ensuring two types of consistencies: consistency of generated code and consistency of branching strategy. 2. Background We overview notebook checkpointing frameworks ( Section 2.1 ) and discuss the limitations of the existing evaluation ( Section 2.2 ). Figure 1 . Background about Kishuboard. The history graph (purple box) shows past commits. The code and variable panes (yellow box) display the information of a selected commit. From any past commit, users can load data only (i.e., execution rollback) or load both code and data (i.e., checkout) using the navigation popup (red box). The image is reproduced with the original authors’ permission. This figure shows Kishuboard User Interface \parThe interface includes: \parSearch Bar: Filters commits based on keywords. History Graph: Displays past commits with code and data states. Navigation Popup: Allows switching between code+data states (rollback or checkout). Code View & Variable Panel: Shows details of the selected commit without changing the active Jupyter session. This interface helps users manage and explore different versions of computational notebooks effectively. 2.1. Notebook Checkpointing Computational notebooks (e.g., Jupyter, Colab) are designed around a linear, cell-by-cell execution model, even though real-world data science exploration is often iterative and non-linear. Practitioners commonly revisit older states, branch the code to try different alternatives, and switch among branches. ( Rule et al., 2018 ; Kery et al., 2019 ) To accommodate branching, they either start a new notebook for each path ( Chattopadhyay et al., 2020 ) or mix branch-specific code cells in a single notebook ( Liu et al., 2019 ; Subramanian et al., 2019 ; Kery et al., 2018 ) . Both approaches are time-consuming (re-running shared cells) and cognitively taxing (tracking which cells belong to each branch). Inspired by code version control systems ( Loeliger and McCullough, 2012 ) , Kishuboard ( Fang et al., 2025 ) addresses these issues by introducing two-dimensional code+data checkpointing for computational notebooks (see Fig. 1 for its user interface). Unlike one-dimensional version control tools that track only code revisions, Kishuboard also version-controls the evolving “variable state in the kernel” or “data state” produced by each cell’s execution. Based on that, users can checkout the code and data state of notebook to any previous point and start exploring a new branch from that point. This feature, together with Kishuboard’s UI that visualizes the branched structure of exploration similar to Git GUI ( Git SCM, 2024 ) , enables fast, nonlinear data science exploration and reduces user’s cognitive burden of managing multiple exploration branches simultaneously. 2.2. Existing Evaluation of Notebook Checkpointing and its Limitations The existing evaluation is based on human subject studies ( Fang et al., 2025 ) . Specifically, to evaluate whether notebook checkpointing improves data science productivity, a user study was conducted with 20 student participants randomly split into experimental (with Kishuboard to checkout states) and control (without Kishuboard) groups. Both groups performed the same notebook-based tasks—building models, branching workflows, reporting metrics, retrieving previous variable states, debugging, and recovering from system crashes. While data practitioners may write their own code, participants in this study chose from a set of pre-written code snippets (with the option to modify them), ensuring that notebook checkpointing was the primary independent variable. The study measured each participant’s task completion time and gathered feedback through surveys, then compared performance between the two groups. Evaluation using traditional user study has several limitations: (1) As it relies on a small group of participants, its statistical power is limited. (2) The short study duration restricts participants’ ability to engage in extended exploration. Even if they are allowed to branch