Why Stop at One Error? Benchmarking LLMs as Data Science Code Debuggers for Multi-Hop and Multi-Bug Errors

Zhiyu Yang, Shuo Wang, Yukun Yan, Yang Deng · 2025

Read Why Stop at One Error? Benchmarking LLMs as Data Science Code Debuggers for Multi-Hop and Multi-Bug Errors by Zhiyu Yang, Shuo Wang, Yukun Yan, Yang Deng (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-22388.txt.

Introduction

Zhiyu Yang and co-authors benchmark language models as debuggers of data science code in a 2025 paper. Models write code well, but finding and fixing logic errors at runtime is another skill. Most repair tests use simple code with a single bug. Real data science scripts fail in chains: one error triggers the message, while several bugs hide underneath.

Their benchmark, DSDBench, builds over a thousand annotated tasks from existing data science sets. Auto-generated snippets carry multiple linked bugs, with cause-effect pairs and runtime messages recorded. Top current models left large performance gaps on the set, struggling to trace multi-step error chains and to catch every bug. The authors release the benchmark openly, aiming to make AI help for data work more reliable.

Catalog record

CREATORZhiyu Yang, Shuo Wang, Yukun Yan, Yang Deng
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE52 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-22388

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Why Stop at One Error? Benchmarking LLMs as Data Science Code Debuggers for Multi-Hop and Multi-Bug Errors Zhiyu Yang, Shuo Wang, Yukun Yan, Yang Deng Abstract LLMs are transforming software development, yet most code benchmarks still emphasize syntactic or functional correctness in simple, single-error cases. These settings miss the core difficulty of real-world data science debugging, where runtime errors propagate across multiple lines (multi-hop) and often appear in sets (multi-bug). We introduce DSDBench : D ata S cience D ebugging Bench mark, the first benchmark to systematically evaluate LLMs on this challenge. Unlike general debugging benchmark suites such as SWE-bench, DSDBench targets non-expert, data-centric scripting, where practitioners rely heavily on black-box libraries and write exploratory code that is error-prone and difficult to debug. Evaluations of state-of-the-art LLMs reveal large performance gaps: even frontier models that excel at code generation fail to reliably trace and resolve these errors, exposing a critical “generation versus understanding” gap. DSDBench provides a resource to drive progress toward more robust and trustworthy AI-assisted data science. 1 1 1 DSDBench is publicly available at https://github.com/KevinCL16/DSDBench . 1 Introduction Recent advancements in Large Language Models (LLMs) have significantly reshaped software development practices, particularly in automating code generation and debugging. Benchmarks like DebugBench ( Tian et al., 2024 ) , CodeEditorBench ( Guo et al., 2024a ) , and DebugEval ( Yang et al., 2025 ) have played a pivotal role in evaluating LLMs’ capabilities in code repair. However, these benchmarks largely rely on simplified programming exercises from platforms like LeetCode , which prioritize syntactic correctness and functional accuracy in isolated and single-error scenarios, far removed from real-world software complexity. Meanwhile, growing research efforts are exploring LLMs’ potential in data science coding Yang et al. (2024) ; Hu et al. (2024) ; Zhang et al. (2024b) ; Hong et al. (2024) , where practitioners routinely tackle challenges involving black-box library functions, intricate data transformations, and statistical modeling. Yet, a critical gap persists: despite this emerging focus, there remains a striking lack of investigation into LLMs’ ability to debug dynamic logical errors in data science code . Such errors, manifesting only at runtime, are endemic to this domain due to hidden dependencies in data pipelines, implicit assumptions in mathematical operations, and unpredictable interactions with external resources. Figure 1: Dataset construction pipeline of DSDBench. As illustrated in Figure 1 , unlike constrained programming exercises, debugging data science codebases presents unique challenges: 1) Its heavy reliance on external libraries (e.g., pandas, NumPy, scikit-learn, matplotlib) means subtle misuses or incorrect data processing steps can easily trigger downstream runtime exceptions. 2) Data scientists often work in interactive environments like Jupyter Notebooks, which lack robust debugging tools. This makes it harder to identify and fix runtime bugs, especially when multiple subtle errors , such as incorrect data transformations or misaligned indices, coexist and interact within the code, complicating the debugging process. 3) Standard debugging tools offer limited assistance in diagnosing multi-hop logical errors within complex workflows. The root cause of these errors can be distantly located from the point of error manifestation. Standard debuggers typically report the symptom (the line of error manifestation in the stack trace) rather than the root cause responsible for the program’s termination. Linking these two is essential not only for reasoning, but also for trustworthy assistance grounded in what users actually see. Overall, a dedicated benchmark for rigorously assessing LLMs’ dynamic debugging of multi-hop logical errors in complex multi-bug data science code is still lacking. Unlike general debugging suites such as SWE-bench that evaluate patch-level fixes in professional OSS repositories via unit tests, our setting targets non-expert, data-centric scripting where failures are often data-dependent and surface as interpreter-visible runtime errors. Crucially, we evaluate whether models can connect root causes to these interpreter-visible failure points ( effect lines ), a prerequisite for trustworthy explanations. Motivated by this evident gap in evaluating LLMs’ dynamic debugging skills for data science, we introduce DSDBench : the D ata S cience D ebugging Bench mark. Distinct from prior works that primarily focus on repairing single, syntactic and static errors, DSDBench is the first benchmark to systematically evaluate LLMs on: (1) Multi-Hop Error Tracing : requiring models to trace runtime errors back through multiple lines of data science code to identify the root cause; and (2) Multi-Bug Error Detection : assessing their ability to concurrently detect and reason about multiple logical errors within a single data science code snippet. We focus on interpreter-visible (crashing) bugs to enable scalable, deterministic annotation of cause/effect lines and messages. For clarity, we define a “hop” whenever the cause and effect lines differ, and for multi-bug items a prediction is counted correct only if all constituent bugs are identified. Table 1 summarizes the comparisons between DSDBench and existing code debugging benchmarks. Benchmark Domain Error Multi-Hop Error Complexity Error Type DebugBench General Multi-Bug ✗ Static DebugEval General Multi-Bug ✗ Static CodeEditorBench General Single-Bug ✗ Static DSDBench Data Science Multi-Bug ✓ Runtime Table 1: Comparison with existing benchmarks. DSDBench leverages datasets and tasks from established data science coding benchmarks like DABench ( Hu et al., 2024 ) , MatPlotBench ( Yang et al., 2024 ) , and DSEval ( Zhang et al., 2024b ) . We systematically inject errors into data science code, synthesizing multi-error scenarios by combining individual bugs. Our dataset comprises 1,117 meticulously annotated samples, complete with ground-truth cause-effect error line pairs and captured runtime error messages. In summary, our contributions are threefold: • DSDBench Benchmark: We release the first dedicated benchmark and dataset for evaluating LLMs in runtime, multi-bug debugging of data science code. DSDBench features realistic logical errors, multi-hop error scenarios, and detailed annotations, addressing a critical gap in current debugging benchmarks. • Automated Error Injection and Annotation Framework: We develop a robust pipeline for automated error injection, runtime execution tracing, and alignment of interpreter outputs with error-originating code lines, facilitating scalable benchmark creation and future expansion. • Empirical Analysis and Insights: We present a comprehensive empirical evaluation of state-of-the-art closed-source and open-source LLMs on DSDBench. Our findings reveal significant performance gaps and highlight critical challenges in dynamic debugging for complex, real-world data science code. 2 DSDBench Construction The creation of a high-quality dataset is paramount for a robust benchmark. As illustrated in Figure 1 , DSDBench is meticulously constructed through a multi-stage process encompassing data sourcing, correct code preparation, error injection, error annotation, and quality assurance. Dataset Size Example Type Multi-Error Examples Code Complexity Question Complexity Total # Examples # Single-Error # Multi-Error # Single-hop # Multi-hop Avg Errors/Example Avg Code Length Avg Question Length 1,117 741 376 385 356 2.87 \pm 1.14 65.31 \pm 21.31 92.42 \pm 55.86 Table 2: Dataset statistics of DSDBench. 2.1 Data Collection We build the DSDBench upon three widely-adopted data science coding benchmarks for their realistic data science tasks and diverse scenarios, including DABench Hu et al. (2024) , MatPlotBench Yang et al. (2024) , and DSEval Zhang et al. (2024b) . We focus on the hard subset of DABench because error injection in its easy and medium subsets rarely produces runtime exceptions. MatPlotBench and DSEval supplement DABench, expanding task diversity and library coverage ( pandas , sklearn , scipy , matplotlib , numpy ) to represent typical data science workflows. These

Read the complete text