Towards a Neural Debugger for Python

Maximilian Beck 1,∗, Jonas Gehring 2, Jannik Kossen 2, Gabriel Synnaeve 2 · 2026

Read Towards a Neural Debugger for Python by Maximilian Beck 1,∗, Jonas Gehring 2, Jannik Kossen 2, Gabriel Synnaeve 2 (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-09951.txt.

Introduction

Maximilian Beck and coauthors train language models to act as interactive debuggers for Python code. Models trained on execution traces can predict a program line by line, yet real developers jump to breakpoints and inspect variables. Their neural debuggers support stepping into, over and out of functions, plus setting breakpoints. The models predict both future states and, in reverse, earlier states, scoring well on CruxEval.

Catalog record

CREATORMaximilian Beck 1,∗, Jonas Gehring 2, Jannik Kossen 2, Gabriel Synnaeve 2
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE60667
LICENSEhttp://creativecommons.org/licenses/by-sa/4.0/
RIGHTSCC BY SA
SOURCEarXiv

Source: https://arxiv.org/abs/2603.09951

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Towards a Neural Debugger for Python Maximilian Beck 1,∗, Jonas Gehring 2, Jannik Kossen 2, Gabriel Synnaeve 2 Abstract Training large language models (LLMs) on Python execution traces grounds them in code execution and enables the line-by-line execution prediction of whole Python programs, effectively turning them into neural interpreters ( 8 ) . However, developers rarely execute programs step by step; instead, they use debuggers to stop execution at certain breakpoints and step through relevant portions only while inspecting or modifying program variables. Existing neural interpreter approaches lack such interactive control. To address this limitation, we introduce neural debuggers : language models that emulate traditional debuggers, supporting operations such as stepping into, over, or out of functions, as well as setting breakpoints at specific source lines. We show that neural debuggers—obtained via fine-tuning large LLMs or pre-training smaller models from scratch—can reliably model both forward execution (predicting future states and outputs) and inverse execution (inferring prior states or inputs) conditioned on debugger actions. Evaluated on CruxEval, our models achieve strong performance on both output and input prediction tasks, demonstrating robust conditional execution modeling. Our work takes first steps towards future agentic coding systems in which neural debuggers serve as a world model for simulated debugging environments, providing execution feedback or enabling agents to interact with real debugging tools. This capability lays the foundation for more powerful code generation, program understanding, and automated debugging. † † affiliation: 1 Johannes Kepler University Linz, Institute for Machine Learning † † affiliation: 2 Meta FAIR CodeGen Team † † contribution: ∗ Work done during an internship at Meta FAIR † † date: March 2026 1 Introduction Debugging is the process of isolating and correcting mistakes in computer programs ( 14 ) . It is a fundamental task in software engineering that is often considered separate from writing programs ( 31 ) . A debugger is a collection of software tools to aid debugging ( 14 ) that allows developers to inspect and control program execution through actions such as stepping into or over function calls, setting breakpoints, or returning from functions. By observing how program states evolve line by line, developers can localize faults, understand control flow, and reason about program correctness. Recent advances in large language models (LLMs) trained on massive code corpora have shown remarkable capabilities in code generation, completion, and repair ( 5 ; 18 ; 13 ; 11 ; 27 ; 23 ) . In the coding domain, LLMs have evolved from supportive programmer tools—such as code completion—to agents writing complete codebases autonomously and are increasingly used to assist developers in debugging or fixing software bugs ( 12 ) . However, open-source and academic models primarily reason over static code and are not explicitly grounded in program execution. Therefore, several approaches expose language models to program execution data. One line of work incorporates feedback such as test results ( 9 ) , error messages ( 34 ) , or runtime/shell outputs ( 32 ; 30 ) into reinforcement learning ( 25 ) or iterative generation loops ( 6 ; 19 ) . Another complementary direction uses execution traces , i.e., records of variable states and control flow transitions, to directly teach models the semantics of code execution ( 20 ; 16 ; 2 ; 8 ) in order to improve code generation and reasoning tasks such as verification, testing, and repair. Specifically, these approaches train language models on complete execution traces, which enable the line-by-line prediction of whole programs, effectively turning these models into neural interpreters . However, this overlooks how developers actually use debuggers to interact with programs and resolve bugs. Instead of executing programs strictly sequentially, they pause execution at certain breakpoints and step only through relevant parts while inspecting or modifying program variables. Previous works fall short of modeling this interactive, non-sequential debugging behavior. To address this shortcoming, we introduce neural debuggers : neural networks that can serve as simulated debugging environments for Python programs. Specifically, given the program code as context, neural debuggers are language models trained to predict the line-by-line execution of a computer program, conditioned on typical debugger actions such as step_into , step_over , breakpoint , or step_return . Unlike conventional debuggers, neural debuggers can simulate program execution even for non-executable or partially specified programs, making them applicable to debugging, testing, or synthesis scenarios where access to full execution environments is unavailable or restricted ( 35 ) . Moreover, traditional debuggers require re-executing the program after code or state modifications, leading to slow iteration cycles. In contrast, neural debuggers enable efficient reinitialization of both program state and execution context through prompting and regeneration. Beyond standard forward execution, our neural debuggers also support approximate inverse execution : given an arbitrary program state, they can infer or sample plausible preceding program states or inputs. In agentic coding systems, neural debuggers can function as learned world models of debugging environments. They can simulate execution feedback and external interactions—such as file reads, API requests, or operating system calls—or allow the agent to interact with real debugging environments. Incorporating neural debuggers in this manner can augment existing coding systems with debugging and planning abilities, facilitating more effective code generation, comprehension, and repair. In this paper, we take the first steps towards this vision by introducing neural debuggers as a Markov Decision Process (MDP), where each state represents the current program location and variable values, and transitions correspond to debugger actions that traverse a call-stack tree reconstructed from execution traces. We introduce a data pipeline (Figure 1 ) that supports both forward and inverse execution prediction of Python programs. The pipeline takes execution traces as input, samples debugger trajectories, and tokenizes them into a structured format compatible with standard off-the-shelf language models. We finetune 32\text{\,}\mathrm{B} -parameter and pre-train 1.8\text{\,}\mathrm{B} -parameter LLMs on this data to obtain neural debuggers that accurately predict program execution conditioned on debugger actions. Our 32\text{\,}\mathrm{B} parameter neural debugger consistently achieves forward next state prediction accuracies beyond 90\text{\,}\mathrm{\%} across key actions: step into, step over, step return, and breakpoint. In addition, neural debuggers exhibit strong transfer to CruxEval output and input prediction tasks ( 10 ) , demonstrating their enhanced code understanding and interpretation capabilities. In particular, our 1.8\text{\,}\mathrm{B} -parameter neural debugger LLM, trained from scratch on 150\text{\,}\mathrm{B} tokens, attains CruxEval input and output pass@1 scores of 53.6\text{\,} and 57.7\text{\,} , respectively, while our finetuned 32\text{\,}\mathrm{B} -parameter neural debugger LLM reaches scores of 66.5\text{\,} and 83.2\text{\,} , respectively. Figure 1 : Neural Debugger Data Pipeline. Our pipeline prepares training data for neural debuggers by transforming stack-frame sequences recorded via sys.settrace in three steps: (1) we construct a state tree (Section 4.1 ) from frame events; (2) we sample trajectories by traversing the state tree using a data-generating action policy; and (3) we tokenize each trajectory using our formal neural debugger language grammar (Section 4.2 ). In summary, in this work we make the following contributions: • We introduce neural debuggers: language models that can predict forward and inverse execution of Python programs conditioned on source code and debugger actions. • We describe a data pipeline for building neural debugger models by starting from pre-trained LLMs via finetuning or integration into pre- or mid-training data mixes. •

Read the complete text