Outrunning LLM Cutoffs: A Live Kernel Crash Resolution Benchmark for All
Chenxi Huang*, Alex Mathai*, Feiyang Yu, Aleksandr Nogikh, Petros Maniatis, Franjo Ivančić, Eugene Wu, Kostis Kaffes, Junfeng Yang, Baishakhi Ray · 2026
Read Outrunning LLM Cutoffs: A Live Kernel Crash Resolution Benchmark for All by Chenxi Huang*, Alex Mathai*, Feiyang Yu, Aleksandr Nogikh, Petros Maniatis, Franjo Ivančić, Eugene Wu, Kostis Kaffes, Junfeng Yang, Baishakhi Ray (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-02690.txt.
Introduction
Chenxi Huang and 9 co-authors build a living benchmark for AI agents that fix Linux kernel crashes, in a 2026 paper. Kernel fuzzers surface crashes constantly, and language-model agents have begun attempting repairs. Static benchmarks age badly against an evolving kernel and risk contamination, since models may already know the fixes from training.
Live-kBench continuously scrapes freshly discovered bugs and evaluates agents only on new material. Its companion environment standardizes compilation, execution, and feedback, decoupling agent logic from heavy infrastructure so frameworks compare fairly. The inaugural dataset holds five hundred thirty-four bugs. Agents scored notably higher on bugs fixed before their knowledge cutoffs, quantifying the contamination gap. Three leading agents resolved most crashes with plausible patches, yet only about a fifth closely matched developer fixes. Feeding back crash results improved resolution substantially.
Catalog record
| CREATOR | Chenxi Huang*, Alex Mathai*, Feiyang Yu, Aleksandr Nogikh, Petros Maniatis, Franjo Ivančić, Eugene Wu, Kostis Kaffes, Junfeng Yang, Baishakhi Ray |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 41 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-02690
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Outrunning LLM Cutoffs: A Live Kernel Crash Resolution Benchmark for All Chenxi Huang*, Alex Mathai*, Feiyang Yu, Aleksandr Nogikh, Petros Maniatis, Franjo Ivančić, Eugene Wu, Kostis Kaffes, Junfeng Yang, Baishakhi Ray Abstract Repairing system crashes discovered by kernel fuzzers like Syzkaller is a critical yet underexplored challenge in software engineering. While recent works have introduced Large Language Model (LLM) based agents for Linux kernel crash-resolution, their evaluation benchmarks are usually static and thus, do not capture the evolving nature of the Linux kernel, and suffer from potential data contamination due to LLM knowledge cutoffs. To address the above problem, we present (i) Live-kBench , an evaluation framework for self-evolving benchmarks that continuously scrapes and evaluates agents on freshly discovered kernel bugs, and (ii) kEnv , an agent-agnostic standardized crash-resolution environment for kernel compilation, execution, and feedback. This design decouples agent workflows from heavy-weight execution, enabling fair and scalable comparison across diverse agent frameworks under identical conditions. To this end, we curate an inaugural dataset of 534 534 Linux kernel bugs and empirically demonstrate a significant performance gap, with agents achieving up to 25% higher equivalent patch rate on bugs fixed before the LLM knowledge cutoff. Using kEnv , we benchmark three state-of-the-art agents, showing that they resolve 74 % 74\% of crashes on the first attempt (plausible patches); however only ∼ 20 % \sim 20\% of generated patches closely match developer fixes. Additionally, exposing crash resolution feedback improves crash resolution rate by 29 % 29\% . Live-kBench provides the community with an evaluation infrastructure for self-evolving benchmarks that is both time and attribute sensitive ; complete with a public dashboard to track agent progress on Linux kernel bugs. † † affiliationnotice: Equal contribution 1 Introduction In recent years, software-engineering agents have made rapid progress in user-space development, where automated bug repair is increasingly achievable in modern programming environments ( 2 ; 9 ; 6 ; 8 ; 1 ) . This progress, however, does not readily extend to system-level failures. When the Linux kernel—the foundation of much of modern computing infrastructure—crashes, debugging and repair are substantially more challenging due to limited observability, strict execution constraints, the sheer size of the codebase, and high correctness and safety requirements. At the same time, kernel fuzzers have become highly effective at uncovering such failures. Tools such as Syzkaller ( 18 ) use coverage-guided fuzzing to explore diverse kernel execution paths and have reported over 6 K kernel crashes, many of which expose high direct or indirect risk ( 26 ) and are assigned CVE identifiers ( 4 ) . However, the automated and large-scale nature of fuzzing introduces fundamental obstacles for repair and evaluation. Fuzzer-discovered bugs typically come with machine-generated reports and a single proof-of-concept input, leaving vulnerabilities underspecified and allowing many candidate patches that suppress the crash while breaking functionality ( 13 ) . These challenges are further compounded by the scale of the Linux kernel—over 20 M lines of code—which complicates code localization within LLM context limits, and by the high cost of evaluation, as compiling and testing a single kernel patch can take at least 30 minutes. Limitations of Existing Work. Recent efforts have taken initial steps toward kernel crash resolution. 14 introduced kBenchSyz , a small-scale benchmark for kernel crash repair, and subsequent work proposed LLM-based agents for fixing kernel crashes ( 13 ; 16 ) . In parallel, rapid advances have been made in general-purpose automated program repair agents in user-space settings ( 21 ; 23 ; 19 ) . Despite their close conceptual relationship, these lines of work have largely evolved in isolation: modern repair agents have not been systematically evaluated on kernel crash-resolution tasks, and existing studies rely on older, static datasets that are vulnerable to data contamination and do not reflect the constantly evolving Linux kernel. As a result, prior work provides limited insight into the effectiveness of state-of-the-art agents for real-world kernel crash resolution. To this end, we identify two major limitations of existing work. (i) Lack of a kernel-scale, agent-agnostic computer-use environment. Existing benchmarks such as SWE-Bench and kBenchSyz ( 11 ; 14 ) standardize scoring but leave the execution environment unspecified, forcing agent developers to construct their own computer-use environments. Thus, each agent defines its own benchmark-specific logic to set up the environment, which involves preparing the correct codebase and artifacts, supporting local testing, and enforcing safety constraints. While this tightly coupled design is workable for user-space, interpreted languages with low execution overhead, it does not scale to Linux kernel crash resolution. Preparing the kernel codebase is costly, testing requires privileged virtualization, and compiling and executing modified kernels is slow and resource-intensive ( 14 ) . As a result, evaluation cannot be performed directly within an agent’s local workflow and must rely on external execution infrastructure. We therefore argue for a principled decoupling of agent workflows from heavy-weight computer-use environments: an agent-agnostic environment layer that automates setup and offloads kernel compilation and testing to a scalable execution backend. This separation enables fair comparison across agents by allowing workflows and tool usage to vary, while environment preparation, execution, and feedback are handled uniformly. (ii) Distribution shift and data contamination. Although recent work has proposed kernel-specific approaches ( 14 ; 13 ; 16 ) , evaluations rely primarily on older, static datasets (e.g., kBenchSyz ), with fixes dating back to 2018. The Linux kernel evolves rapidly through refactoring, subsystem changes, and shifting execution contexts, leading to persistent distribution shift. At the same time, older fixes increasingly appear in LLM training data, which risks data contamination. As a result, static benchmarks may overestimate generalization, underscoring the need for time-aware evaluation. Our Approach. To address these challenges, we develop an agent-agnostic crash-resolution environment and a live evaluation framework for Linux kernel crash resolution. (I) kEnv standardizes computer-use environment preparation and execution for crash resolution by initializing the Linux codebase at the correct commit, enforcing execution guardrails, and exposing a single run_kernel interface that submits compilation and execution jobs to kGymSuite , a scalable kernel execution platform ( 13 ) . Thus, by decoupling agent workflows from heavy-weight execution, kEnv enables scalable, reproducible, and fair comparisons across agentic frameworks. (ii) Live-kBench builds on this execution layer by continuously sourcing fresh kernel bugs from public dashboards and mailing lists, filtering for reproducibility, running registered agents via kEnv , and evaluating resulting patches along multiple dimensions—including crash resolution, localization accuracy, and similarity to human fixes—before reporting results on a public leaderboard. Results. Using our systems, we curate Live-kBench -2512: a fresh dataset with the latest Linux bugs. We then use Live-kBench to measure the difference in performance on bugs before and after LLM knowledge cutoffs. In our experiments, we show that agents exhibit up to 25\% better equivalent patch rate on older data (before the cutoff) compared to fresh data (after the cutoff). Additionally, we benchmark three open-source agents by providing each agent access to kEnv and then running comprehensive evaluations. Our results confirm that, at a first glance, current agents resolve 74\% of kernel crashes in the first try (plausible patches), with only \sim 20\% of the patches being exactly equivalent to the kernel developer’s patch. We also show that the crash resolution rate (CRR) improves by 25\%