SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks
Gabriel Orlanski, Devjeet Roy, Alexander Yun, Changho Shin, Alex Gu, Albert Ge, Dyah Adila, Frederic Sala, Aws Albarghouthi · 2026
Read SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks by Gabriel Orlanski, Devjeet Roy, Alexander Yun, Changho Shin, Alex Gu, Albert Ge, Dyah Adila, Frederic Sala, Aws Albarghouthi (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-24755.txt.
Introduction
Orlanski and co-authors introduce SlopCodeBench, a benchmark for how coding agents degrade over long iterative tasks. Single-shot benchmarks hide design flaws, and earlier iterative ones over-constrain design choices. Their benchmark holds 36 problems and 196 checkpoints where agents repeatedly extend their own solutions. Two failure modes are measured: structural erosion and verbosity. Across 15 agents, none solves any problem end to end, and the best passes 14.8 percent of checkpoints. Agent code is over twice as verbose and eroded as code in 473 open-source repositories.
Catalog record
| CREATOR | Gabriel Orlanski, Devjeet Roy, Alexander Yun, Changho Shin, Alex Gu, Albert Ge, Dyah Adila, Frederic Sala, Aws Albarghouthi |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 64032 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.24755
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks Gabriel Orlanski, Devjeet Roy, Alexander Yun, Changho Shin, Alex Gu, Albert Ge, Dyah Adila, Frederic Sala, Aws Albarghouthi Abstract Software development is iterative, yet agentic coding benchmarks overwhelmingly evaluate single-shot solutions against complete specifications. Code can pass the test suite but become progressively harder to extend. Recent iterative benchmarks attempt to close this gap, but constrain the agent’s design decisions too tightly to faithfully measure how code quality shapes future extensions. We introduce SlopCodeBench, a language-agnostic benchmark comprising 20 problems and 93 checkpoints, in which agents repeatedly extend their own prior solutions under evolving specifications that force architectural decisions without prescribing internal structure. We track two trajectory-level quality signals: verbosity, the fraction of redundant or duplicated code, and structural erosion, the share of complexity mass concentrated in high-complexity functions. No agent solves any problem end-to-end across 11 models; the highest checkpoint solve rate is 17.2%. Quality degrades steadily: erosion rises in 80% of trajectories and verbosity in 89.8%. Against 48 open-source Python repositories, agent code is 2.2x more verbose and markedly more eroded. Tracking 20 of those repositories over time shows that human code stays flat, while agent code deteriorates with each iteration. A prompt-intervention study shows that initial quality can be improved, but it does not halt degradation. These results demonstrate that pass-rate benchmarks systematically undermeasure extension robustness, and that current agents lack the design discipline iterative software development demands. 1 Introduction Every design decision in software engineering is a compromise with unknown future requirements. A code search program built around regular expressions works until the specification demands structural pattern matching, at which point the entire architecture must be rewritten. Existing coding-agent benchmarks systematically undermeasure this failure mode, evaluating models once against complete task specifications ( Jimenez et al. 2024 ; Lu et al. 2026 ; Tran et al. 2026 ; Badertdinov et al. 2026 ) . They measure whether an agent can produce correct code for the current specification, not whether that code remains extensible under future change. Initial Spec: Basic Python code search Spec 2: Add JavaScript & C++ support Spec 3+: More Features Update Requirements Further Updates main.py rules.py main.py +100 rules.py +60 main.py +260 rules.py +150 Solution 1 Good Quality Solution 2 Degradation Begins Solution N Unmaintainable Figure 1 : Iterative evaluation in SlopCodeBench. At each checkpoint, the agent receives an updated specification and extends its own prior solution. File-level diffs grow across checkpoints, reflecting accumulated code changes. Under repeated editing, agent-generated code often deteriorates in recognizable ways. LLMs favor verbose constructions over concise idioms ( Dou et al. 2026 ; Abbassi et al. 2025 ) , and each multi-turn edit preserves and extends the anti-patterns of prior turns ( Chen and Jiang 2025 ; Nakashima et al. 2026 ; Watanabe et al. 2026 ) . The resulting low-quality, high-volume code is often colloquially called “slop.” In traditional software engineering, such accumulation is associated with higher maintenance cost and slower modification ( Lacerda et al. 2020 ; Le et al. 2021 ; Li et al. 2022 ) , yet pass rates can remain stable even as the underlying code becomes harder to extend. Pass-rate-centric single-shot benchmarks do not capture this. Recent benchmarks push toward multi-turn or long-horizon coding, but none of them isolate true iterative coding . Some construct iterative tasks by decomposing monolithic solutions into dependency-ordered subproblems, producing a self-contained test bed rather than a realistic setting where the agent selects an architecture and must live with it later ( Wang et al. 2025b ) . Others derive tasks from the commit histories of mature open-source repositories ( Thai et al. 2025 ; Deng et al. 2026 ; Chen et al. 2026a ) . These are valuable for studying maintenance and feature work in existing systems, but they do not test iterative coding ability. Using human-built workspaces and historically realized evolution paths means the agent never pays the cost of its own early design decisions. In some cases the task formulation is also tied to test- or oracle-derived signals, which further undercuts the benchmark’s ability to measure open-ended iterative design ( Chen et al. 2026a ) . To properly measure this, a benchmark needs four things: the agent builds on its own prior code; problems specify only external behavior, not internal interfaces; the test suite stays hidden so it can’t leak architectural hints; and each task is a black-box contract that’s implementable in any language. We therefore introduce SlopCodeBench (SCBench) , a benchmark for measuring how code quality evolves as agents repeatedly extend their own prior code under changing specifications. SCBench contains 20 problems spanning 93 checkpoints. Each checkpoint specifies only observable behavior at a CLI or API boundary, leaving internal structure unconstrained and keeping the test suite hidden. The benchmark is language-agnostic by construction; in this paper we focus on the Python track. Beyond correctness, we track two trajectory-level quality signals: verbosity , which measures redundant or duplicated code growth, and structural erosion , which measures the concentration of complexity in already-complex functions. Our contributions are: 1. SlopCodeBench , 1 1 1 Benchmark, data, and code available at https://www.scbench.ai a language-agnostic benchmark of 20 iterative software-development problems spanning 93 checkpoints. No evaluated agent solves a problem end-to-end; the highest checkpoint solve rate is 17.2%. 2. Two trajectory-level quality metrics , verbosity and structural erosion, that separate redundant code growth from concentration of complexity mass. Erosion rises in 80% of trajectories and verbosity in 89.8%. 3. Calibration against human code. Agent code is 2.2x more verbose and more eroded than 20 maintained repositories, and the gap widens every iteration. 4. Prompt intervention study. Quality-aware prompts reduce initial verbosity and erosion but do not slow the degradation, improve pass rates, or reduce cost. 2 SlopCodeBench SlopCodeBench contains 20 language-agnostic problems spanning 93 checkpoints. Each problem is specified only through observable behavior at a CLI or API boundary, so it can be evaluated in any implementation language. The experiments in this paper report the Python implementation track. An agent implements the first specification from scratch, then repeatedly modifies and extends its own prior code as specifications evolve. The benchmark measures correctness and tracks code quality across that trajectory. We use the code_search problem as a running example throughout this section because its checkpoints apply escalating architectural pressure to early design decisions. The agent builds a CLI tool for semantic source-file search, inspired by ast-grep , across five checkpoints: C_{1} - Python-only exact and regex matching. 2 2 2 Here “Python-only” refers to the source files being searched at C_{1} , not the language used to implement the solution. Establishes the core CLI contract and rule format. C_{2} - Multi-language support (JavaScript, C++). C_{3} - AST-based pattern matching with metavariable capture. C_{4} - Selector rules and auto-fix functionality. C_{5} - Add support for Go, Rust, and Java. An agent that hardcodes language-specific logic at C_{1} faces cascading rewrites at C_{2} and C_{5} ; one that builds an extensible parser interface does not. These structural choices at the first checkpoint determine whether slop accumulates or stays contained. The full problem specifcations can be found in Appendix D . 2.1 Design Principles The core goal of SlopCodeBench is to