ContextBench : A Benchmark for Context Retrieval in Coding Agents
Han Li, Letian Zhu, Bohan Zhang, Rili Feng, Jiaming Wang, Yue Pan, Earl T. Barr, Federica Sarro, Zhaoyang Chu, He Ye · 2026
Read ContextBench : A Benchmark for Context Retrieval in Coding Agents by Han Li, Letian Zhu, Bohan Zhang, Rili Feng, Jiaming Wang, Yue Pan, Earl T. Barr, Federica Sarro, Zhaoyang Chu, He Ye (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-05892.txt.
Description
Coding agents are graded on whether they fix an issue, yet almost nothing is known about how they gather code context along the way. Han Li and 9 co-authors introduce ContextBench, a process oriented benchmark for context retrieval in coding agents. It holds over a thousand issue resolution tasks from dozens of repositories across eight programming languages, each with human annotated gold context.
An automated framework tracks agent trajectories and measures context recall, precision, and efficiency during resolution. Four frontier models and five coding agents were evaluated. Sophisticated agent scaffolding bought only marginal gains in retrieval quality, a result the authors call a bitter lesson. Models consistently favored recall over precision, and a wide gap separated the context agents explored from the context they actually used. The benchmark adds intermediate signals to end to end scores.
Catalog record
| CREATOR | Han Li, Letian Zhu, Bohan Zhang, Rili Feng, Jiaming Wang, Yue Pan, Earl T. Barr, Federica Sarro, Zhaoyang Chu, He Ye |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 81 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-05892