ContextBench : A Benchmark for Context Retrieval in Coding Agents

Han Li, Letian Zhu, Bohan Zhang, Rili Feng, Jiaming Wang, Yue Pan, Earl T. Barr, Federica Sarro, Zhaoyang Chu, He Ye · 2026

Read ContextBench : A Benchmark for Context Retrieval in Coding Agents by Han Li, Letian Zhu, Bohan Zhang, Rili Feng, Jiaming Wang, Yue Pan, Earl T. Barr, Federica Sarro, Zhaoyang Chu, He Ye (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-05892.txt.

Description

Coding agents are graded on whether they fix an issue, yet almost nothing is known about how they gather code context along the way. Han Li and 9 co-authors introduce ContextBench, a process oriented benchmark for context retrieval in coding agents. It holds over a thousand issue resolution tasks from dozens of repositories across eight programming languages, each with human annotated gold context.

An automated framework tracks agent trajectories and measures context recall, precision, and efficiency during resolution. Four frontier models and five coding agents were evaluated. Sophisticated agent scaffolding bought only marginal gains in retrieval quality, a result the authors call a bitter lesson. Models consistently favored recall over precision, and a wide gap separated the context agents explored from the context they actually used. The benchmark adds intermediate signals to end to end scores.

Catalog record

CREATORHan Li, Letian Zhu, Bohan Zhang, Rili Feng, Jiaming Wang, Yue Pan, Earl T. Barr, Federica Sarro, Zhaoyang Chu, He Ye
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE81 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-05892

Browse the full library · About the Tower of Babel