SWE-Interact : Reimagining SWE Benchmarks as User-Driven Long-Horizon Coding Sessions

Mohit Raghavendra, Anisha Gunjal, Aakash Sabharwal, Yunzhong He · 2026

Read SWE-Interact : Reimagining SWE Benchmarks as User-Driven Long-Horizon Coding Sessions by Mohit Raghavendra, Anisha Gunjal, Aakash Sabharwal, Yunzhong He (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-30573.txt.

Description

Mohit Raghavendra and three colleagues argue that coding-agent benchmarks miss how developers really work, and their 2026 response is a new testbed. Existing software-engineering tests hand agents complete requirements up front and grade solo implementation. Real users, the authors note, start vague, reveal needs gradually, inspect the workspace and add constraints along the way. SWE-Interact recreates that loop with a carefully designed user simulator.

Across frontier and open-weight models, skill at single-turn tasks transferred poorly to these interactive sessions. The best models solved about half of the single-turn baseline tasks but only about a quarter of the matching SWE-Interact tasks. Top models coped better with vagueness, kept going until every requirement surfaced, and wrote cleaner code. Even they overreached at times, forgot requirements and made technical mistakes. Weaker models stalled early, ignored instructions and reworked code more often.

Catalog record

CREATORMohit Raghavendra, Anisha Gunjal, Aakash Sabharwal, Yunzhong He
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE59 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.30573

Browse the full library · About the Tower of Babel