SWE-Interact : Reimagining SWE Benchmarks as User-Driven Long-Horizon Coding Sessions
Mohit Raghavendra, Anisha Gunjal, Aakash Sabharwal, Yunzhong He · 2026
Read SWE-Interact : Reimagining SWE Benchmarks as User-Driven Long-Horizon Coding Sessions by Mohit Raghavendra, Anisha Gunjal, Aakash Sabharwal, Yunzhong He (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-30573.txt.
Description
Mohit Raghavendra and three colleagues argue that coding-agent benchmarks miss how developers really work, and their 2026 response is a new testbed. Existing software-engineering tests hand agents complete requirements up front and grade solo implementation. Real users, the authors note, start vague, reveal needs gradually, inspect the workspace and add constraints along the way. SWE-Interact recreates that loop with a carefully designed user simulator.
Across frontier and open-weight models, skill at single-turn tasks transferred poorly to these interactive sessions. The best models solved about half of the single-turn baseline tasks but only about a quarter of the matching SWE-Interact tasks. Top models coped better with vagueness, kept going until every requirement surfaced, and wrote cleaner code. Even they overreached at times, forgot requirements and made technical mistakes. Weaker models stalled early, ignored instructions and reworked code more often.
Catalog record
| CREATOR | Mohit Raghavendra, Anisha Gunjal, Aakash Sabharwal, Yunzhong He |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 59 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.30573