SWE Atlas: Benchmarking Coding Agents Beyond Issue Resolution
Mohit Raghavendra 1, Soham Dan 1,∗, Miguel Romero Calvo 1,∗, Yannis Yiming He 1, Johannes Baptist Mols 1, Gautam Anand 1, Cole McCollum 1, Edgar Arakelyan 1, Vijay Bharadwaj 1, Andrew Park 1, Jeff Da 1, MohammadHossein Rezaei 1, Bing Liu 1, Brad Kenstler 1, Yunzhong He 1 · 2026
Read SWE Atlas: Benchmarking Coding Agents Beyond Issue Resolution by Mohit Raghavendra 1, Soham Dan 1,∗, Miguel Romero Calvo 1,∗, Yannis Yiming He 1, Johannes Baptist Mols 1, Gautam Anand 1, Cole McCollum 1, Edgar Arakelyan 1, Vijay Bharadwaj 1, Andrew Park 1, Jeff Da 1, MohammadHossein Rezaei 1, Bing Liu 1, Brad Kenstler 1, Yunzhong He 1 (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-08366.txt.
Description
Mohit Raghavendra, Soham Dan, Miguel Romero Calvo, Yannis Yiming He, and coauthors broaden how coding agents are graded. Most benchmarks reward fixing a reported issue, while professional work also means answering questions about a codebase, writing tests, and refactoring. SWE Atlas covers those three workflows with two hundred eighty-four tasks in total.
Its tasks are deliberately under-specified, the way real requests arrive, and scoring mixes programmatic checks with rubric judgments of completeness, maintainability, reusable abstraction, and codebase hygiene. Frontier closed models led the field in the authors' runs, while the best open-weight models scored poorly, exposing how much agent evaluation has been missing.
Catalog record
| CREATOR | Mohit Raghavendra 1, Soham Dan 1,∗, Miguel Romero Calvo 1,∗, Yannis Yiming He 1, Johannes Baptist Mols 1, Gautam Anand 1, Cole McCollum 1, Edgar Arakelyan 1, Vijay Bharadwaj 1, Andrew Park 1, Jeff Da 1, MohammadHossein Rezaei 1, Bing Liu 1, Brad Kenstler 1, Yunzhong He 1 |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 75 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.08366