SWE Atlas: Benchmarking Coding Agents Beyond Issue Resolution

Mohit Raghavendra 1, Soham Dan 1,∗, Miguel Romero Calvo 1,∗, Yannis Yiming He 1, Johannes Baptist Mols 1, Gautam Anand 1, Cole McCollum 1, Edgar Arakelyan 1, Vijay Bharadwaj 1, Andrew Park 1, Jeff Da 1, MohammadHossein Rezaei 1, Bing Liu 1, Brad Kenstler 1, Yunzhong He 1 · 2026

Read SWE Atlas: Benchmarking Coding Agents Beyond Issue Resolution by Mohit Raghavendra 1, Soham Dan 1,∗, Miguel Romero Calvo 1,∗, Yannis Yiming He 1, Johannes Baptist Mols 1, Gautam Anand 1, Cole McCollum 1, Edgar Arakelyan 1, Vijay Bharadwaj 1, Andrew Park 1, Jeff Da 1, MohammadHossein Rezaei 1, Bing Liu 1, Brad Kenstler 1, Yunzhong He 1 (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-08366.txt.

Description

Mohit Raghavendra, Soham Dan, Miguel Romero Calvo, Yannis Yiming He, and coauthors broaden how coding agents are graded. Most benchmarks reward fixing a reported issue, while professional work also means answering questions about a codebase, writing tests, and refactoring. SWE Atlas covers those three workflows with two hundred eighty-four tasks in total.

Its tasks are deliberately under-specified, the way real requests arrive, and scoring mixes programmatic checks with rubric judgments of completeness, maintainability, reusable abstraction, and codebase hygiene. Frontier closed models led the field in the authors' runs, while the best open-weight models scored poorly, exposing how much agent evaluation has been missing.

Catalog record

CREATORMohit Raghavendra 1, Soham Dan 1,∗, Miguel Romero Calvo 1,∗, Yannis Yiming He 1, Johannes Baptist Mols 1, Gautam Anand 1, Cole McCollum 1, Edgar Arakelyan 1, Vijay Bharadwaj 1, Andrew Park 1, Jeff Da 1, MohammadHossein Rezaei 1, Bing Liu 1, Brad Kenstler 1, Yunzhong He 1
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE75 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.08366

Browse the full library · About the Tower of Babel