AgentLens : Revealing The Lucky Pass Problem in SWE-Agent Evaluation

Priyam Sahoo 1,2, , Gaurav Mittal 2, , Xiaomin Li 2, , Shengjie Ma 2 , Benjamin Steenhoek 2 , Pingping Lin 2 , Yu Hu 2 1 University of Illinois, Urbana-Champaign, IL, USA 2 Microsoft, Redmond, WA, USA [email protected] , [email protected] · 2026

Read AgentLens : Revealing The Lucky Pass Problem in SWE-Agent Evaluation by Priyam Sahoo 1,2, , Gaurav Mittal 2, , Xiaomin Li 2, , Shengjie Ma 2 , Benjamin Steenhoek 2 , Pingping Lin 2 , Yu Hu 2 1 University of Illinois, Urbana-Champaign, IL, USA 2 Microsoft, Redmond, WA, USA [email protected] , [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-12925.txt.

Description

Sahoo and six colleagues expose a flaw in how software-engineering agents are graded: a final patch that passes tests can hide a chaotic path to get there. Studying 2,614 OpenHands runs, they find 10.7% of passing runs show a Lucky Pass, with blind retries, missing checks or jumbled exploration and verification. Their AgentLens framework scores the process itself, splitting passes into Lucky, Solid and Ideal tiers, and reshuffles model rankings by up to five places.

Catalog record

CREATORPriyam Sahoo 1,2, , Gaurav Mittal 2, , Xiaomin Li 2, , Shengjie Ma 2 , Benjamin Steenhoek 2 , Pingping Lin 2 , Yu Hu 2 1 University of Illinois, Urbana-Champaign, IL, USA 2 Microsoft, Redmond, WA, USA [email protected] , [email protected]
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE87897
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.12925

Browse the full library · About the Tower of Babel