SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering?

Samuel Miserendino, Michele Wang, Tejal Patwardhan, Johannes Heidecke · 2025

Read SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering? by Samuel Miserendino, Michele Wang, Tejal Patwardhan, Johannes Heidecke (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-12115.txt.

Description

Samuel Miserendino, Michele Wang, Tejal Patwardhan, and Johannes Heidecke built a bench of real freelance coding jobs in a 2025 paper. The jobs come from the Upwork platform, more than fourteen hundred in all, worth one million dollars in real pay. They ask whether top language models could earn that money.

Jobs split into solo coding tasks, from small bug fixes to large features, and manager tasks, where the model picks between plans. Solo work is graded by full tests checked three times by seasoned engineers, and manager picks are set against the real hires' choices. Top models still failed most tasks. The team shared a public test split and a ready test setup to spur work on the cash value of AI skill.

Catalog record

CREATORSamuel Miserendino, Michele Wang, Tejal Patwardhan, Johannes Heidecke
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE82 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-12115

Browse the full library · About the Tower of Babel