SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering?
Samuel Miserendino, Michele Wang, Tejal Patwardhan, Johannes Heidecke · 2025
Read SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering? by Samuel Miserendino, Michele Wang, Tejal Patwardhan, Johannes Heidecke (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-12115.txt.
Description
Samuel Miserendino, Michele Wang, Tejal Patwardhan, and Johannes Heidecke built a bench of real freelance coding jobs in a 2025 paper. The jobs come from the Upwork platform, more than fourteen hundred in all, worth one million dollars in real pay. They ask whether top language models could earn that money.
Jobs split into solo coding tasks, from small bug fixes to large features, and manager tasks, where the model picks between plans. Solo work is graded by full tests checked three times by seasoned engineers, and manager picks are set against the real hires' choices. Top models still failed most tasks. The team shared a public test split and a ready test setup to spur work on the cash value of AI skill.
Catalog record
| CREATOR | Samuel Miserendino, Michele Wang, Tejal Patwardhan, Johannes Heidecke |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 82 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-12115