CodeClash: Benchmarking Goal-Oriented Software Engineering

John Yang* 1, Kilian Lieret* 2, Joyce Yang 3, Carlos E. Jimenez 2, 2 Princeton University, 3 Cornell University · 2025

Read CodeClash: Benchmarking Goal-Oriented Software Engineering by John Yang* 1, Kilian Lieret* 2, Joyce Yang 3, Carlos E. Jimenez 2, 2 Princeton University, 3 Cornell University (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-00839.txt.

Introduction

John Yang and colleagues introduce CodeClash, a benchmark that tests language models the way real software work happens: chasing broad goals over many rounds, not fixing one named bug. Models compete in tournaments, editing their code in one phase and then facing rival codebases head to head in arenas scored on goals like resource gain or survival.

Across many tournaments with eight models in six arenas, the team finds wide differences in development style but shared weaknesses. Models reason poorly about long term strategy, and their codebases grow messy and redundant over time. The results suggest current models handle isolated coding tasks far better than the open ended, self directed upkeep that real projects demand.

Catalog record

CREATORJohn Yang* 1, Kilian Lieret* 2, Joyce Yang 3, Carlos E. Jimenez 2, 2 Princeton University, 3 Cornell University
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE168 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-00839

Browse the full library · About the Tower of Babel