From Code to Play: Benchmarking Program Search for Games Using Large Language Models
Manuel Eberhardinger, James Goodman, Alexander Dockhorn, Diego Perez-Liebana, Raluca D. Gaina, Duygu Çakmak, Setareh Maghsudi, Simon Lucas · 2024
Read From Code to Play: Benchmarking Program Search for Games Using Large Language Models by Manuel Eberhardinger, James Goodman, Alexander Dockhorn, Diego Perez-Liebana, Raluca D. Gaina, Duygu Çakmak, Setareh Maghsudi, Simon Lucas (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-04057.txt.
Description
Manuel Eberhardinger and seven co-authors benchmark language models that write game code in a 2024 paper. Models can now synthesize programs, and games offer a lively test bed for usable code. The question is whether model-written programs actually play well, not just run.
An evolutionary hill-climbing loop mutates model-written code and keeps the winners. Tasks span small Atari-style games, puzzle levels, a maze builder, and tabletop games. Twenty-nine tasks cover Python and Java tracks. Twelve models were judged on one track and eight on the other. Task type mattered more than model size for final quality. Bigger models produced more runnable code, but runnable did not always mean better play.
Catalog record
| CREATOR | Manuel Eberhardinger, James Goodman, Alexander Dockhorn, Diego Perez-Liebana, Raluca D. Gaina, Duygu Çakmak, Setareh Maghsudi, Simon Lucas |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 62 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-04057