From Code to Play: Benchmarking Program Search for Games Using Large Language Models

Manuel Eberhardinger, James Goodman, Alexander Dockhorn, Diego Perez-Liebana, Raluca D. Gaina, Duygu Çakmak, Setareh Maghsudi, Simon Lucas · 2024

Read From Code to Play: Benchmarking Program Search for Games Using Large Language Models by Manuel Eberhardinger, James Goodman, Alexander Dockhorn, Diego Perez-Liebana, Raluca D. Gaina, Duygu Çakmak, Setareh Maghsudi, Simon Lucas (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-04057.txt.

Description

Manuel Eberhardinger and seven co-authors benchmark language models that write game code in a 2024 paper. Models can now synthesize programs, and games offer a lively test bed for usable code. The question is whether model-written programs actually play well, not just run.

An evolutionary hill-climbing loop mutates model-written code and keeps the winners. Tasks span small Atari-style games, puzzle levels, a maze builder, and tabletop games. Twenty-nine tasks cover Python and Java tracks. Twelve models were judged on one track and eight on the other. Task type mattered more than model size for final quality. Bigger models produced more runnable code, but runnable did not always mean better play.

Catalog record

CREATORManuel Eberhardinger, James Goodman, Alexander Dockhorn, Diego Perez-Liebana, Raluca D. Gaina, Duygu Çakmak, Setareh Maghsudi, Simon Lucas
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE62 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-04057

Browse the full library · About the Tower of Babel