A Third Paradigm for LLM Evaluation: Dialogue Game-Based Evaluation using clembench

David Schlangen Sherzod Hakimov Jonathan Jordan Philipp Sadler · 2025

Read A Third Paradigm for LLM Evaluation: Dialogue Game-Based Evaluation using clembench by David Schlangen Sherzod Hakimov Jonathan Jordan Philipp Sadler (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-08491.txt.

Description

Schlangen and co-authors present in 2025 a mature toolkit for a third way to test large language models: through dialogue games with set goals. The two reigning ways each miss something. Fixed tests with known answers are controlled but stiff, while open user arenas feel real but cannot be rerun or steered. Game based tests sit between them, offering repeatable multi turn play with clear aims and no stored answers.

Their bench, in steady growth since 2023, was rebuilt for easy use by outside teams. It ships with a set of games and the harness to run one's own models through them. The authors argue this middle path can catch skills that both older ways overlook, such as holding a goal across a long talk.

Catalog record

CREATORDavid Schlangen Sherzod Hakimov Jonathan Jordan Philipp Sadler
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE22 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-08491

Browse the full library · About the Tower of Babel