A Third Paradigm for LLM Evaluation: Dialogue Game-Based Evaluation using clembench
David Schlangen Sherzod Hakimov Jonathan Jordan Philipp Sadler · 2025
Read A Third Paradigm for LLM Evaluation: Dialogue Game-Based Evaluation using clembench by David Schlangen Sherzod Hakimov Jonathan Jordan Philipp Sadler (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-08491.txt.
Description
Schlangen and co-authors present in 2025 a mature toolkit for a third way to test large language models: through dialogue games with set goals. The two reigning ways each miss something. Fixed tests with known answers are controlled but stiff, while open user arenas feel real but cannot be rerun or steered. Game based tests sit between them, offering repeatable multi turn play with clear aims and no stored answers.
Their bench, in steady growth since 2023, was rebuilt for easy use by outside teams. It ships with a set of games and the harness to run one's own models through them. The authors argue this middle path can catch skills that both older ways overlook, such as holding a goal across a long talk.
Catalog record
| CREATOR | David Schlangen Sherzod Hakimov Jonathan Jordan Philipp Sadler |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 22 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-08491