Democratizing Diplomacy: A Harness for Evaluating Any Large Language Model on Full-Press Diplomacy

Alexander Duffy, Samuel J Paech, Ishana Shastri, Elizabeth Karpinski, Baptiste Alloui-Cros, Tyler Marques, Matthew Lyle Olson · 2025

Read Democratizing Diplomacy: A Harness for Evaluating Any Large Language Model on Full-Press Diplomacy by Alexander Duffy, Samuel J Paech, Ishana Shastri, Elizabeth Karpinski, Baptiste Alloui-Cros, Tyler Marques, Matthew Lyle Olson (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-07485.txt.

Description

Alexander Duffy and co-authors let ordinary language models play full Diplomacy in a 2025 paper. The board game demands negotiation, persuasion, and long plans. Its state is dense and matches vary wildly. Prior play needed frontier models or special training.

Their harness needs neither fine-tuning nor giant models. A tuned text representation carries the game state. Even a mid-size local model finishes matches reliably. Tooling supports hypothesis testing and statistical analysis. Case studies probe persuasion and aggressive playstyles. Many popular models were run through the harness. Larger models played best, as expected. Smaller ones still played adequately for research use. Critical state analysis zooms into pivotal game moments. Strategic reasoning can now be studied cheaply. Emergent negotiation skills become measurable. The code accompanies the paper for replication.

Catalog record

CREATORAlexander Duffy, Samuel J Paech, Ishana Shastri, Elizabeth Karpinski, Baptiste Alloui-Cros, Tyler Marques, Matthew Lyle Olson
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE59 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-07485

Browse the full library · About the Tower of Babel