Benchmarking the Rationality of AI Decision Making Using the Transitivity Axiom

Kiwon Song, James M. Jennings III, Clintin P. Davis-Stober · 2025

Read Benchmarking the Rationality of AI Decision Making Using the Transitivity Axiom by Kiwon Song, James M. Jennings III, Clintin P. Davis-Stober (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-10554.txt.

Description

Kiwon Song, James Jennings, and Clintin Davis-Stober test in 2025 whether AI models choose in a rational, consistent way. A basic rule of rational choice is transitivity: if A beats B and B beats C, then A should beat C. Human choices often bend that rule, and models trained on human data may inherit the bend.

The team ran choice tests, first designed for people, on ten versions of Meta's Llama models. A Bayesian method judged whether each model's answers broke either of two formal accounts of transitivity. Most models in fact held to the rule in a broad way. The paper sets out where the models passed and where they strained. Since AI advice now shapes human calls, the consistency of the adviser itself is worth grading.

Catalog record

CREATORKiwon Song, James M. Jennings III, Clintin P. Davis-Stober
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-10554

Browse the full library · About the Tower of Babel