Benchmarking the Rationality of AI Decision Making Using the Transitivity Axiom
Kiwon Song, James M. Jennings III, Clintin P. Davis-Stober · 2025
Read Benchmarking the Rationality of AI Decision Making Using the Transitivity Axiom by Kiwon Song, James M. Jennings III, Clintin P. Davis-Stober (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-10554.txt.
Description
Kiwon Song, James Jennings, and Clintin Davis-Stober test in 2025 whether AI models choose in a rational, consistent way. A basic rule of rational choice is transitivity: if A beats B and B beats C, then A should beat C. Human choices often bend that rule, and models trained on human data may inherit the bend.
The team ran choice tests, first designed for people, on ten versions of Meta's Llama models. A Bayesian method judged whether each model's answers broke either of two formal accounts of transitivity. Most models in fact held to the rule in a broad way. The paper sets out where the models passed and where they strained. Since AI advice now shapes human calls, the consistency of the adviser itself is worth grading.
Catalog record
| CREATOR | Kiwon Song, James M. Jennings III, Clintin P. Davis-Stober |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-10554