Measure what Matters: Psychometric Evaluation of AI with Situational Judgment Tests

Alexandra Yost, Shreyans Jain, Shivam Raval, Grant Corser, Allen G. Roush, Nina Xu, Jacqueline Hammack, Ravid Shwartz - Ziv, Amirali Abdullah · 2025

Read Measure what Matters: Psychometric Evaluation of AI with Situational Judgment Tests by Alexandra Yost, Shreyans Jain, Shivam Raval, Grant Corser, Allen G. Roush, Nina Xu, Jacqueline Hammack, Ravid Shwartz - Ziv, Amirali Abdullah (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-22170.txt.

Introduction

Alexandra Yost leads a nine-author team testing whether a persona changes model behavior in a stable way. Giving a language model a persona is common practice. The change might be deep, or only surface flavor. The team borrows the tools of psychometrics, the science of testing minds.

Structured synthetic personas were run through situational judgment tests. These are workplace-style dilemmas with no single right answer. A multidimensional item response model extracted hidden traits behind the answers. Persona-conditioned behavior proved stable across repeated runs. Trait scores predicted performance on outside benchmarks. Truthfulness and emotion tests were among them. Human checks confirmed internal consistency. The authors are careful about meaning. The scores describe steady behavioral tendencies. They are not a human-like personality inside the machine.

Catalog record

CREATORAlexandra Yost, Shreyans Jain, Shivam Raval, Grant Corser, Allen G. Roush, Nina Xu, Jacqueline Hammack, Ravid Shwartz - Ziv, Amirali Abdullah
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE144 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-22170

Browse the full library · About the Tower of Babel