Measure what Matters: Psychometric Evaluation of AI with Situational Judgment Tests
Alexandra Yost, Shreyans Jain, Shivam Raval, Grant Corser, Allen G. Roush, Nina Xu, Jacqueline Hammack, Ravid Shwartz - Ziv, Amirali Abdullah · 2025
Read Measure what Matters: Psychometric Evaluation of AI with Situational Judgment Tests by Alexandra Yost, Shreyans Jain, Shivam Raval, Grant Corser, Allen G. Roush, Nina Xu, Jacqueline Hammack, Ravid Shwartz - Ziv, Amirali Abdullah (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-22170.txt.
Introduction
Alexandra Yost leads a nine-author team testing whether a persona changes model behavior in a stable way. Giving a language model a persona is common practice. The change might be deep, or only surface flavor. The team borrows the tools of psychometrics, the science of testing minds.
Structured synthetic personas were run through situational judgment tests. These are workplace-style dilemmas with no single right answer. A multidimensional item response model extracted hidden traits behind the answers. Persona-conditioned behavior proved stable across repeated runs. Trait scores predicted performance on outside benchmarks. Truthfulness and emotion tests were among them. Human checks confirmed internal consistency. The authors are careful about meaning. The scores describe steady behavioral tendencies. They are not a human-like personality inside the machine.
Catalog record
| CREATOR | Alexandra Yost, Shreyans Jain, Shivam Raval, Grant Corser, Allen G. Roush, Nina Xu, Jacqueline Hammack, Ravid Shwartz - Ziv, Amirali Abdullah |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 144 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-22170