Applying Psychometrics to Large Language Model Simulated Populations: Recreating the HEXACO Personality Inventory Experiment with Generative Agents

Sarah Mercer, Daniel P. Martin, Phil Swatton · 2025

Read Applying Psychometrics to Large Language Model Simulated Populations: Recreating the HEXACO Personality Inventory Experiment with Generative Agents by Sarah Mercer, Daniel P. Martin, Phil Swatton (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-00742.txt.

Introduction

Sarah Mercer, Daniel Martin, and Phil Swatton tested whether AI agents playing scripted personas can stand in for human subjects in a classic personality study, in a paper from 2025. They surveyed more than three hundred agents powered by a leading language model, using the same six trait inventory as the original human work.

A coherent trait structure could be recovered from the agents' answers, partly matching the human framework. Results were stable within the one model when the agent population was carefully built. Other models, though, gave different trait profiles, which hints at model specific bias. The authors discuss when such simulated samples may help social science and where they can mislead.

Catalog record

CREATORSarah Mercer, Daniel P. Martin, Phil Swatton
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE73 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-00742

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Applying Psychometrics to Large Language Model Simulated Populations: Recreating the HEXACO Personality Inventory Experiment with Generative Agents Sarah Mercer, Daniel P. Martin, Phil Swatton Abstract Generative agents powered by Large Language Models demonstrate human-like characteristics through sophisticated natural language interactions. Their ability to assume roles and personalities based on predefined character biographies has positioned them as cost-effective substitutes for human participants in social science research. This paper explores the validity of such persona-based agents in representing human populations; we recreate the HEXACO personality inventory experiment by surveying 310 GPT-4 powered agents, conducting factor analysis on their responses, and comparing these results to the original findings presented by Ashton, Lee, & Goldberg in 2004. Our results found 1) a coherent and reliable personality structure was recoverable from the agents’ responses demonstrating partial alignment to the HEXACO framework. 2) the derived personality dimensions were consistent and reliable within GPT-4, when coupled with a sufficiently curated population, and 3) cross-model analysis revealed variability in personality profiling, suggesting model-specific biases and limitations. We discuss the practical considerations and challenges encountered during the experiment. This study contributes to the ongoing discourse on the potential benefits and limitations of using generative agents in social science research and provides useful guidance on designing consistent and representative agent personas to maximise coverage and representation of human personality traits. Keywords: Generative AI, Large Language Models, Generative Agents, Machine Psychology, Psychometrics, Lexical Analysis, Persona Prompting, Trait Consistency. 1 Introduction One of the primary goals of Artificial Intelligence is to develop systems that can replicate human behaviours and interactions. The field of Natural Language Processing (NLP) plays a crucial role in enabling AI systems to understand and generate human language. Recent advancements in Generative AI, specifically Large Language Models (LLMs), have enhanced these capabilities, raising significant interest in such systems’ ability to mimic human cognitive, emotional, and social behaviours and resulted in an increased use of generative agents for social simulations [ 1 , 2 , 3 , 4 , 5 , 6 ] , within computational social science [ 7 , 8 ] and the emerging field of machine psychology [ 9 , 10 , 11 ] . In parallel with the developments in social simulations, a separate body of research examines the extent to which LLMs can be used to simulate human participants in experiments, opinion polls, and surveys in social scientific research [ 12 , 13 , 14 , 15 ] . There are several perceived benefits, such as reduced costs, collecting hard-to-acquire data, or simply reduced administrative burden. What makes generative agents appropriate for both types of simulation is the data upon which the encompassed LLMs have been trained, the huge amount of human generated text, in the form of books, website articles and social media content, which inherently describes a diverse set of human expressions, interactions, and nuances of language use commonly exposed by humans. As such, generative agents can mimic (to varying degrees, and not without some contention [ 16 , 17 ] ) linguistic styles, knowledge, and cognitive processes present in the training data, which can appear to be humanlike. However, as this mimicry is fundamentally computational (next word prediction) it raises the question of how successfully these agents emulate human behaviour (i.e. how valid are they beyond surface level responses). 1.1 Related Work To quantify the inherent qualities and attributes of LLMs and generative agents, AI researchers have drawn on the field of psychology. Several studies have explored the psychological aspects of LLMs, investigating whether the cognitive and reasoning abilities these models appear to display align closely with those observed in humans. Notably, Hagendorff et al ., used several behaviour tests; the Cognitive Reflection Test (CRT) [ 18 ] and semantic illusions [ 19 ] designed to investigate intuitive decision-making in humans (e.g. fast ‘v’ slow thinking), concluding that ‘investigating LLMs with methods from psychology has the potential to reveal otherwise unknown emergent traits’ [ 20 ] . Macmillan-Scott and Musolesi [ 21 ] used “cognitive illusion” tests (brain teasers) originally designed by Kahneman and Tversky [ 22 ] to illustrate cognitive biases and heuristics in human reasoning, to evaluate the rationality of LLMs. They noted that given the fact that responses varied for the same prompt and model, a slightly different approach to evaluating LLMs would be required. They also observed that the models’ incorrect responses were ‘incorrect in ways different to human subjects’. In addition to cognitive testing, psychometric tests have also become popular for measuring the personalities demonstrated by Generative AI. In 2022, Miotto et al. [ 23 ] employed HEXACO-60 [ 24 ] on OpenAI’s Davinci, noting that its results were similar to human samples when provided with a history of previous responses. There have been many similar studies since then, the results of which have varied due to differences in choice of psychometric test, prompting strategy, and degree of academic rigour. Even as the approaches mature, researchers are still reporting mixed results. Safdari et al. [ 25 ] used two measures; IPIP-NEO [ 26 ] and BFI [ 27 ] , and a principled methodology to explore how a model’s size and training procedures affected simulated personalities traits, finding that fine-tuning increased the stability of personality structures. Gupta et al. [ 28 ] argued against that conclusion, suggesting that validity tests must also be rooted in LLM understanding and not human-based best practice. Using IPIP-300 [ 29 ] , they concluded that due to LLMs’ unreliability at answering multiple choice questions (MCQs), any personality scores derived from MCQs would also be unreliable. Huang et al. [ 30 ] also used BFI to test the degree to which varying the format of the assessment questions affected the resulting personality scores, showing that GPT-3.5-Turbo, GPT-4 and Gemini-Pro were able to generate stable responses across diverse settings. All three models exhibited tendencies towards specific personality traits. Muhua Huang et al. [ 31 ] concluded that although agents could be assigned personas that performed similarly to humans in psychometric tests (BFI), these assigned traits did not translate into personality-consistent behaviours. In particularly, the agents’ risk-taking behaviours aligned with human patterns, but ethical dilemma decisions did not. de Winter et al. [ 32 ] generated 2,000 personas, evaluated them using the BFI-10 [ 33 ] , and found their results aligned closely with human baselines. However, when evaluating a second persona set, they found that the correlation matrix among personality constructs was affected by persona set. Psychometric tests such as BFI-10 and HEXACO-60 are short-form tests developed for practical human assessments and are secondary interpretations of broader personality structure theories [ 33 , 34 , 24 ] . Applying these measures to LLMs compounds the conceptual limitations, as they assume a stable identity and the ability for introspection, neither of which truly apply to LLMs. Consequently, any results from such psychometric testing of LLMs should be interpreted with caution, as outcomes may reflect superficial patterns rather than meaningful personality attributes, leading researchers to draw misleading conclusions about the capabilities and validation of generative agents as accurate proxies for human participants in social science. 1.2 Our Contribution To explore the validity of applying psychometric tests to LLMs, this research replicates the foundational experiment that

Read the complete text