CAPE: Context-Aware Personality Evaluation Framework for Large Language Models
Jivnesh Sandhan, Fei Cheng, Tushar Sandhan, Yugo Murawaki, Kyoto University, Japan, IIT Kanpur, India, {jivnesh,feicheng,murawaki}@i.kyoto-u.ac.jp, [email protected] · 2025
Read CAPE: Context-Aware Personality Evaluation Framework for Large Language Models by Jivnesh Sandhan, Fei Cheng, Tushar Sandhan, Yugo Murawaki, Kyoto University, Japan, IIT Kanpur, India, {jivnesh,feicheng,murawaki}@i.kyoto-u.ac.jp, [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-20385.txt.
Introduction
Jivnesh Sandhan, Fei Cheng, Tushar Sandhan, and Yugo Murawaki gave a framework for testing personality in language models with context in two thousand twenty-five. Personality tests made for people are now applied to models, but usually one question at a time. Real use is a conversation, where history shapes each reply.
Their framework, CAPE, feeds prior exchanges into the test and adds metrics for response consistency. Across seven models, history improved consistency through in-context learning but also shifted measured personality. Two GPT models shifted the most, while others were more sensitive to question order. Applied to role-playing agents, context shifts improved consistency and matched human judgments better. Code and data are public.
Catalog record
| CREATOR | Jivnesh Sandhan, Fei Cheng, Tushar Sandhan, Yugo Murawaki, Kyoto University, Japan, IIT Kanpur, India, {jivnesh,feicheng,murawaki}@i.kyoto-u.ac.jp, [email protected] |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 53 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-20385
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
CAPE: Context-Aware Personality Evaluation Framework for Large Language Models Jivnesh Sandhan, Fei Cheng, Tushar Sandhan, Yugo Murawaki, Kyoto University, Japan, IIT Kanpur, India, {jivnesh,feicheng,murawaki}@i.kyoto-u.ac.jp, [email protected] Abstract Psychometric tests, traditionally used to assess humans, are now being applied to Large Language Models (LLMs) to evaluate their behavioral traits. However, existing studies follow a context-free approach, answering each question in isolation to avoid contextual influence. We term this the Disney World test, an artificial setting that ignores real-world applications, where conversational history shapes responses. To bridge this gap, we propose the first Context-Aware Personality Evaluation (CAPE) framework for LLMs, incorporating prior conversational interactions. To thoroughly analyze the influence of context, we introduce novel metrics to quantify the consistency of LLM responses, a fundamental trait in human behavior. Our exhaustive experiments on 7 LLMs reveal that conversational history enhances response consistency via in-context learning but also induces personality shifts, with GPT-3.5-Turbo and GPT-4-Turbo exhibiting extreme deviations. While GPT models are robust to question ordering, Gemini-1.5-Flash and Llama-8B display significant sensitivity. Moreover, GPT models response stem from their intrinsic personality traits as well as prior interactions, whereas Gemini-1.5-Flash and Llama--8B heavily depend on prior interactions. Finally, applying our framework to Role Playing Agents (RPAs) shows context-dependent personality shifts improve response consistency and better align with human judgments. 1 1 1 Our code and datasets are publicly available at: https://github.com/jivnesh/CAPE 1 Introduction Large Language Models (LLMs) have made significant advances in generating human-like text Spangher et al. 2024 ; Ou et al. 2024 . Moving beyond linguistic fluency, this raises the fundamental question: “How human-like are LLMs?” Researchers are utilizing psychometrics to assess whether LLMs exhibit human-like personality traits Huang et al. 2024b ; Serapio-García et al. 2023 . This question is crucial in contexts where LLMs may act as human proxies in surveys Dillion et al. 2023 ; Harding et al. 2024 or personalized human-AI interactions Tseng et al. 2024 . Figure 1: Illustration of the difference between existing context-free (left) and the proposed context-aware (right) evaluation framework. In the left setting, each question is asked independently, with no prior history (indicated by a different background color). In the right setting, all questions are part of the same session, where prior answers can influence future responses. Psychology provides well-established personality assessment frameworks, such as the Big Five Personality model McCrae and John 1992 , which evaluate individuals through a series of questions rated on a Likert scale. Analogously, LLMs are assessed using a zero-shot multiple-choice question (MCQ) format (§ 2 ). However, the reliability of these assessments remains a subject of debate. Some researchers advocate for methodologies that measure intrinsic personality traits in LLMs Jiang et al. 2023 ; Wang et al. 2024b ; Yang et al. 2023 ; Jiang et al. 2024 , while others highlight inconsistencies stemming from prompt sensitivity Shu et al. 2024 ; Gupta et al. 2024 ; Song et al. 2023 . We identify a critical research gap in existing research: LLM personality assessments are conducted in an isolated, context-independent manner. We term this the Disney World test setting, where each question is answered without influence from prior responses. In contrast, real-world applications of LLMs necessitate exposure to conversational history. Before deploying LLMs in critical domains such as education and healthcare, it’s vital to understand how conversational history impacts personality assessments. To bridge this gap, we propose the first Context-Aware Personality Evaluation (CAPE) framework, where prior questions and responses are retained in the conversational history to evaluate their impact on LLM personality (§ 3.1 ). To thoroughly analyze the influence of context, we assess response consistency, a fundamental human trait, by introducing various inconsistency factors related to prompt sensitivity such as temperature, option wording, option order, instruction and item paraphrasing. To quantify this consistency, we present novel metrics that measure the response pattern similarity across multiple LLM runs (§ 3.2 ). Our key findings on the impact of context on LLM personality are: Prior conversational history enhances response consistency by serving as few-shot in-context learning (§ 6.1 ). However, introducing context leads to deviations in responses compared to context-independent setting, with GPT-3.5-Turbo and GPT-4-Turbo exhibiting extreme personality shifts (§ 6.2 ). Additionally, GPT models maintain intrinsic personality despite contextual influence, whereas Gemini-1.5-Flash and Llama-3.1-8B rely heavily on prior conversation (§ 6.3 ). While these GPT models remain robust to question ordering, Gemini-1.5-Flash and Llama-3.1-8B display significant sensitivity (§ 6.4 ). Finally, we demonstrate our framework on Role Playing Agents (RPAs) and show context-dependent personality shifts improve response consistency and better align with human scores (§ 6.5 ). Our key contributions are: • To the best of our knowledge, we introduce the first Context-Aware Personality Evaluation (CAPE) framework, demonstrating its role in enhancing consistency (§ 3 ). • We propose novel metrics to quantify the consistency of LLM in assessments (§ 3.2 ). • We conduct an in-depth analysis of how context influences LLM’s personality (§ 6 ). 2 Preliminaries: LLM’s Personality Test The Big Five personality framework McCrae and John 1992 ; John and Srivastava 1999 characterizes human personality using 5 fundamental traits: O penness (artistic, imaginative), C onscientiousness (organized, thorough), E xtraversion (assertive, talkative), A greeableness (appreciative, kind), and N euroticism (anxious, worrying), collectively known as OCEAN. Following earlier works Huang et al. 2024a ; Jiang et al. 2023 ; Zhou et al. 2023 , we assess the personality of an LLM by formulating the evaluation as a zero-shot multiple-choice question-answering task. Each assessment item consists of a self-descriptive statement and a set of response options. The model is prompted to evaluate how accurately the statement aligns with its personality by selecting the most appropriate response. The prompt for it as: Given a statement about yourself: “You { Item } .” Please select the most accurate description of how well this statement applies to you from these options: (A) Very Accurate (B) Moderately Accurate (C) Neither Accurate Nor Inaccurate (D) Moderately Inaccurate (E) Very Inaccurate where the Item describes behavioral tendency from a second-person perspective. Each item corresponds to one of the 5 OCEAN dimensions and is either positively (+Key) or negatively (-Key) related to that dimension. For example, ( -O ): “Do not like poetry” and ( +O ): “Love to daydream” are negatively and positively correlated to openness, respectively. The responses are numerically scored based on their alignment with the corresponding trait dimension. If an item is positively correlated, options (A) to (E) are scored from 5 to 1; otherwise, if negatively correlated, they are scored from 1 to 5. For an assessment consisting of m items, the sequence of scores assigned to an LLM’s responses forms a scoring trajectories , which serve as the foundation for consistency analysis in our proposed framework. Mathematically, we define the scoring trajectory as, \mathcal{T}=[s_{1},s_{2},\dots,s_{m}] where \mathcal{T} is the scoring trajectory of the LLM, s_{i} is the score assigned to the LLM’s response for item i and m is the total number of items. This trajectory captures the model’s response pattern across all items and is later utilized in our framework to analyze the consistency of LLM-generated personality. For a given OCEAN trait d (where d\in\{O,C,E,A,N\} ), the trait score of an LLM is computed as the average score across