Principled Personas: Defining and Measuring the Intended Effects of Persona Prompting on Task Performance

Pedro Henrique Luz de Araujo, Paul Röttger, Dirk Hovy, Benjamin Roth · 2025

Read Principled Personas: Defining and Measuring the Intended Effects of Persona Prompting on Task Performance by Pedro Henrique Luz de Araujo, Paul Röttger, Dirk Hovy, Benjamin Roth (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-19764.txt.

Introduction

Pedro Henrique Luz de Araujo and colleagues gave persona prompting a principled audit in 2025. Telling a language model to answer as an expert is everyday practice, yet the evidence for it was anecdotal and mixed. The authors distilled three standards from the literature: an expert persona should beat no persona, irrelevant persona details should not matter, and behavior should stay faithful to the persona given.

Nine leading language models were run across twenty-seven tasks against those standards. Expert personas usually helped or at least did no harm. The robustness standard failed badly: irrelevant persona details swung performance by as much as thirty percentage points, a sensitivity nobody designing prompts had priced in. Fidelity results were mixed, with boosts from education, specialization, and domain match appearing inconsistently across models and tasks.

Catalog record

CREATORPedro Henrique Luz de Araujo, Paul Röttger, Dirk Hovy, Benjamin Roth
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE56 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-19764

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Principled Personas: Defining and Measuring the Intended Effects of Persona Prompting on Task Performance Pedro Henrique Luz de Araujo, Paul Röttger, Dirk Hovy, Benjamin Roth Abstract Expert persona prompting—assigning roles such as expert in math to language models—is widely used for task improvement. However, prior work shows mixed results on its effectiveness, and does not consider when and why personas should improve performance. We analyze the literature on persona prompting for task improvement and distill three desiderata: 1) performance advantage of expert personas, 2) robustness to irrelevant persona attributes, and 3) fidelity to persona attributes. We then evaluate 9 state-of-the-art LLMs across 27 tasks with respect to these desiderata. We find that expert personas usually lead to positive or non-significant performance changes. Surprisingly, models are highly sensitive to irrelevant persona details, with performance drops of almost 30 percentage points. In terms of fidelity, we find that while higher education, specialization, and domain-relatedness can boost performance, their effects are often inconsistent or negligible across tasks. We propose mitigation strategies to improve robustness—but find they only work for the largest, most capable models. Our findings underscore the need for more careful persona design and for evaluation schemes that reflect the intended effects of persona usage. Figure 1: We define three desiderata for persona prompting : Task experts should perform on par or better than the no-persona model ( Expertise Advantage ); Irrelevant attributes such as names should not influence model performance ( Robustness ); relevant attributes such as domain expertise should shape performance accordingly ( Fidelity ). 1 Introduction Shortly after the release of ChatGPT, users started exploring the use of expert persona prompts to improve task performance. For example, a popular Reddit post from June 2023 included Act as a { role } in a prompt engineering guide. 1 1 1 https://www.reddit.com/r/ChatGPTPromptGenius/comments/144i0tb/the_complete_chatgpt_cheatsheet/. Since then, a large body of academic research has sought to evaluate the impact of different personas on large language model (LLM) task performance, often finding conflicting results Kong et al. 2024 ; Zheng et al. 2024 . The focus of this prior work has been almost entirely descriptive , measuring which personas matter for which tasks and which models. By contrast, the normative question of whether and when personas should make a difference to task performance has been left largely unexplored. This is a missed opportunity because, from a model development perspective, it is much more valuable to define what effects from persona prompting are desirable or not, and to then compare these expectations to real model behaviors. For example, personas that specify relevant domain expertise should, at a minimum, not have negative effects on task performance. Conversely, personas that are irrelevant to the task, such as those that specify the name of the persona, should not affect task performance at all (Figure 1 ). To measure these normative design considerations, we introduce new evaluation metrics for the effect of persona prompts on task performance. Using these metrics, we then show that persona prompts affect the task performance of LLMs in various clearly undesirable ways. For example, even state-of-the-art models like Llama-3.1-70B and Qwen2.5-72B are often not robust to irrelevant persona attributes such as names and favorite colors. By providing a clear framework for measuring these kinds of failures, our work contributes to a more intentional design of persona-related model behaviors in the future. Overall, we make four main contributions : 1. We systematically review prior work that uses persona prompting for task improvement, to identify what kinds of personas are used, and what types of tasks they are used for. 2. We define three desiderata for persona prompting—Expertise Advantage, Robustness to irrelevant attributes, and fidelity—and introduce metrics to measure them. 3. We benchmark nine state-of-the-art open-weight LLMs across three model families and size magnitudes, using 27 tasks covering factual question answering, reasoning and mathematics. 4. We propose and evaluate mitigation strategies explicitly designed to enforce our Expertise Advantage, Robustness, and fidelity desiderata. All our experimental code and data is available at https://github.com/peluz/principled-personas . 2 Literature Review: Persona Prompting for Task Performance Improvement On October 17th 2024, we searched the ACL Anthology for papers published in or after 2021 using the keywords “persona” and “role-play”. This resulted in 170 papers, of which we retained those 9 papers that used personas explicitly to improve task performance. We then recursively examined papers citing these 9 papers, applying the same criteria, and thus identified an additional 12 papers. Table 3 in Appendix A lists the full set of 21 papers, summarizing the personas they used, the tasks they evaluated on, and the models they tested. 2.1 Review Findings Persona prompting is used across a wide range of tasks , from closed-form tasks such as code generation Dong et al. 2024 ; Hong et al. 2024 ; Qian et al. 2024 , mathematical reasoning Du et al. 2024 ; Kong et al. 2024 , and factual QA Salewski et al. 2023 ; Chen et al. 2024b ; Tang et al. 2024 , to more open-ended settings like research ideation Nigam et al. 2024 and creative writing Wang et al. 2024c . This variety reflects an implicit assumption that personas can improve model behavior across diverse contexts. The types of personas used are also diverse. Papers often assign task-relevant persona attributes, such as occupation—for example, a medical doctor Tang et al. 2024 or software developer Qian et al. 2024 —and domain expertise, such as an LLM-generated domain expert Wang et al. 2024c , an expert in computer science Salewski et al. 2023 , or an information specialist Wang et al. 2023 . Other papers use more unconventional or abstract personas, such as a devil’s advocate Kim et al. 2024 and inanimate objects, e.g., a coin for a coin-flipping task Kong et al. 2024 . Some works also include attributes with unclear relevance to the task, ranging from clearly irrelevant ones such as persona name Chan et al. 2024 ; Hong et al. 2024 to maybe behaviorally relevant attributes like age or education level Salewski et al. 2023 ; Wang et al. 2024c . The set of models used is quite restricted. 15 out of 21 papers evaluate only OpenAI models—often without specifying which one, referring vaguely to ChatGPT or GPT-3.5. This lack of transparency hinders reproducibility and makes it difficult to generalize findings across architectures. Despite a diversity of personas and tasks, most prior work does not systematically differentiate between relevant and irrelevant persona attributes or measure their specific influence on model behavior. Moreover, methodological gaps make it difficult to assess the impact of personas on task performance: unequal comparisons, such as using a stronger model to process persona responses Li et al. 2023 , and a lack of no-persona controls Hong et al. 2024 ; Salewski et al. 2023 ; Lin et al. 2022 make it difficult to isolate the effects of personas on task performance. Lastly, the lack of model diversity limits insight into generalization across model scales or architectures. 2.2 Implications for Experimental Design Our experiments are designed to fill these gaps by explicitly testing the effects of different persona types across a diverse range of tasks and models. To do so, we cover several task types (§ 4 ),

Read the complete text