Can LLMs Simulate Personas with Reversed Performance? A Benchmark for Counterfactual Instruction Following

Sai Adith Senthil Kumar, Hao Yan, Saipavan Perepa, Murong Yue, Ziyu Yao · 2025

Read Can LLMs Simulate Personas with Reversed Performance? A Benchmark for Counterfactual Instruction Following by Sai Adith Senthil Kumar, Hao Yan, Saipavan Perepa, Murong Yue, Ziyu Yao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-06460.txt.

Introduction

Sai Adith Senthil Kumar and co-authors present a 2025 computing paper on a blind spot in persona simulation by language models. Models are often asked to play roles, such as a student in a tutoring scene. The team found that even top models cannot play a role whose performance is reversed, like a weak student who should get answers wrong.

They built the first benchmark for this skill, which they call counterfactual instruction following, using math reasoning as the test bed. Open and closed models were tried, including a leading reasoning model. All struggled to stay in the weaker role. Adding a racial identity to the persona made the failure worse. The authors argue that this gap limits the range of simulated worlds these models can support.

Catalog record

CREATORSai Adith Senthil Kumar, Hao Yan, Saipavan Perepa, Murong Yue, Ziyu Yao
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE55 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-06460

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Can LLMs Simulate Personas with Reversed Performance? A Benchmark for Counterfactual Instruction Following Sai Adith Senthil Kumar, Hao Yan, Saipavan Perepa, Murong Yue, Ziyu Yao Abstract Large Language Models (LLMs) are now increasingly widely used to simulate personas in virtual environments, leveraging their instruction-following capability. However, we discovered that even state-of-the-art LLMs cannot simulate personas with reversed performance (e.g., student personas with low proficiency in educational settings), which impairs the simulation diversity and limits the practical applications of the simulated environments. In this work, using mathematical reasoning as a representative scenario, we propose the first benchmark dataset for evaluating LLMs on simulating personas with reversed performance, a capability that we dub “counterfactual instruction following” . We evaluate both open-weight and closed-source LLMs on this task and find that LLMs, including the OpenAI o1 reasoning model, all struggle to follow counterfactual instructions for simulating reversedly performing personas. Intersectionally simulating both the performance level and the race population of a persona worsens the effect even further. These results highlight the challenges of counterfactual instruction following and the need for further research. 1 1 1 Dataset and source code will be released soon. 1 Introduction Leveraging the generalized knowledge they have learned from large-scale pre-training and the instruction following capability they obtained from careful post-training, Large Language Models (LLMs) have now been increasingly widely used to simulate personas. This simulation is typically implemented via prompt engineering, where an LLM is instructed by a description of the persona specification (e.g., name, age, profession, etc.) to react to the given context as role-playing. Today, LLM-based persona simulation has been applied to simulate celebrities ( Shao et al. 2023 ; Zhou et al. 2023 ) , collaborative roles in workplaces (e.g., software development, online recruiting) ( Li et al. 2023 ; Sun et al. 2024 ; Hong et al. 2024 ) , or general characters ( Park et al. 2023 ; Zhou et al. 2023 ; Xie et al. 2024 ; Samuel et al. 2024 ; Hu & Collier 2024 ; Tu et al. 2024 ; Tseng et al. 2024 ; Wang et al. 2025b ; Wang et al. 2025a ) . Despite these promising developments, existing research on persona simulation has primarily focused on generating personas that consistently demonstrate high competence, accuracy, and task success. These traits align with the default behavior of LLMs, which are trained to optimize performance and reliability on downstream tasks. However, many real-world applications require personas that exhibit reversed or intentionally low-performing behaviors. For instance, in education, simulating low-performing students can provide more realistic peer learning opportunities, such as encouraging students to identify common mistakes made by their classmates and learn to explain concepts for more robust concept understanding (i.e., learning by teaching) ( Weijers et al. 2024 ; Hu et al. 2025 ) . To our knowledge, there has been no investigation into whether LLMs can effectively follow instructions to simulate such counterfactual personas, whose task performance runs contrary to the typically expected optimal abilities. This capability is distinct from conventional instruction-following, as it requires the model to override its general behavior to provide optimal solutions and instead simulate a persona that demonstrates limited understanding or lower task performance ( Mao et al. 2024 ; Tan et al. 2024 ) . We refer to this capability as Counterfactual Instruction Following as shown in Figure 1 . Figure 1: Illustration of evaluating LLMs for simulating personas with high- and low-proficiency in math reasoning. For low-performing persona simulation, we additionally instruct the model to leave comments and explain evidence of low performance, which encourages more faithful simulation and helps response analysis. To perform this investigation, we present ReverSO , a novel benchmark dataset evaluating whether an LLM can follow counterfactual instructions to simulate personas with reversed performance (Figure 1 ). Our dataset is designed using mathematical reasoning as a representative scenario and is adapted from the widely adopted GSM8k dataset ( Cobbe et al. 2021 ) . An LLM is tasked with two types of instructions, corresponding to simulating students with high- and low-performing in math reasoning, respectively, and is then asked to answer a given math reasoning question exhibiting the desired persona. The model is finally evaluated based on a contrast of its behaviors under the two personas. We expect that, for high-performing student simulation, the model should provide advanced reasoning and correct solutions, whereas for low-performing student simulation, it should show misunderstandings, errors, or incomplete reasoning, as specified in the instruction. To further examine the robustness of LLMs in counterfactual instruction following, ReverSO also includes an intersectional setting where the persona’s race is specified. This extension is motivated by recent findings that assigning racial attributes to LLMs can implicitly alter their behavior and introduce biases in reasoning tasks ( Gupta et al. 2024 ) . By incorporating race attributes, we aim to test whether such racial context affects a model’s ability to simulate reversed-performance personas. We evaluated both open-weight LLMs, including Llama3.1–8B/70B ( Grattafiori et al. 2024 ) , and closed-source LLMs, including GPT-4o/4-turbo ( Achiam et al. 2023 ) , 3.5-turbo ( OpenAI 2022 ) , o1 ( Jaech et al. 2024 ) , and Claude-3.5 ( Anthropic 2024 ) , on ReverSO and experimented with multiple approaches, including employing self-refinement ( Madaan et al. 2023 ; Kim et al. 2023 ) to enhance the simulation. Surprisingly, our results show that while some models are able to reduce task accuracy when simulating low-performing personas, this behavior is inconsistent and often limited. Most models show less than a 5% accuracy drop under zero-shot prompting, despite explicit instructions to underperform. For example, OpenAI o1 maintains 99.0% accuracy across all personas, and GPT-4-turbo drops by only 4.6%. Notably, even when models fail to lower their accuracy, some still produce noticeably different reasoning styles—such as changes in tone, confidence, or logical flow—when switching between personas. OpenAI o1 is a notable example: it generates distinct reasoning patterns for high- and low-performing personas despite producing the same correct answers. When an LLM intersectionally simulates both the performance level and the race population of a persona, the desired discrepancy between the personas’ task accuracy further shrinks. These results underscore the difficulty for LLMs to perform counterfactual instruction following and the need for further research to enable more controllable and diverse persona simulation. 2 ReverSO : A Benchmark of Counterfactual Instruction Following for Reversed Persona Simulation In this section, we describe our benchmark dataset, ReverSO , and its evaluation metrics. 2.1 Task Setting Motivated by Yue et al. 2024 , we consider the application of persona simulation in mathematics education, where the goal is to simulate students with two (i.e., high and low) levels of proficiency in math reasoning. The task is illustrated in Figure 1 . We note that our goal is not to simulate realistic student behaviors, such as how real-life students at different competence levels will solve the same math problem; rather, we aim to use this motivating scenario to evaluate whether LLMs can correctly follow common versus counterfactual instructions. As a result, we designed both instructions to be highly specific and expected capable LLMs to display distinct behaviors when they simulated the two types of student personas in answering math questions. 2

Read the complete text