Lost in Simulation: LLM-Simulated Users are Unreliable Proxies for Human Users in Agentic Evaluations

Preethi Seshadri Samuel Cahyawijaya Ayomide Odumakinde, Sameer Singh Seraphina Goldfarb-Tarrant, [0.4em] UC Irvine Cohere [0.4em] {preethis, sameer}@uci.edu {samuelcahyawijaya, ayomideodumakinde, seraphina}@cohere.com · 2026

Read Lost in Simulation: LLM-Simulated Users are Unreliable Proxies for Human Users in Agentic Evaluations by Preethi Seshadri Samuel Cahyawijaya Ayomide Odumakinde, Sameer Singh Seraphina Goldfarb-Tarrant, [0.4em] UC Irvine Cohere [0.4em] {preethis, sameer}@uci.edu {samuelcahyawijaya, ayomideodumakinde, seraphina}@cohere.com (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-17087.txt.

Introduction

Preethi Seshadri and a team of four co-authors test fake users in chatbot grades. Bench tests now use chatbot-made users to save cost. Those fake users chat with the bot on test tasks. No one had checked if fakes stand in well for folks. Their 2026 preprint runs a user study on four lands' folk. The lands are the US, India, Kenya, and Nigeria.

Real users did shop-type tasks from a known bench set. Their acts were set next to fake users' acts. Gaps in grade, truth, and fair marks then show. Fake users may grade bots in skewed ways. Bench teams need that truth to trust their scores.

Catalog record

CREATORPreethi Seshadri Samuel Cahyawijaya Ayomide Odumakinde, Sameer Singh Seraphina Goldfarb-Tarrant, [0.4em] UC Irvine Cohere [0.4em] {preethis, sameer}@uci.edu {samuelcahyawijaya, ayomideodumakinde, seraphina}@cohere.com
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE47 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-17087

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Lost in Simulation: LLM-Simulated Users are Unreliable Proxies for Human Users in Agentic Evaluations Preethi Seshadri Samuel Cahyawijaya Ayomide Odumakinde, Sameer Singh Seraphina Goldfarb-Tarrant, [0.4em] UC Irvine Cohere [0.4em] {preethis, sameer}@uci.edu {samuelcahyawijaya, ayomideodumakinde, seraphina}@cohere.com Abstract Agentic benchmarks increasingly rely on LLM-simulated users to scalably evaluate agent performance, yet the robustness, validity, and fairness of this approach remain unexamined. Through a user study with participants across the United States, India, Kenya, and Nigeria, we investigate whether LLM-simulated users serve as reliable proxies for real human users in evaluating agents on τ \tau -Bench retail tasks. We find that user simulation lacks robustness, with agent success rates varying up to 9 percentage points across different user LLMs. Furthermore, evaluations using simulated users exhibit systematic miscalibration, underestimating agent performance on challenging tasks and overestimating it on moderately difficult ones. African American Vernacular English (AAVE) speakers experience consistently worse success rates and calibration errors than Standard American English (SAE) speakers, with disparities compounding significantly with age. We also find simulated users to be a differentially effective proxy for different populations, performing worst for AAVE and Indian English speakers. Additionally, simulated users introduce conversational artifacts and surface different failure patterns than human users. These findings demonstrate that current evaluation practices risk misrepresenting agent capabilities across diverse user populations and may obscure real-world deployment challenges. 1 Introduction Figure 1: Conversational snippets between an agent and simulated (top) vs. human user (bottom) on the same task. Simulated users exhibit increased question-asking and politeness compared to human users. AI agents designed to assist with everyday tasks such as travel reservations, order management, and appointment scheduling are becoming increasingly prevalent ( 37 ; 42 ; 6 ) , but present significant challenges to effective evaluation. Agentic benchmarks have needed to evolve beyond static question-answering and other single-turn formats to capture the dynamic, multi-turn nature of real user interactions ( 5 ; 7 ) . Many recent works have proposed benchmarks that reflect this shift and instead measure sustained, context-aware interaction ( 2 ; 29 ; 34 ; 39 ; 40 ) . These benchmarks improve complexity and ecological validity over prior static evaluations by requiring agents to demonstrate a range of capabilities, including conversing naturally and coherently with users, adhering to policies, following instructions closely, and making appropriate tool calls over multiple turns. To facilitate automated and scalable evaluation, these benchmarks typically simulate conversations between an LLM agent and a user, where the “user” is an LLM ( 14 ; 13 ; 19 ) . While this approach reduces the cost and operational overhead of human evaluation, it raises critical questions about the robustness , validity , and fairness of user simulation. First, evaluations typically rely on a single user simulation model, yet results may vary across different user LLMs ( robustness ). Second, without validation with actual users ( 24 ) , it remains unclear whether interactions between agents and LLM-simulated users accurately reflect and predict interactions between agents and real people ( validity , Figure 1 ). If simulated users systematically differ from actual users in their interaction patterns ( 41 ) , benchmarks may provide a misleading picture of agent capabilities due to miscalibration (i.e., simulated results do not reliably predict real user outcomes). Third, these evaluations often treat users as a homogeneous group, overlooking variation in how people communicate and interact with AI systems ( 12 ; 17 ; 3 ) . In practice, users differ widely in their communication styles, linguistic backgrounds, and cultural norms ( 20 ; 23 ) . For example, even in a simple retail assistance scenario, users might vary along dimensions such as formality, verbosity, and politeness norms—but it remains unclear how much this diversity meaningfully impacts agent performance and task success ( 32 ) . Without further investigation, simulated users may approximate some populations better than others, resulting in non-uniform calibration errors that advantage or disadvantage certain user groups ( fairness ). Despite the widespread adoption of user simulation, prior work has overlooked validation against real human interactions in agentic benchmarks. In this paper, we address this gap by conducting a user study with participants from the United States, India, Kenya, and Nigeria to directly evaluate user simulation as a proxy for actual users. Specifically, we ask the following questions: • Robustness: How consistent are agentic evaluations across different user simulation LLMs? • Validity: Do simulated users serve as reliable proxies for real human users in agentic evaluations? • Fairness: How does human-agent performance vary across different user groups, and does user simulation represent certain groups better than others? Using \tau -Bench retail tasks ( 40 ) as a case study, we find that user simulation lacks robustness, and systematically misestimates performance with human users by underestimating success on the most challenging tasks while overestimating outcomes on moderately difficult scenarios. More critically, simulated users exhibit notable demographic biases and perform particularly poorly as proxies for African American Vernacular English speakers, with disparities compounding with age. Simulated users also introduce artificial conversational artifacts such as heightened question-asking and politeness (Figure 1 ). Together, these findings call into question the validity of user simulation as a stand-alone evaluation paradigm and underscore the need for more robust and fair approaches to agentic evaluation. 2 Related Work User Simulation in Interactive Settings 8 and 36 investigate user simulation in tasks such as math tutoring and daily planning, but primarily focus on conversational characteristics (e.g., politeness) and behavioral realism (e.g., Turing-style tests). Additionally, 8 optimize alignment with human ratings using simulated user profiles. 18 analyze simulation fidelity by measuring how accurately simulated users replicate human intermediate steps in real-world online shopping sessions, and find substantial deviations between simulated and actual user action sequences. None of these works examine the robustness of agentic evaluations across different user simulation models, nor do they consider fairness implications across different user groups. Finally, while 43 study the outcome and task validity of agentic benchmark results, they do not consider the validity of user simulation. Demographic Skews in NLP Datasets and Models Several previous works highlight that datasets and models exhibit systematic skews toward specific demographic perspectives. Research on annotator disagreement reveals that perceptions of safety, offensiveness, and toxicity meaningfully vary along demographic axes like race, gender, and political affiliation ( 28 ; 15 ; 22 ) . These patterns of disagreement reflect broader issues of positionality—both 26 and 16 show that NLP datasets and models tend to align predominantly with Western, educated, and Anglosphere populations. Similarly, LLMs have been found to reflect the opinions of Western countries ( 9 ; 4 ) as well as wealthy and liberal groups ( 25 ) , and align more closely with White annotators than Black or Asian groups on subjective tasks like politeness ( 30 ) . Attempts to broaden model inclusivity through sociodemographic prompting have shown mixed results, often failing to consistently improve alignment or relying on harmful stereotypes ( 9 ; 30 ) . Overall, our work builds on this line of research by examining demographic differences in agentic settings. 3 Methodology 3.1 Benchmark Background We use \tau -Bench ( 40 ) as the testbed for our evaluations, since it is a well-known and widely-adopted benchmark for agentic tool

Read the complete text