Synthetic Users, Real Differences: an Evaluation Framework for User Simulation in Multi-Turn Conversations
Yu Lu LiuHyokun Yun, Tanya Roosta, Ziang Xiao · 2026
Read Synthetic Users, Real Differences: an Evaluation Framework for User Simulation in Multi-Turn Conversations by Yu Lu LiuHyokun Yun, Tanya Roosta, Ziang Xiao (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-02624.txt.
Description
Yu Lu Liu and three co-authors build a framework for judging how realistic simulated users are in multi-turn chats with chatbots. Simulated users could replace costly collection of real user data for testing chatbots, but only if their dialogues feel real. Older checks give a coarse score for a single dialogue, with no wider view.
Their framework, called realsim, compares real and simulated dialogues as distributions across eight dimensions. Those dimensions cover what the talk is doing, the user's state, and the surface form of messages. It is tested on a curated set of one thousand real task-focused dialogues across sixteen application domains. Simulated users, they find, miss the friction real users bring, which can make simulation-based tests too optimistic. Performance also varied by domain.
Catalog record
| CREATOR | Yu Lu LiuHyokun Yun, Tanya Roosta, Ziang Xiao |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 73 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.02624