Synthetic Users, Real Differences: an Evaluation Framework for User Simulation in Multi-Turn Conversations

Yu Lu LiuHyokun Yun, Tanya Roosta, Ziang Xiao · 2026

Read Synthetic Users, Real Differences: an Evaluation Framework for User Simulation in Multi-Turn Conversations by Yu Lu LiuHyokun Yun, Tanya Roosta, Ziang Xiao (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-02624.txt.

Description

Yu Lu Liu and three co-authors build a framework for judging how realistic simulated users are in multi-turn chats with chatbots. Simulated users could replace costly collection of real user data for testing chatbots, but only if their dialogues feel real. Older checks give a coarse score for a single dialogue, with no wider view.

Their framework, called realsim, compares real and simulated dialogues as distributions across eight dimensions. Those dimensions cover what the talk is doing, the user's state, and the surface form of messages. It is tested on a curated set of one thousand real task-focused dialogues across sixteen application domains. Simulated users, they find, miss the friction real users bring, which can make simulation-based tests too optimistic. Performance also varied by domain.

Catalog record

CREATORYu Lu LiuHyokun Yun, Tanya Roosta, Ziang Xiao
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE73 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.02624

Browse the full library · About the Tower of Babel