Simulating User Diversity in Task-Oriented Dialogue Systems using Large Language Models

Adnan Ahmad, Stefan Hillmann, Sebastian Möller · 2025

Read Simulating User Diversity in Task-Oriented Dialogue Systems using Large Language Models by Adnan Ahmad, Stefan Hillmann, Sebastian Möller (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-12813.txt.

Description

Adnan Ahmad and co-authors study how large language models can act as fake users for task based dialogue systems in 2025. Such systems need tests with many kinds of people, not just one plain user type. The team builds profiles that vary in age and background, goals, talk style, prior knowledge, and interests.

Those profiles then hold multi turn talks with a dialogue system, and the talks are scored for success. A close look at the profiles checks spread, sameness, and bias. The authors find that one model, GPT-o1, made a more mixed set across most traits, while GPT-4o made more skewed sets. The work shows both promise and limits for model made test users.

Catalog record

CREATORAdnan Ahmad, Stefan Hillmann, Sebastian Möller
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE22 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-12813

Browse the full library · About the Tower of Babel