Sleepless Nights, Sugary Days: Creating Synthetic Users with Health Conditions for Realistic Coaching Agent Interactions

Taedong Yun, Eric Yang, Mustafa Safdari, Jong Ha Lee, Vaishnavi Vinod Kumar, S. Sara Mahdavi, Jonathan Amar, Derek Peyton, Reut Aharony, Andreas Michaelides, Logan Schneider, Isaac Galatzer-Levy, Yugang Jia, John Canny, Arthur Gretton, Maja Matarić · 2025

Read Sleepless Nights, Sugary Days: Creating Synthetic Users with Health Conditions for Realistic Coaching Agent Interactions by Taedong Yun, Eric Yang, Mustafa Safdari, Jong Ha Lee, Vaishnavi Vinod Kumar, S. Sara Mahdavi, Jonathan Amar, Derek Peyton, Reut Aharony, Andreas Michaelides, Logan Schneider, Isaac Galatzer-Levy, Yugang Jia, John Canny, Arthur Gretton, Maja Matarić (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-13135.txt.

Introduction

Taedong Yun and co-authors present a 2025 frame for building fake users to test health coaching agents. Real trials with people are slow, so test users shaped by real health traits can give a safer first check. The study grounds its fake users in sleep habits and diabetes care.

Fake users are built in two steps, first as structured data on health and daily life, then as full profiles. Talks between the fake users and a coach are then played out with agent models or a language model. Human experts judged the talks blind, and two real coaching agents served as test cases for the frame.

Catalog record

CREATORTaedong Yun, Eric Yang, Mustafa Safdari, Jong Ha Lee, Vaishnavi Vinod Kumar, S. Sara Mahdavi, Jonathan Amar, Derek Peyton, Reut Aharony, Andreas Michaelides, Logan Schneider, Isaac Galatzer-Levy, Yugang Jia, John Canny, Arthur Gretton, Maja Matarić
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE73 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-13135

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Sleepless Nights, Sugary Days: Creating Synthetic Users with Health Conditions for Realistic Coaching Agent Interactions Taedong Yun, Eric Yang, Mustafa Safdari, Jong Ha Lee, Vaishnavi Vinod Kumar, S. Sara Mahdavi, Jonathan Amar, Derek Peyton, Reut Aharony, Andreas Michaelides, Logan Schneider, Isaac Galatzer-Levy, Yugang Jia, John Canny, Arthur Gretton, Maja Matarić Abstract We present an end-to-end framework for generating synthetic users for evaluating interactive agents designed to encourage positive behavior changes, such as in health and lifestyle coaching. The synthetic users are grounded in health and lifestyle conditions, specifically sleep and diabetes management in this study, to ensure realistic interactions with the health coaching agent. Synthetic users are created in two stages: first, structured data are generated grounded in real-world health and lifestyle factors in addition to basic demographics and behavioral attributes; second, full profiles of the synthetic users are developed conditioned on the structured data. Interactions between synthetic users and the coaching agent are simulated using generative agent-based models such as Concordia, or directly by prompting a language model. Using two independently-developed agents for sleep and diabetes coaching as case studies, the validity of this framework is demonstrated by analyzing the coaching agent’s understanding of the synthetic users’ needs and challenges. Finally, through multiple blinded evaluations of user-coach interactions by human experts, we demonstrate that our synthetic users with health and behavioral attributes more accurately portray real human users with the same attributes, compared to generic synthetic users not grounded in such attributes. The proposed framework lays the foundation for efficient development of conversational agents through extensive, realistic, and grounded simulated interactions. 1 Introduction Personalization is a key capability for any interactive agents aiming to encourage positive behavior changes, for instance in health and lifestyle ( Christakopoulou et al. 2024 ; Yang et al. 2024 ) . The efficacy of such agents must be assessed via interactions with users; however, collecting and evaluating diverse, long-term human interactions with agents is costly and time-consuming, and the development of such agents can be significantly accelerated with realistic simulation. We address the challenge of realistic user simulation in this context by proposing an end-to-end framework for generating cohorts of synthetic users grounded in health, lifestyle, and behavior, and having them interact with coaching agents. The framework is generally applicable across a range of health and lifestyle domains; we validate it in the domains of sleep and diabetes coaching. Synthetic users generated with large language models (LLMs) have been widely studied in recent years ( Kapania et al. 2024 ; Wang et al. 2025 ; Moon et al. 2024 ; Park et al. 2024 ; Park et al. 2023 ) , often for general purpose alignment and personalization ( Castricato et al. 2024 ) , and have been designed to reflect demographic information about the population at large. We focus on a more targeted application, where the synthetic users are employed for interaction with foundation models or agents specialized in health, as done in recent work Tu et al. 2024 ; Yu et al. 2024 ; Johri et al. 2025 . We provide an overview of such work in Section 2 . We demonstrate an end-to-end-framework for designing a cohort of synthetic users exhibiting health conditions consistent with their specific demographics and behavioral factors, in order for their interaction with a coaching agent to realistically reflect their needs and challenges. We construct the synthetic users by first generating a sample of natural language "vignettes", which are in turn generated based on real demographics, health, and behavioral data ( Goldberg 1992 ; Serapio-García et al. 2023 , such as the Big Five markers;) . Our framework also allows for optionally including additional information: domain-specific rubrics such as “barriers” and “goals” as used by human coaches; challenges to action derived from the COM-B behavioural model ( Michie et al. 2011 ) ; or rich backstories for the synthetic users, generated consistently with the structured attributes from real data. Finally, based on this cohort of vignettes, realistic user-coach simulation can be performed by direct LLM calls or by using generative agent-based models such as the Concordia ( Vezhnevets et al. 2023 ) open-source system. More details will be discussed in Section 3 . We demonstrate the efficacy of our methodology for generating health-grounded synthetic personas in two separate domains, sleep coaching ( Section 4 ) and diabetes coaching ( Section 5 ), using two independently developed sleep coaching and diabetes coaching agents. We verify that our synthetic users indeed communicate behaviors, health conditions, and barriers consistent with their assigned attributes, by inspecting the internal state-of-user model of the coaching agent for each synthetic user, by a holistic evaluation of the user-coach interactions by trained human experts, and by a comparative evaluation of our health-grounded synthetic users against generic synthetic users, again by trained human experts. 2 Background In this section, we briefly survey the development and goals of LLM-based synthetic personas in general, and then provide further details on prior literature on the design and application of synthetic personas for health. We next describe software systems for simulating agent interactions. Finally, we discuss some challenges in using LLM-derived synthetic users for agent development and beyond. 2.1 LLMs as Synthetic Personas The time and expense required in human evaluation of LLM agents naturally evokes the need for auto-evaluation. This has led to the development of “reinforcement learning from AI feedback (RLAIF)”, where LLMs serve to evaluate the output of other LLMs, in a now well-established strategy for maximizing alignment, factuality, helpfulness, and other desiderata ( Bai et al. 2022 ; Lee et al. 2024 ) . One recent approach to RLAIF uses LLMs to construct synthetic personas and to judge their interaction with the model being evaluated. Castricato et al. 2024 generated these synthetic personas from the US census data, and employed them to build an evaluation dataset of prompt and feedback pairs obtained solely from synthetic users. A strength of the approach is that results for specific users can be modeled, rather than general user categories. This approach also motivates the work of Moon et al. 2024 . In addition to demographic information, Serapio-García et al. 2023 showed that LLMs can be designed to display specific personality traits, such as those along the Big Five dimensions ( Goldberg 1992 ) . Effective modeling and control of personality in LLMs is important for two reasons: first since personality is a key factor in determining effective communication; second, because it is important to model interaction of synthetic users over the broad range of personality profiles to be encountered in practice. Serapio-García et al. 2023 found that instruction-tuned models displayed more reliable and externally valid personality types than pre-trained variants (perhaps due to superior instruction following), and that larger models are better equipped to express complex traits, and to simulate social behaviours. 2.2 LLMs and Synthetic Personas in Health Several works have proposed the use of LLMs to provide advice in health or wellness settings. Tu et al. 2024 introduced Articulate Medical Intelligence Explorer (AMIE), an LLM agent designed for conversational medical diagnosis. The AMIE agent achieved remarkable success, with greater diagnostic accuracy compared to primary care physicians in a randomized, double-blind crossover study. AMIE was trained using self-play with

Read the complete text