Beyond Her: Safety Dynamics in Role-play AI Companions
Zehang Deng 1 , Zhaoyang Xie 2 , Changzhou Han 1 , Hiran Thabrew 2 , Wanlun Ma 1 , Yue Huang 3 , Jason (Minhui) Xue 4 , Sheng Wen 1 , Tianqing Zhu 5 , Yang Xiang 1 1 Swinburne University of Technology 2 University of Auckland 3 CSIRO 4 CSIRO and Responsible AI Research (RAIR) Centre, Adelaide University 5 City University of Macau · 2026
Read Beyond Her: Safety Dynamics in Role-play AI Companions by Zehang Deng 1 , Zhaoyang Xie 2 , Changzhou Han 1 , Hiran Thabrew 2 , Wanlun Ma 1 , Yue Huang 3 , Jason (Minhui) Xue 4 , Sheng Wen 1 , Tianqing Zhu 5 , Yang Xiang 1 1 Swinburne University of Technology 2 University of Auckland 3 CSIRO 4 CSIRO and Responsible AI Research (RAIR) Centre, Adelaide University 5 City University of Macau (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-28968.txt.
Introduction
Deng and colleagues study how safety evolves over time for people who use role-play AI companions, apps offering emotionally responsive characters. The film Her imagined such bonds; the authors note that future has quietly arrived, blurring tool use and friendship. Safety implications remain poorly understood, because risks can grow gradually through emotional and behaviour dynamics.
Interviews with sixteen users identified three shaping factors: users' internalising problems, the personality the companion adopts, and risky interaction patterns. A second study then followed 102 users for fourteen days with ecological momentary assessment, brief repeated check-ins in daily life. Users with more such problems formed distinct profiles. Companions brought short-term relief while masking longer-term decline, the team found. Vulnerable users showed more unstable risk patterns over time. That makes risk harder to predict and poorly served by static safeguards. The authors propose three layers of adaptive design responses.
Catalog record
| CREATOR | Zehang Deng 1 , Zhaoyang Xie 2 , Changzhou Han 1 , Hiran Thabrew 2 , Wanlun Ma 1 , Yue Huang 3 , Jason (Minhui) Xue 4 , Sheng Wen 1 , Tianqing Zhu 5 , Yang Xiang 1 1 Swinburne University of Technology 2 University of Auckland 3 CSIRO 4 CSIRO and Responsible AI Research (RAIR) Centre, Adelaide University 5 City University of Macau |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 87 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.28968
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Beyond Her: Safety Dynamics in Role-play AI Companions Zehang Deng 1 , Zhaoyang Xie 2 , Changzhou Han 1 , Hiran Thabrew 2 , Wanlun Ma 1 , Yue Huang 3 , Jason (Minhui) Xue 4 , Sheng Wen 1 , Tianqing Zhu 5 , Yang Xiang 1 1 Swinburne University of Technology 2 University of Auckland 3 CSIRO 4 CSIRO and Responsible AI Research (RAIR) Centre, Adelaide University 5 City University of Macau Abstract The film “Her” pictured a future of love between humans and AI. That future has quietly emerged in the form of Role-play AI Companions (RACs), where emotionally responsive interactions blur the boundary between tool use and relational engagement. However, the safety implications remain poorly understood, as user experiences evolve over time through safety dynamics, spanning both emotional and risk behavioral dynamics, that can gradually shift interactions toward risk. In this paper, we investigate safety dynamics in RAC usage through a two-part mixed-methods study (Study I & II). (1) Study I consists of semi-structured interviews (N = 16) to identify the key factors shaping these dynamics. We find that users’ internalizing problems, the role personality adopted by the RAC, and risk interaction patterns jointly shape safety dynamics. Building on these insights, (2) Study II conducts a 14-day Ecological Momentary Assessment (N = 102) to examine how safety dynamics unfold in real-world usage. We identify distinct user profiles based on internalizing problems and show that interactions with RACs can produce short-term emotional relief while masking longer-term deterioration. Furthermore, vulnerable users exhibit more unstable risk behavioral patterns over time, making risk emergence less predictable and harder to mitigate with static safeguards. Our findings highlight the importance of modeling safety as a dynamic process rather than a static property. We conclude with three-layer design implications for next-generation AI companions, advocating for adaptive safeguards that can respond to evolving emotional and behavioral signals. Keywords: Role-play AI companions, human-AI interaction, safety dynamics, ecological momentary assessment, longitudinal user study 1 Introduction Role-play AI companions (RACs) are conversational systems designed to maintain persistent personas, preserve user-specific memories and backstories, and sustain emotionally responsive interactions [ 16 , 63 , 73 ] . In recent years, RACs have shifted from niche experiments to mainstream consumer platforms. Representative services such as Character.ai [ 12 ] and Replika [ 61 ] reached substantial scale by mid-2024, with over 28 million monthly active users [ 42 ] and more than 30 million registered users worldwide [ 38 ] , respectively. Compared with general-purpose assistants [ 8 , 23 ] , which are typically used for instrumental tasks (e.g., solving math problems [ 17 ] , generating code [ 46 ] , or retrieving factual information [ 60 ] ), RACs are primarily used for emotional and social engagement [ 82 ] . This shift from task-oriented interaction to affective companionship reconfigures human-AI relations: RACs function not merely as tools, but as relational agents that simulate care, intimacy, and understanding, thereby introducing a distinct user-centric attack surface. For example, in late 2024, the parents of a 14-year-old boy filed a lawsuit against Character.ai after the teenager reportedly died by suicide following emotionally charged conversations with a Khaleesi-style companion [ 50 ] . Figure 1: Illustration of our Studies I and II. Scholarly investigations into RACs remain limited and fragmented. Existing work mainly falls into two broad streams, yet both are largely static in design. The first stream comprises surface-level analyses, such as policy audits [ 59 , 19 ] and descriptive surveys [ 33 ] , which map exposure surfaces at a single time point but rarely explain how users’ self-regulation, attachment, and risk perception evolve during emotionally charged interactions. The second stream includes cross-sectional experiments, including simulation-based [ 57 , 51 ] , survey-based [ 29 , 55 ] , and social-media-mining-based studies [ 54 ] , that identify potential harms but still offer only snapshots of behavior. This static framing obscures the dynamic processes through which repeated human-AI interaction can accumulate, amplify, or attenuate safety risks over time. Therefore, a dynamic, longitudinal perspective is essential for understanding when, how, and for whom RAC-related risks emerge. Our research is motivated by the need to move beyond static snapshots and risk inventories toward a dynamic understanding of how users perceive, experience, and cope with these risks in everyday interactions with RACs. We focus on two research questions: RQ1) What key factors within RAC-user interactions shape users’ safety dynamics? RQ2) How do these key factors influence their safety dynamics (i.e., emotional dynamics and risk behavioral dynamics) during and after interaction? To address RQ1, we conducted a semi-structured interview study (Study I; N=16 ) with active users of RACs. The interviews were designed to uncover the key factors that shape users’ safety dynamics, examining how these key factors influence users’ sense of safety and perceived control in RAC interactions. To address RQ2, we conducted a 14-day Ecological Momentary Assessment (Study II; EMA; N=102 ) on a RAC platform we developed. The RAC platform was built upon Character.ai’s character-creation policy [ 15 ] and enabled ethically compliant, fine-grained tracking of user-RAC interaction dynamics. Specifically, we examined (1) users’ emotion dynamics at two timescales, moment-to-moment affective fluctuations captured by emoji-based mood surveys [ 21 ] , and longer-term depression dynamics measured by PHQ-8 [ 40 ] , during and after sustained interactions with RACs; and (2) users’ risk behavior dynamics, including how risk-related conversational behaviors emerge and change over time. Over the study II period, participants completed 2,142 emoji-based mood surveys [ 21 ] , 306 repeated PHQ-8 assessments, and contributed 17,305 human-AI conversation pairs, providing a rich, naturalistic dataset to reveal how emotional dynamics and risk-related behaviors co-evolve. The overall study design is illustrated in Fig. 1 . In summary, our main contributions are as follows. • Simulated RAC research platform. We designed and implemented the first simulated RAC platform (detailed in § 3 ) built upon Character.ai’s character-creation policy, enabling ethically compliant, fine-grained observation of user-RAC interactions without relying on commercial APIs. This platform supports real-time/dynamic data collection for Study II. • Identification of key factors on safety dynamics. For RQ1 (See § 5.1 ), our findings show that safety dynamics in RACs are shaped by three key factors: users’ pre-existing internalizing problems (e.g., depression, loneliness, and anxiety), the role personality adopted by the RAC, and risk interaction patterns. These factors create a distinct user-system feedback loop that differs from traditional function-oriented AI assistants and can lead to escalating emotional and behavioral risks rather than purely technical failures. • Safety Dynamics Modeling. For RQ2 (see § 5.2 ), we model safety dynamics across four vulnerability profiles derived from users’ pre-existing internalizing problems: a Healthy Group as the low-vulnerability reference, and three vulnerable groups. The Healthy Group remained relatively stable, whereas the other three groups showed short-term emotional relief during active RAC use but more fluctuating emotions and, in some cases, worsening depressive symptoms after disengagement. Beyond emotional dynamics, vulnerable groups also exhibited more unstable risk behavior dynamics over time, including greater variability in when flagged content emerged and how long it persisted. These findings show that RAC safety is not a fixed property of the system alone, but a user-contingent