Proactive Conversational Agents with Inner Thoughts Conference: CHI Conference on Human Factors in Computing Systems; April 26-May 1, 2025; Yokohama, Japan CHI Conference on Human Factors in Computing Systems (CHI ’25), April 26-May 1, 2025, Yokohama, Japan DOI: 10.1145/3706598.3713760 ISBN: 979-8-4
Xingyu Bruce Liu, Shitao Fang, Weiyan Shi, Chien-Sheng Wu, Takeo Igarashi, Xiang ‘Anthony’ Chen · 2025
Read Proactive Conversational Agents with Inner Thoughts Conference: CHI Conference on Human Factors in Computing Systems; April 26-May 1, 2025; Yokohama, Japan CHI Conference on Human Factors in Computing Systems (CHI ’25), April 26-May 1, 2025, Yokohama, Japan DOI: 10.1145/3706598.3713760 ISBN: 979-8-4 by Xingyu Bruce Liu, Shitao Fang, Weiyan Shi, Chien-Sheng Wu, Takeo Igarashi, Xiang ‘Anthony’ Chen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-00383.txt.
Introduction
Xingyu Bruce Liu and co-authors design chat agents that can start talking on their own, in work published in 2024. Most agents only react to cues about whose turn it is, which is limiting in group chats. The authors argue that people act from inner thoughts, forming ideas as a chat unfolds and speaking when the moment feels right.
Their framework gives an agent a quiet, ongoing train of thoughts that runs beside the open chat. A study with twenty-four people helped shape the design, which draws on language and mind research. Two live systems were built and tested with users. The new agents beat standard ones on how humanlike, coherent, and well-timed their turns felt to participants.
Catalog record
| CREATOR | Xingyu Bruce Liu, Shitao Fang, Weiyan Shi, Chien-Sheng Wu, Takeo Igarashi, Xiang ‘Anthony’ Chen |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 90 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-00383
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Proactive Conversational Agents with Inner Thoughts Conference: CHI Conference on Human Factors in Computing Systems; April 26-May 1, 2025; Yokohama, Japan CHI Conference on Human Factors in Computing Systems (CHI ’25), April 26-May 1, 2025, Yokohama, Japan DOI: 10.1145/3706598.3713760 ISBN: 979-8-4007-1394-1/25/04 CCS: Human-centered computing HCI theory, concepts and models CCS: Computing methodologies Discourse, dialogue and pragmatics CCS: Computing methodologies Cognitive science Xingyu Bruce Liu, Shitao Fang, Weiyan Shi, Chien-Sheng Wu, Takeo Igarashi, Xiang ‘Anthony’ Chen Abstract . One of the long-standing aspirations in conversational AI is to allow them to autonomously take initiatives in conversations, i.e., being proactive . This is especially challenging for multi-party conversations. Prior NLP research focused mainly on predicting the next speaker from contexts like preceding conversations. In this paper, we demonstrate the limitations of such methods and rethink what it means for AI to be proactive in multi-party, human-AI conversations. We propose that just like humans, rather than merely reacting to turn-taking cues, a proactive AI formulates its own inner thoughts during a conversation, and seeks the right moment to contribute. Through a formative study with 24 participants and inspiration from linguistics and cognitive psychology, we introduce the Inner Thoughts framework. Our framework equips AI with a continuous, covert train of thoughts in parallel to the overt communication process, which enables it to proactively engage by modeling its intrinsic motivation to express these thoughts. We instantiated this framework into two real-time systems: an AI playground web app and a chatbot. Through a technical evaluation and user studies with human participants, our framework significantly surpasses existing baselines on aspects like anthropomorphism, coherence, intelligence, and turn-taking appropriateness. Figure 1. A comparison of three types of conversational agents with different proactivity strategies. (1) Reactive Conversational Agents: AI only responds when addressed. (2) Next-Speaker Prediction: AI predicts who will speak next based on contextual cues such as previous utterances. However, it overlooks the agents’ intrinsic thought processes. This strategy fails particularly when no explicit turn is allocated, and often leads to incoherent contributions. (3) Conversational Agents with Inner Thoughts (ours): AI generates a train of thoughts and evaluates them based on their intrinsic motivation to participate. A comparison of three types of conversational agents with different proactivity strategies. (1) Reactive Conversational Agents: AI only responds when explicitly addressed. (2) Next-Speaker Prediction: AI predicts who will speak based on previous utterances. However, this strategy struggles in multi-party conversations particularly in cases of self-selection where no explicit turn is allocated. This often leads to irrelevant or incoherent contributions. (3) Conversational Agents with Inner Thoughts (ours): AI generates a train of thoughts (e.g., “I’m a yoga instructor”) and evaluates them based on intrinsic motivation to participate. The AI’s decision to engage is guided by the motivation score of each thought. Here, the AI adds to the discussion by sharing its experience as a yoga instructor since it is highly relevant. † † cc-license: by 1. Introduction Recent advances in Large Language Models (LLMs) have demonstrated their ability to generate high-quality text in response to human input, finding application in areas ranging from Q&A systems to writing assistants. Yet, most current LLM-based systems treat AI as passive respondents, responding only to explicit human prompts. Imagine a scenario where people are planning a trip with an AI agent: they must constantly prompt the AI, which passively waits for instructions instead of actively contributing. On the other end of the spectrum, systems like GitHub Copilot 1 1 1 https://github.com/features/copilot tend to overcompensate, offering constant suggestions that can overwhelm users. Neither extreme — AI that is only reactive nor AI that is always responding — is ideal. In the context of conversations, a proactive AI agent should be able to autonomously participate in socially appropriate moments, providing relevant input without requiring explicit cues. This is particularly challenging in multi-party conversations. Dyadic human-AI interactions ( e.g., using Siri) often predict turn-taking based on speech features such as pause or stop words, and the next turn will be automatically allocated to the other party ( Skantze 2021 ; Chang et al. 2022 ) . However, in multi-party settings, these cues could be ambiguous, and multiple possible speakers may take the floor. Repeatedly prompting AI during group interactions can also become cumbersome and can disrupt the natural flow of the conversation, as illustrated in the example of trip planning. Previous systems typically first predict the next speaker ( i.e., turn-taking prediction) and then generate the next response based on conversational and contextual information. For instance, some approaches rely on the last few turns of conversations to predict the subsequent speaker ( Wei et al. 2023 ; de Bayser et al. 2019 ; Ekstedt and Skantze 2020 ) , while others utilize multimodal cues, such as eye gaze and non-verbal signals ( Bohus and Horvitz 2009a ; Bohus and Horvitz 2009b ; Bohus and Horvitz 2011 ) . Despite these efforts, on turn-taking prediction, they still fall short and struggle to beat the simple “repeat last” baseline strategy in social conversation contexts ( Wei et al. 2023 ; de Bayser et al. 2019 ) . Our formative evaluation ( Table 1 ) also shows that when it comes to predicting the next speaker, fine-tuned LLMs perform no better than random guessing unless the next speaker is allocated ( e.g., “What do you think, Alice?” ). In addition, after determining the next speaker, existing works tend to use predefined speaker personas ( Zhang 2018 ; Yamashita et al. 2023 ) as additional input to guide response generation, or expand persona with commonsense ( Kim et al. 2023 ) . However, these additional inputs and profiles are fixed and static during conversations, instead of changing through time as humans did. We suggest an alternative and reversed perspective to think about AI proactivity: Consider how humans chat about what we did over the weekend. As we listen to others speak, we process their words, reflect on our experiences, and develop an internal train of thoughts — cognitive psychologists highlight this as the distinction between covert responses (internal thoughts and feelings) and overt responses (verbal utterances or gestures) ( Ekman and Friesen 1969 ; Ruesch et al. 2017 ) in the human communication process. Then, at some point, we may feel a strong urge to share our thoughts. This might happen when we seek clarification or when someone mentions an activity we also participated in, sparking our desire to contribute. With this intention in mind, we then look for a socially appropriate moment to participate. In this paper, we propose a new approach to proactive AI in the context of multi-party, text-based conversations: rather than simply predicting conversational turns, we explore proactive AI driven by its own internal “thoughts”. We introduce the Inner Thoughts framework. Inspired by cognitive architectures and LLM prompting techniques, this framework comprises five stages: trigger, retrieval, thought formation, evaluation, and participation , which enable AI to continuously generate a train of thoughts in parallel with an ongoing conversation, utilizing both long-term and working memory. The AI participant then determines whether to engage in the conversation based on an evaluation of its intrinsic motivation to express a particular thought at that moment. To model intrinsic motivation, we conducted a think-aloud study