Substance over Style: Evaluating Proactive Conversational Coaching Agents

Vidya Srinivas, Xuhai Xu, Xin Liu, Kumar Ayush, Isaac Galatzer-Levy, Shwetak Patel, Daniel McDuff, Tim Althoff · 2025

Read Substance over Style: Evaluating Proactive Conversational Coaching Agents by Vidya Srinivas, Xuhai Xu, Xin Liu, Kumar Ayush, Isaac Galatzer-Levy, Shwetak Patel, Daniel McDuff, Tim Althoff (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-19328.txt.

Introduction

Chatbots that coach people face a hard test: goals start vague and shift mid-talk, and praise or blame stays personal. Vidya Srinivas and co-authors built five coaching bots with distinct styles and asked users how they landed, in work from twenty twenty-five. The study drew first-person notes from more than one hundred fifty talks.

Users prized core help most; style without substance drew cold reviews. Expert raters and a model judge scored the same talks in other ways, far from the users' view. The clash shows that who grades a coach bot can flip the verdict. Design teams, the authors argue, should grade on the coached person's word first.

Catalog record

CREATORVidya Srinivas, Xuhai Xu, Xin Liu, Kumar Ayush, Isaac Galatzer-Levy, Shwetak Patel, Daniel McDuff, Tim Althoff
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE79 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-19328

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Substance over Style: Evaluating Proactive Conversational Coaching Agents Vidya Srinivas, Xuhai Xu, Xin Liu, Kumar Ayush, Isaac Galatzer-Levy, Shwetak Patel, Daniel McDuff, Tim Althoff Abstract While NLP research has made strides in conversational tasks, many approaches focus on single-turn responses with well-defined objectives or evaluation criteria. In contrast, coaching presents unique challenges with initially undefined goals that evolve through multi-turn interactions, subjective evaluation criteria, and mixed-initiative dialogue. In this work, we describe and implement five multi-turn coaching agents that exhibit distinct conversational styles, and evaluate them through a user study, collecting first-person feedback on 155 conversations. We find that users highly value core functionality, and that stylistic components in absence of core components are viewed negatively. By comparing user feedback with third-person evaluations from health experts and an LM, we reveal significant misalignment across evaluation approaches. Our findings provide insights into design and evaluation of conversational coaching agents and contribute toward improving human-centered NLP applications. 1 Introduction Figure 1: Example Coaching Conversation. Different coaching behaviors are present throughout the conversation (highlighted in red) based on user responses, corresponding to recommended coaching behaviors (see Section 2 ). Full transcripts in Section A.7 . Recent NLP research has focused on evaluating tasks in conversational settings, including proactive information-seeking Li et al. (2024) , multi-turn conversational dialogue Yi et al. (2024) ; Kwan et al. (2024) ; Bai et al. (2024) , and others. While significant progress has been made, the focus has largely been on exercises that represent one or multiple of the following properties: a single, unchanging, clearly communicated goal, a single correct answer, a high-quality answer expressed in a single turn or utterance, an assumption that objective quality can be evaluated by third parties (e.g., experts or language models), or a clear interaction structure (e.g., all user or model-initiated). However, proactive human-centered NLP applications such as personalized target-guided dialogues Deng et al. (2024) often exhibit several, concurrent, contrastive challenges. This work focuses on coaching conversations, which differ from the aforementioned tasks with the presence of: 1) open-ended multi-turn interactions, 2) initially ambiguous task definition that must be informed through multiple user interactions, 3) multiple and changing goals that must be refined and prioritized, 4) potentially off-topic user input throughout long-form conversation, 5) varied conversational styles and preferences, 6) mixed-initiative interactions, in which a coach must balance addressing user goals, needs and empowerment, 7) unspoken user needs and incomplete context, and the 8) absence of a single, objectively correct answer (instead relying primarily on subjective first-person evaluation). As a result, coaching remains difficult and underexplored for proactive agents. In this work, we aim to 1) determine essential components of LM agent coaching, 2) describe and evaluate LM coaching agents that express distinct coaching styles (from common directive and interrogative to facilitative styles informed by human coaching experts), and 3) better understand agreement or discrepancies between first and third person evaluations of such conversations to inform future approaches. We use health coaching as an illustrative example to frame the coaching domain. Through our study, we find that questioning-only conversation styles typically reduce user engagement and satisfaction. We also find that users place high value on core functional components, which we refer to as substance, and perceive stylistic elements negatively when this substance is lacking. Finally, we find that LMs perform poorly as auto-raters for human-centered subjective tasks, but demonstrate reasonable agreement with users for objective human-centered tasks. Our contributions are as follows: • We synthesize the expertise and recommendations of health coaching experts ( N=11 ) across a variety of coaching domains to define key capabilities for open-ended coaching. • We identify LM coaching characteristics and create five conversational multi-turn coaching agents that incorporate different levels of these characteristics to embody directive, interrogative, and facilitative coaching styles • We conduct a user study ( N=31 , 50 person hours of feedback, 155 conversations) where participants engage in unsupervised discussions on self-directed, open-ended coaching topics with these agents. We collect first-person feedback. • We examine the alignment between first-person feedback and third-person annotations from health experts and an LM, highlighting shortcomings of evaluation approaches in this setting. 2 Initial Health Expert Interviews Health coaches provide general guidance to a user who is looking to pursue a health goal. To inform the design of our coaching agents across general coaching domains, we conducted a formative study with 11 coaches across domains of sleep, fitness, and life coaching. These coaches, or experts, possessed advanced degrees in their respective fields, coupled with substantial professional experience ranging from 4 to 46 years. The one-hour interview consisted of two parts––open-ended questions through which experts shared their expertise on coaching best-practices, and a case study, in which experts reviewed, turn-by-turn, a conversation between a real end-user and a single-prompt LM. 2.1 Health Expert Insights Based on our interviews, we identified conversational expert insights which we classified into two categories: style and substance . Style refers to how something is presented—-its form or delivery. This includes language, tone, and overall impression. Substance refers to core components of a helpful and actionable coaching experience, including exploration, context gathering, goal-seeking, and relevant recommendation. • I1 Goal and purpose understanding ( substance ): Understanding goals and motivations is important. Coaches maintain a goal-oriented conversation and bring clients back to the main goal when they deviate off-topic. • I2 Context clarification ( substance ): Coaches collect and understand clients’ constraints, preferences, and prior resolution attempts to better personalize recommendations. • I3 Relevant recommendations ( substance ): Coaches provide recommendations that are relevant, actionable, and context-sensitive. • I4 Feedback seeking ( substance ): Coaches solicit user feedback and update recommendations as necessary based on this feedback. • I5 Active listening ( style ): Coaches provide occasional reflections to ensure correct understanding and goal alignment promoting client engagement and trust. • I6 Client empowerment ( style ): Coaches build trust with their clients and guide them to navigate their own paths when exploring potential solutions. Coaches are supportive when clients are moving in a good direction. 2.2 Turn-by-Turn Expert Evaluation The single prompt used to generate example conversations for the turn-by-turn part of the health expert interviews contained both control flow decisions (i.e. when to change topic, when to ask questions, etc.) and coaching instructions. From expert annotations, we identified cases of premature recommendation and fixated questioning, in which the LM either made recommendations before seeking relevant information or continued down a single, narrow line of questioning without considering additional factors (expert quotes available in Section A.1 ). These health expert interviews revealed consistent, significant shortcomings of static LM prompting approaches. These issues, along with the expert insights, informed our designs of conversational flow ( Section 4.2 ). 3 Conversational Paradigms Before developing a coaching agent that embodies health expert insights, we recognize two conversational paradigms that already exist between humans and LMs, and point out how expert insights emphasize a third, distinct form of conversation . Interrogative interaction occurs when either a human or an LM provides one-sided questioning in a conversation while the opposite party only provides answers; a recommendation is potentially made at the end of the conversation by the questioning party. Such an approach maximizes knowledge gain but minimizes interaction. Questionnaires have

Read the complete text