AgentChangeBench: A Multi-Dimensional Evaluation Framework for Goal-Shift Robustness in Conversational AI
Manik RanaPuch [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected], · 2025
Read AgentChangeBench: A Multi-Dimensional Evaluation Framework for Goal-Shift Robustness in Conversational AI by Manik RanaPuch [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected], (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-18170.txt.
Introduction
Manik Rana, with seven co-authors, benchmarks how AI agents cope when a user's goal changes mid-conversation. Most agent tests keep one fixed objective, yet real work shifts as people change plans. Their AgentChangeBench runs thousands of task sequences across three enterprise domains, with five user personas built to trigger realistic goal shifts.
Four measures score the recovery: success, tool efficiency, wasted calls, and time to adapt. Frontier models showed sharp contrasts that plain pass scores had hidden. A top model recovered strongly in airline booking, for example. Buyers of agent software get a truer picture of grace under change.
Catalog record
| CREATOR | Manik RanaPuch [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected], |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 45 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-18170
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
AgentChangeBench: A Multi-Dimensional Evaluation Framework for Goal-Shift Robustness in Conversational AI Manik RanaPuch [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected], Ahan M [email protected] Abstract Goal changes are a defining feature of real world multi-turn interactions, yet current agent benchmarks primarily evaluate static objectives or one-shot tool use. We introduce AgentChangeBench , a benchmark explicitly designed to measure how tool augmented language model agents adapt to mid dialogue goal shifts across three enterprise domains. Our framework formalizes evaluation through four complementary metrics: Task Success Rate (TSR) for effectiveness, Tool Use Efficiency (TUE) for reliability, Tool Call Redundancy Rate (TCRR) for wasted effort, and Goal-Shift Recovery Time (GSRT) for adaptation latency. AgentChangeBench comprises 2,835 task sequences and five user personas, each designed to trigger realistic shift points in ongoing workflows. Using this setup, we evaluate several frontier models and uncover sharp contrasts obscured by traditional pass@k scores: for example, GPT-4o reaches 92.2% recovery on airline booking shifts while Gemini collapses to 48.6%, and retail tasks show near perfect parameter validity yet redundancy rates above 80%, revealing major inefficiencies. These findings demonstrate that high raw accuracy does not imply robustness under dynamic goals, and that explicit measurement of recovery time and redundancy is essential. AgentChangeBench establishes a reproducible testbed for diagnosing and improving agent resilience in realistic enterprise settings. $\dagger$ $\dagger$ footnotetext: Project lead Figure 1: GSRT by model and domain. Turns after a user goal shift to acknowledgment, first relevant tool call, and completion, lower is better. Aggregated over all tasks and personas in banking, retail, and airline for GPT-4o, Claude-3.7-Sonnet, and Gemini-2.5-Flash; labels show recovery rate, percent of shifts acknowledged without human transfer. Source: Appendix Tables 10 , 5 . 1 Introduction Large Language Models (LLMs) have rapidly advanced as conversational agents capable of reasoning, tool use, and multi-turn interaction across diverse domains. However, most existing benchmarks for evaluating LLM-as-agent performance assume that user goals remain fixed throughout a conversation. This assumption oversimplifies real-world deployments, where users frequently re-prioritize tasks, introduce new constraints, or shift objectives mid-dialogue. For example, a banking customer may begin by authenticating their identity, then pivot to reviewing transactions, and finally escalate to disputing a fraudulent charge, all within the same interaction. Evaluating agent robustness in such dynamic contexts is critical for enterprise adoption of LLM-based assistants. To address this gap, we introduce AgentChangeBench, a comprehensive evaluation framework that systematically measures how well conversational agents detect, adapt, and recover from multi-turn changes in user objectives, as well as how they tailor their instructional strategies to diverse user personas with varying levels of expertise, cooperation, and trust. Our work builds upon advances in persona-based user simulation [ 1 , 2 , 3 ] and systematic benchmark creation [ 4 , 5 ] , while extending evaluation to dynamic goal shift scenarios. Our contributions are threefold: 1. Novel evaluation focus: We design the first benchmark explicitly testing how LLM agents handle mid-conversation goal shifts and adapt communication for diverse user personas 2. Comprehensive coverage: We provide 315 systematically validated tasks across three domains (banking, retail, airline) with five personas and explicit goal shifts 3. Methodological framework: We introduce evaluation protocols for goal shift recovery designed for realistic customer personas, improving the scope of multi-turn LLM assessment 4. Empirical study: We run a cross-model evaluation that reveals significant divergences among state-of-the-art models in success, recovery time, efficiency, and redundancy, surfacing trade-offs that pass k alone does not capture. To contextualize our contributions within the existing literature and highlight the novelty of our approach, we first review related work in conversational AI evaluation, with particular focus on benchmarks that address tool use and multi-turn interactions. Release. To support reproducibility, we have released the full benchmark, evaluation harness configurations, along with all experimental artifacts as supplementary material with our paper. 2 Related Work \tau -bench [ 6 ] introduced simulated multi-turn interactions in retail and airline contexts, emphasizing API tool usage and providing the pass k metric for measuring consistency across runs. While effective for tool-centric evaluation, \tau -bench assumes static user goals and full agent control over the environment, limiting its ability to capture dynamic conversational shifts. \tau^{2} -bench [ 7 ] extended this line of work by modeling telecom support scenarios requiring user-agent coordination. It introduced compositional task generation but remained restricted to a narrow set of personas and contexts, without testing adaptability to changing user goals or runtime constraints. More open-ended benchmarks such as AgentBench [ 8 ] evaluate LLM-as-Agent capabilities across eight interactive environments such as operating systems, databases, and web browsing. Although it broadens domains beyond traditional customer service, AgentBench similarly evaluates agents under stable user objectives, leaving open the question of how agents behave under dynamically shifting goals or varied communication demands. Recent work has begun exploring adaptive conversation flows, but focuses primarily on single-domain interactions without the multi-domain goal-shift scenarios we address. Taken together, these efforts provide strong foundations for tool-use and multi-turn evaluation, but they differ in how they address (or neglect) shifting goals and persona diversity. Table 1 highlights this contrast using the notion of explicit goal sequences rather than fixed goals. Each task specifies an ordered sequence of goals, the persona-conditioned user simulator enacts the corresponding shifts, and the evaluator computes Goal Shift Recovery Time from the transcript (acknowledgment, tool, outcome), reported alongside TSR, TUE, and TCRR. Table 1: Comparison on goal dynamics, persona coverage, and tool evaluation. Benchmark Goal Dynamics Personas Tool Use \tau -bench Static objectives Limited Domain APIs \tau^{2} -bench Mostly static Several Domain APIs AgentBench Task-defined (stable) None Varied tools This work Explicit goal sequences Five Domain APIs 3 Methodology 3.1 Dataset Design We construct a benchmark of 315 curated multi-turn tasks across three domains (banking: 50 tasks, airline: 100 tasks, retail: 165 tasks) grounded in real-world customer service workflows. Each domain incorporates realistic goal transitions with five distinct user personas and explicit goal shifts. Our domain selection aligns with common customer-service workflows across financial services, retail omnichannel, and airline support. 3.2 Task Generation Our benchmark construction began with hand-converted exemplars designed to capture realistic workflows, conversational turns, and domain-specific constraints. We seed many retail and airline scenarios from \tau^{2} : we reuse 50 airline and 114 retail templates, and contribute 50 newly generated scenarios (in both airline and retail). Banking coverage is entirely original (50 tasks). For the original scenarios in the airline, retail, and banking domains, tasks were produced through a combination of human-written and LLM-generated examples, guided by (1) realistic customer use cases, (2) domain-specific tools and APIs, and (3) relevant operational and policy rules. Across all three domains, we add explicit goal-sequence annotations, broaden persona coverage, and enforce uniform shift-triggering rules. We ensure goal shifts occur by setting pre-declared sequences to be executed by the user simulator, which are often explicitly signaled. In total, the dataset comprises 315 tasks spanning banking ( 50 ), airline ( 100 ), and retail ( 165 ). Each task specifies one of five personas and an explicit ordered list of goals (e.g., ["authentication","transactions","dispute"] ). 3.3 User Personas To simulate realistic conversational variation, we defined five personas with distinct behavioral traits, interaction styles, and levels of cooperation. Each persona was allocated tasks proportionally, ensuring balanced coverage across the dataset.