τ \tau -Knowledge: Evaluating Conversational Agents over Unstructured Knowledge
Quan Shi, Alexandra Zytek, Pedram Razavi, Karthik Narasimhan, Victor Barres · 2026
Read τ \tau -Knowledge: Evaluating Conversational Agents over Unstructured Knowledge by Quan Shi, Alexandra Zytek, Pedram Razavi, Karthik Narasimhan, Victor Barres (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-04370.txt.
Introduction
Shi and colleagues test agents that must use unstructured knowledge live with users. Current benchmarks split retrieval from tool use, missing fully agentic long horizon evaluation. Tau-Knowledge extends tau-Bench, joining external knowledge with tool outputs for verifiable policy compliant state changes. Tau-Banking models fintech support across about 700 linked documents plus account updates by tools. Even frontier models with large reasoning budgets pass only about 25.5 percent at first try, and reliability drops sharply on repeats.
Catalog record
| CREATOR | Quan Shi, Alexandra Zytek, Pedram Razavi, Karthik Narasimhan, Victor Barres |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 94889 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.04370
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
τ \tau -Knowledge: Evaluating Conversational Agents over Unstructured Knowledge Quan Shi, Alexandra Zytek, Pedram Razavi, Karthik Narasimhan, Victor Barres Abstract Conversational agents are increasingly deployed in knowledge-intensive settings, where correct behavior depends on retrieving and applying domain-specific knowledge from large, proprietary, and unstructured corpora during live interactions with users. Yet most existing benchmarks evaluate retrieval or tool use independently of each other, creating a gap in realistic, fully agentic evaluation over unstructured data in long-horizon interactions. We introduce τ \tau -Knowledge , an extension of τ \tau -Bench for evaluating agents in environments where success depends on coordinating external, natural-language knowledge with tool outputs to produce verifiable, policy-compliant state changes. Our new domain, τ \tau -Banking , models realistic fintech customer support workflows in which agents must navigate roughly 700 interconnected knowledge documents while executing tool-mediated account updates. Across embedding-based retrieval and terminal-based search, even frontier models with high reasoning budgets achieve only ∼ \sim 25.5% passˆ1, with reliability degrading sharply over repeated trials. Agents struggle to retrieve the correct documents from densely interlinked knowledge bases and to reason accurately over complex internal policies. Overall, τ \tau -Knowledge provides a realistic testbed for developing agents that integrate unstructured knowledge in human-facing deployments. † † affiliationnotice: Equal contribution Code 1 Introduction Figure 1: Overview of the \tau -Banking domain. Agents must interact with a knowledge base to acquire procedural knowledge, policies, tools, and business offerings in order to resolve complex user requests by invoking discovered tools that modify underlying databases. The example on the right illustrates an agent assisting a user who has lost a wallet containing bank cards: although the user initially requests to freeze the card, card-specific policies and transaction-history constraints require the agent to instead cancel the card. Today, language agents are frequently deployed in settings that require interactions with private, custom, and unstructured knowledge bases (KBs). In such deployments, agents must ground their actions in these knowledge sources, respond to human users, and discover internal capabilities and the rules and constraints that govern their actions ( 36 ; 39 ) . Despite the prevalence of this setting in the real world, existing evaluation benchmarks for language agents rarely capture this complexity. Most prior work evaluates either retrieval in isolation, typically for question answering or web search ( 33 ; 22 ; 31 ) , or tool use in isolation ( 40 ; 14 ; 9 ) , without requiring agents to reason over private knowledge bases containing out-of-distribution product terms, detailed policies, and discoverable capabilities. Additionally, retrieval in these settings typically occurs during live interactions with users, which introduces significant stochasticity in the form of underspecified goals, ambiguous queries, and evolving user intent. To address this gap, we introduce \tau -Knowledge, an extension of \tau -Bench ( 40 ) that evaluates agents in knowledge-grounded environments through a new domain, \tau -Banking. \tau -Banking is a new fintech-based domain in which task success depends on finding and applying information from a natural-language knowledge base of roughly 700 documents covering product details, procedural policies, and tool documentation. To construct this corpus at scale while preserving internal consistency, we use a structured-to-unstructured generation pipeline that expands a latent structured specification of products, policies, and tools into natural-language documents. The tasks mirror realistic customer support flows, such as opening and closing accounts, redeeming referral promotions, and handling direct-deposit inquiries, while requiring agents to coordinate knowledge-base evidence with tool outputs over long-horizon conversations. Importantly, as in real-world deployments, capabilities are not fully observed by the agent — tools are referenced only in documentation and must be found to be used. This design ensures that access to state-changing operations is contingent on correct knowledge retrieval. We build \tau -Knowledge to be agnostic of the retrieval mechanism, enabling evaluation across diverse search strategies. The benchmark supports and evaluates arbitrary strategies for searching and interacting with a corpus, including dense and sparse retrieval, long-context processing, filesystem-based exploration, and hybrid approaches. This flexibility allows \tau -Knowledge to evaluate emerging paradigms beyond semantic retrieval, such as terminal-based navigation through unstructured documents. Our results show that \tau -Knowledge effectively exposes realistic bottlenecks in knowledge-grounded agent performance. Across all tested frontier agent models and retrieval configurations, the best observed result achieves only 25.52% passˆ1 (GPT-5.2, high reasoning), where passˆk denotes the probability that a task is successfully completed in each of k independent trials. Reliability degrades sharply, dropping to at most 13.40% passˆ4. Notably, performance remains low even when retrieval is removed as a bottleneck via a golden-retriever setting, in which agents are given the task-critical documents directly in context. The strongest model in this configuration achieves only 39.69% passˆ1 (Claude-4.5-Opus, high reasoning). This gap demonstrates that \tau -Knowledge cannot be solved by retrieval alone, and requires agents to reason over complex policies, cross-document dependencies, and an evolving database state. We also find systematic differences between frontier models and retrieval configurations in performance and efficiency. For example, GPT-5.2 (high) with terminal-based search achieves performance comparable to Claude-4.5-Opus (high), but requires approximately 1.7 \times more tokens, executes roughly 2.3 \times more shell commands, and takes roughly 9 \times longer to complete tasks. One common cause is brittle, assumption-driven search behavior when human intent is underspecified. Rather than resolving ambiguity through clarification or targeted retrieval, some models tend to overcommit to early hypotheses or oscillate between unfocused searches, leading to excessive interaction turns and tool calls. Retrieval configuration further amplifies these differences. For example, while freeform terminal-based search can improve performance for strong reasoning models relative to embedding-based semantic search, it typically requires substantially more search steps and tool interactions. In contrast, dense retrieval often yields faster agent turns and fewer searches. These tradeoffs show that weaker or noisier retrieval is often compensated for by increased search and tool usage, preserving success rates at the cost of higher latency and lower interaction efficiency. These inefficiencies matter greatly in online, human-facing deployments, as extra turns translate into longer resolution times, higher cognitive load, and reduced trust, especially for time-sensitive support scenarios. Therefore, progress on human-facing agents should be measured not only by final task success but also by solution efficiency : the ability to reach correct, policy-compliant outcomes with minimal time and tool calls, and minimal conversational backtracking. \tau -Knowledge provides a controlled testbed for studying how search, reasoning, and efficiency interact in knowledge-grounded agents, revealing substantial gaps in existing systems. 2 Related Work Benchmarks for Agents and Tool Use A number of benchmarks evaluate an agent’s ability to decompose tasks into multi-step plans, invoke external tools, and execute structured procedures to achieve a predefined objective ( 10 ; 29 ; 38 ; 9 ; 37 ; 14 ; 21 ) . However, these benchmarks typically assume a fully specified tool interface and evaluate agents operating in isolation, without explicit modeling of interactive users or conversational dynamics. \tau -Bench ( 40 ; 3 ) addresses these limitations by introducing goal-oriented, partially observable conversational environments in which agents must interact with simulated users over extended horizons. Nevertheless, in these settings, the available tools and procedures are still largely provided to the agent a priori . \tau -Knowledge builds directly on this foundation by requiring agents to acquire procedural knowledge through retrieval from a