Arming Data Agents with Tribal Knowledge

Shubham Agarwal † , Asim Biswal † , Sepanta Zeighami † Alvin Cheung, Joseph Gonzalez, Aditya G. Parameswaran · 2026

Read Arming Data Agents with Tribal Knowledge by Shubham Agarwal † , Asim Biswal † , Sepanta Zeighami † Alvin Cheung, Joseph Gonzalez, Aditya G. Parameswaran (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-13521.txt.

Introduction

Asking a database in plain language should be easy. But agents translating language to SQL still stumble on large real databases: they miss what columns intend and form wrong ideas about the data. Prior fixes generate facts about the database, yet merely restating contents does not correct an agent's misconceptions. Tk-Boost takes a different tack, learning tribal knowledge from the agent's own mistakes.

The framework first has the agent answer a few queries, diagnoses its misconceptions from its errors, and writes knowledge that addresses them. Items are indexed by applicability conditions stating which query features each item suits. When new queries arrive, that knowledge feeds back during SQL generation. Accuracy lifts up to 16.9 percent on Spider 2.0 and 13.7 percent on BIRD. Shubham Agarwal leads the work with five co-authors.

Catalog record

CREATORShubham Agarwal † , Asim Biswal † , Sepanta Zeighami † Alvin Cheung, Joseph Gonzalez, Aditya G. Parameswaran
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE71 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-13521

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Arming Data Agents with Tribal Knowledge Thanks: † Equal contribution in alphabetical order Shubham Agarwal † , Asim Biswal † , Sepanta Zeighami † Alvin Cheung, Joseph Gonzalez, Aditya G. Parameswaran Abstract . Natural language to SQL (NL2SQL) translation enables non-expert users to query relational databases through natural language. Recently, NL2SQL agents, powered by the reasoning capabilities of Large Language Models (LLMs), have significantly advanced NL2SQL translation. Nonetheless, NL2SQL agents still make mistakes when faced with large-scale real-world databases because they lack knowledge of how to correctly leverage the underlying data (e.g., knowledge about the intent of each column) and form misconceptions about the data when querying it, leading to errors. Prior work has studied generating facts about the database to provide more context to NL2SQL agents, but such approaches simply restate database contents without addressing the agent’s misconceptions. In this paper, we propose Tk-Boost , a bolt-on framework for augmenting any NL2SQL agent with tribal knowledge : knowledge that corrects the agent’s misconceptions in querying the database accumulated through experience using the database. To accumulate experience, Tk-Boost first asks the NL2SQL agent to answer a few queries on the database, identifies the agent’s misconceptions by analyzing its mistakes on the database, and generates tribal knowledge to address them. To enable accurate retrieval, Tk-Boost indexes this knowledge with applicability conditions that specify the query features for which the knowledge is useful. When answering new queries, Tk-Boost uses this knowledge to provide feedback to the NL2SQL agent, resolving the agent’s misconceptions during SQL generation, and thus improving the agent’s accuracy. Extensive experiments across the BIRD and Spider 2.0 benchmarks with various NL2SQL agents shows Tk-Boost improves NL2SQL agents accuracy by up to 16.9% on Spider 2.0 and 13.7% on BIRD . 1. Introduction Natural language (NL) interfaces to databases enable users without SQL expertise to query data in relational databases using natural language. In recent years, the proficiency of Large Language Models (LLMs) in writing SQL, coupled with tool-calling capabilities ( Hong et al. 2025 ; Yao et al. 2023 ) has led to significant breakthroughs in accurate NL2SQL translation ( Liu et al. 2025b ; Deng et al. [n.d.] ; Li et al. 2024b ; Schmidt et al. 2025 ) . These NL2SQL agents accept NL queries and return SQL, and vary across prompting strategies, provided tools (e.g., database or file system access), and workflows they are embedded in (e.g., existence of validation loops or consistency mechanisms ( Li et al. 2024b ; Ren et al. 2024 ; Sun et al. 2024 ) ). Limitations of NL2SQL Agents . Despite significant improvements in recent years, agents still make mistakes in large real-world databases, with many columns, tables, and rows. This is due to gaps in the agent’s knowledge about databases not seen during training set—such as its contents, how it was created and its intended use. Such knowledge gaps, especially about subtle data idiosyncrasies, cause logical errors in translation ( Liu et al. 2025c ; Chung et al. 2025 ) , where agents use incorrect data sources (tables or columns) or perform logically incorrect operations on them. We refer to these recurring logical errors in performing queries caused by the agent’s knowledge gaps as misconceptions . For example, multiple tables or columns with similar semantics are common in data lakes, where derived tables with modified columns (e.g., NULLs removed through imputation), are stored alongside the originals, causing misconceptions when deciding which column to use. We use such a scenario as our running example: a table containing product sales information with columns name and brand , with brand inconsistently populated (perhaps because it was added later on), while name reliably contains both the product name and brand, with the latter as a prefix (e.g., “Nike Air Max”). An agent using brand for queries such as “find average sales by brand” produces incorrect SQL, with errors recurring for any query that requires product brand information. Prior work has tried to fix misconceptions by proactively identifying "facts", typically by asking LLMs to generate them about the database content ( Zhang et al. 2025 ; Balaka et al. 2025 ; Luoma and Kumar 2025 ; Zhang et al. 2023b ; Baek et al. 2025a ) . However, doing so in the absence of queries simply restates this content without fixing the misconceptions that arise when answering queries. In our example, one fact may be that both name and brand columns contain product brands, but this information does not help the agent infer that name should be used instead of the brand column. Agents with Tribal Knowledge . We propose Tk-Boost , a framework for correcting NL2SQL agents’ misconceptions. Our key insight is that an NL2SQL agent should accumulate knowledge of how to correctly query a database through experience , rather than relying on facts derived from the data—this process is analogous to how data analysts or engineers accumulate knowledge over time through exposure to the database. A simple approach to accumulate experience is to store the agent’s interactions with the database (i.e., queries and answers)—or their summaries—and retrieve relevant ones to include in the agent’s prompts for future queries via a form of RAG (retrieval augmented generation) using LLM memory stores ( Shinn et al. 2023 ; Xu et al. 2025 ; Li et al. 2025b ; Li et al. 2025a ; Talaei et al. 2024 ; Mem0 [n.d.] ) such as Mem0 ( Mem0 [n.d.] ) . However, as we show, directly including past interactions does not correct misconceptions as past mistakes end up being repeated. Instead, Tk-Boost augments agents with what we call tribal knowledge : natural language statements that correct the agent’s misconceptions and can be reused to correctly answer related future queries (e.g., stating “use name column to find product brands instead of brand ” in our running example). To enable an agent to accumulate tribal knowledge through experience, Tk-Boost uses a few labeled query examples (i.e., NL queries with ground-truth SQL) and asks the agent to answer those queries on the database. Tk-Boost then discovers tribal knowledge from its experience and uses the knowledge to improve the agent’s accuracy for future queries. Operationalizing these two steps is hard as it requires identifying misconceptions, generating reusable knowledge that fixes them, and using it correctly for future queries, as we discuss next. Figure 1 . Example workflow of Tk-Boost . Past experience is used to find agent misconceptions and generate corrective knowledge. The knowledge is paired with applicability conditions to help retrieval for future queries, and used to correct future agent mistakes. Tribal Knowledge Discovery . Tk-Boost identifies agent’s misconceptions through analyzing the agent’s incorrect SQL queries and generates tribal knowledge—reusable natural language statements that correct the misconceptions that lead to incorrect queries. Accurately identifying misconceptions is challenging because many SQL queries contain complex logic that are difficult to comprehend, while errors depend on data idiosyncrasies so detecting them require an understanding of the underlying data. Simply prompting an LLM to derive knowledge by comparing the agent-generated and ground-truth SQL (similar to summarizing past experience in memory stores ( Baek et al. 2025b ; Chhikara et al. 2025 ; Li et al. 2025b ) ) often yields incorrect or trivial statements

Read the complete text