Ambiguity Resolution in Text-to-Structured Data Mapping

Zhibo Hu, Chen Wang, Yanfeng Shu, Helen Hye-Young Paik, Liming Zhu · 2025

Read Ambiguity Resolution in Text-to-Structured Data Mapping by Zhibo Hu, Chen Wang, Yanfeng Shu, Helen Hye-Young Paik, Liming Zhu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-11679.txt.

Introduction

Zhibo Hu and co-authors tackle ambiguity when language models map text to structured data in this 2025 paper. Vague wording can lead a model to pick the wrong tool call or database query. Past fixes rely on trial and error or on task-specific training.

The team instead studies how ambiguous text sits in the model's latent space. They build a distance measure from a path kernel over concepts, not plain embeddings. Patterns in that space flag a question as ambiguous before mapping starts. A second method predicts the missing concept behind the vagueness. Both parts reached top results on their test tasks.

Catalog record

CREATORZhibo Hu, Chen Wang, Yanfeng Shu, Helen Hye-Young Paik, Liming Zhu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-11679

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Ambiguity Resolution in Text-to-Structured Data Mapping Zhibo Hu, Chen Wang, Yanfeng Shu, Helen Hye-Young Paik, Liming Zhu Abstract Ambiguity in natural language is a significant obstacle for achieving accurate text to structured data mapping through large language models (LLMs), which affects the performance of tasks such as mapping text to agentic tool calling and text-to-SQL queries. Existing methods of ambiguity handling either exploit ReACT framework to produce the correct mapping through trial and error, or supervised fine tuning to guide models to produce a biased mapping to improve certain tasks. In this paper, we adopt a different approach that characterizes the representation difference of ambiguous text in the latent space and leverage the difference to identify ambiguity before mapping them to structured data. To detect ambiguity of a sentence, we focused on the relationship between ambiguous questions and their interpretations and what cause the LLM ignore multiple interpretations. Different to the distance calculated by dense embedding vectors, we utilize the observation that ambiguity is caused by concept missing in LLM’s latent space to design a new distance measurement, computed through the path kernel by the integral of gradient values for each concepts from sparse-autoencoder (SAE) under each state. We identify patterns to distinguish ambiguous questions with this measurement. Based on our observation, We propose a new framework to improve the performance of LLMs on ambiguous agentic tool calling through missing concepts prediction. 1 Introduction Effective question answering using large language models (LLMs) often fails when user questions are ambiguous. A growing strand of work like ( Min et al. 2020 ) , ( Stelmakh et al. 2022 ) shows that a surprising fraction of their “errors” can be traced back, not due to lack of knowledge in LLM, but to ambiguity in the user’s question itself. This ambiguity does not just mean that the question does not provide enough information, but also that the question has ambiguous semantics, i.e., multiple interpretations. Existing studies focus more on pragmatic or lexical ambiguity, these methods of ambiguity handling either exploit ReACT ( Yao et al. 2023 ) framework to produce the correct mapping through trial and error, or supervised fine tuning to guide models to produce a biased mapping to improve certain tasks ( Saparina and Lapata 2025 ) . However, there is a lack of research in the representation difference of ambiguous text. A related study from Kamath et al. ( Kamath et al. 2024 ) highlights a complementary, syntactic source of failure: sentences whose meaning changes with the relative scope of quantifiers, negation, or modals. They show that powerful LLMs trained on the most comprehensive datasets, such as GPT-4 sometimes default to a non-preferred semantic reading, and that success of mapping text to disambiguous tasks varies sharply with different phrasing, which indicates disambiguation can not be easily solved with scaling up the model training. This observation motivates us to explore the representation difference of ambiguous text in the latent space and leverage the difference to identify ambiguity. As an ambiguous utterance has multiple interpretations, studying the distribution of interpretations is a natural way for ambiguity detection ( Stengel-Eskin et al. 2023 ) . Figure 1 provides an example of the relationships between the ambiguous query q and its corresponding two interpretations, denoted by i_{1} and i_{2} . Ideally, a good distance measurement may uncover the pattern of the triplet associated with an ambiguous utterance. Unfortunately, current distance measurement by dense embedding vectors ( Karpukhin et al. 2020 ) cannot give us such a measurement. The distances computed by dense vectors focus more on the semantics of individual words than on the structure of the entire sentence, which is not sensitive to the ambiguity caused by the structure of the sentence, particularly when some concepts are missing in the sentence. Figure 1: An example to show the difference of the distance measurement on triplets for ambiguous question ( q , i_{1} , i_{2} ) and two kind of unambiguous question ( q^{\prime} , i^{\prime}_{1} , i^{\prime}_{2} ), ( q^{\prime\prime} , i^{\prime\prime}_{1} , i^{\prime\prime}_{2} ). For similar queries ambiguous q and unambiguous q^{\prime} , q^{\prime\prime} , we expect \overline{(\mathrm{D}(q,i_{1}),\mathrm{D}(q,i_{2}),\mathrm{D}(i_{1},i_{2}))} > \overline{(\mathrm{D}(q^{\prime},i^{\prime}_{1}),\mathrm{D}(q^{\prime},i^{\prime}_{2}),\mathrm{D}(i^{\prime}_{1},i^{\prime}_{2}))} and |\mathrm{D}(q,i_{1})-\mathrm{D}(q,i_{2})|\ll|\mathrm{D}(q^{\prime\prime},i^{\prime\prime}_{1})-\mathrm{D}(q^{\prime\prime},i^{\prime\prime}_{2})| (the overline means average). In our study, we observe that the ambiguity is often associated with missing concepts in the input utterances. With the recent progress on LLM interpretability ( Bricken et al. 2023 ) ( Templeton et al. 2024 ) , the human-understandable concepts embodied in a utterance can be extracted together with their representation in the latent space through a sparse autoencoder (SAE). This inspires us to design methods to learn the concept differences of multiple interpretations of the input utterance in the latent space to identify ambiguity. We further leverage a kernel method ( Domingos 2020 ) to develop a distance metric for such latent concept comparison. We turn SAE into a kernel machine to measure the similarity between the concept representations of different interpretations of an ambiguous utterance. The computing is done through the integral of gradient values in a path kernel for each concept extracted by SAE. To make the similarity measurement focus on the target semantic patterns, we filter out concepts irrelevant to input utterances. By doing this, we successfully discover the pattern of ambiguous questions. Once ambiguous utterances are identified, incorporating additional information may improve the performance of mapping them to structured data. When the structured data requires intermediate such as SQL generation to access, users often need to be asked to clarify the utterances and provide additional information. When the structured data are finite and well-defined, e.g., tools defined in agentic framework, the additional information can be obtained from the concepts embodied in the data. We exploit the difference of concepts between ambiguous queries and target structured data, and design a missing concept prediction model to assist the mapping. We show in the experiments that our method achieve the best API calling performance on Gorilla ( Patil et al. 2024 ) TensorFlow Hub bench. In summary, our work make the following contributions: 1. We observed that ambiguity arises from missing concepts in the latent space of LLMs. Using this insight, we designed a new distance measurement that enhances interpretability and targets specific semantic patterns. 2. We identify patterns to distinguish ambiguous questions with this measurement. 3. We propose a new framework to enhance the performance of LLMs in handling ambiguous agentic tool calls by predicting missing concepts. 2 Preliminary Path Kernel. Path kernels are used to measure how similarly a model at two data points varies during learning. Here we refer to the explanation for kernel machine from ( Domingos 2020 ) , a kernel machine predicts y=g\!\Bigl(\sum_{i}a_{i}\,K(x,x_{i})+b\Bigr), with the kernel K measuring the similarity between data points. Gradient-descent training (learning rate \varepsilon\!\to\!0 ) implies that the final predictor behaves like a path kernel machine: \mathrm{K}_{\text{path}}(x,x^{\prime})\;=\;\int_{c(t)}\nabla_{w}y(x)\!\cdot\!\nabla_{w}y(x^{\prime})\,dt, where c(t) is the parameter trajectory during training. The more aligned the gradients of y at x and x^{\prime} , the larger the kernel value, thus the variations of x and x^{\prime} are more similar during training. Sparse autoencoder (SAE). Neurons in modern language models often behave such that the same unit fires for several unrelated concepts. A leading hypothesis (Superposition Hypothesis) is ( Elhage et

Read the complete text