Retrieval-Augmented Semantic Parsing: Using Large Language Models to Improve Generalization

Xiao Zhang, Qianru Meng, Johan Bos · 2024

Read Retrieval-Augmented Semantic Parsing: Using Large Language Models to Improve Generalization by Xiao Zhang, Qianru Meng, Johan Bos (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-10207.txt.

Introduction

Xiao Zhang, Qianru Meng, and Johan Bos teach language models to turn sentences into formal meaning maps with help from a fetched word store. Their paper appeared in two thousand twenty-four. Semantic parsing maps plain text into a strict form that machines can reason over. Neural parsers often trip on words and ideas they never saw in training.

Their method, RASP, pulls in outside symbolic knowledge about words during parsing. Large language models already beat older encoder-decoder parsers on the task. Adding the fetched knowledge nearly doubled scores on concepts outside the training data. That gain shows the models use the store rather than just their memory. The authors argue that pairing big models with lookup tools is a strong path. It points toward parsers that hold up in open, shifting domains.

Catalog record

CREATORXiao Zhang, Qianru Meng, Johan Bos
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-10207

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Retrieval-Augmented Semantic Parsing: Using Large Language Models to Improve Generalization Xiao Zhang, Qianru Meng, Johan Bos Abstract Open-domain semantic parsing remains a challenging task, as models often rely on heuristics and struggle to handle unseen concepts. In this paper, we investigate the potential of large language models (LLMs) for this task and introduce Retrieval-Augmented Semantic Parsing (RASP), a simple yet effective approach that integrates external lexical knowledge into the parsing process. Our experiments not only show that LLMs outperform previous encoder-decoder baselines for semantic parsing, but that RASP further enhances their ability to predict unseen concepts, nearly doubling the performance of previous models on out-of-distribution concepts. These findings highlight the promise of leveraging large language models and retrieval mechanisms for robust and open-domain semantic parsing. 1 Introduction Open-domain semantic parsing involves mapping natural language text to formal meaning representations, capturing the concepts, relations between them, and the contects in which they appear Oepen and Lønning 2006 ; Hajič et al. 2012 ; Banarescu et al. 2013 ; Bos et al. 2017 ; Martínez Lorenzo et al. 2022 . Such meaning representations are applied in many downstream applications—ranging from database querying to embodied questions answering—where parsers must handle a vast array of concepts that may not appear in the training data. While neural encoder-decoder architectures have shown impressive performance in semantic parsing tasks, their reliance on training distributions constrains their ability to generalize, especially to out-of-distribution (OOD) concepts. Most existing semantic parsers struggle to interpret unseen words or rare senses, often defaulting to the most frequent meaning encountered during training. As a result, they fail to adapt to novel linguistic phenomena and remain limited to fixed patterns. Recent work Zhang et al. 2024a have attempted to mitigate these limitations by encoding concept representations numerically, forcing models to learn underlying ontologies from resources like WordNet Fellbaum 1998 . However, these approaches require substantial preprocessing and intricate encodings that can be difficult for models to fully exploit. In our work, instead, we explore the potential of large language models (LLMs), powerful decoder-only architectures with strong in-context learning capabilities and extensive pretraining, to enhance ability of semantic parsers to generalize. We pose two central research questions: • Do large language models outperform traditional encoder-decoder architectures in semantic parsing? Decoder-only architectures are known to be more scalable and to internalize broader knowledge, potentially leading to stronger generalization and learning abilities. Assessing their performance in semantic parsing tasks can help reveal the architectural advantages of these decoder-only models. • How can these large language models be leveraged to improve generalization to out-of-distribution concepts? Beyond simple architecture comparisons, we investigate whether LLMs can be guided to handle concepts more flexibly, using their ability to interpret and integrate external information. In Section 2 we provide background on the semantic formalism of our choice, earlier approach to semantic parsing, and the challenge of an important sub-task, word sense disambiguation. Then we propose Retrieval-Augmented Semantic Parsing (RASP) in Section 3 , a technique that integrates a retrieval mechanism into the parsing process. RASP leverages external lexical knowledge (e.g., WordNet) in the input, enabling the model to dynamically access and interpret relevant concept information. By incorporating this retrieval step (Section 4 ), we relax the reliance on lemma-based mappings and allow the model to adapt more naturally to unseen words or senses. Our results show that this approach nearly doubles the performance on predicting OOD concepts compared to previous methods, demonstrating a substantial advancement in handling challenging open-domain data (Section 5 ). 2 Background and Related Work 2.1 Discourse Representation Structure Discourse Representation Theory ( Kamp and Reyle 1993 , DRT) is a semantic modeling framework. The core component of DRT is the Discourse Representation Structure (DRS), a formal representation that captures the meaning of a discourse, which captures the essence of the text and covering linguistic phenomena like anaphors and temporal expressions. Unlike many other formalisms such as Abstract Meaning Representation ( Banarescu et al. 2013 , AMR) used for large-scale semantic annotation efforts, DRS covers logical negation, quantification, and discourse relations, has complete word sense disambiguation, and offers a language-neutral meaning representation. A Discourse Representation Structure (DRS) can be coded and visualised in various ways, which are all provided in Parallel Meaning Bank Abzianidze et al. 2017 . In formal semantics they are often pictured in a human-readable box format. The clause notation was introduced to represent DRS in a sequential format suitable for machine learning models van Noord et al. 2018 . To further simplify DRS, Bos 2023 proposed a variable-free format known as Sequence Box Notation (SBN). An example of the three different but logically equivalent formats is shown in Figure 1 . Recent trends in using seq2seq models have led to a preference for sequence notation, which is also the format used in this paper. Figure 1: Three formats of Discourse Representation Structure (DRS) for "Mary didn’t commit a crime.": the box notation, a directed acyclic graph, and the sequence notation. 2.2 Semantic Parsing Semantic parsing, as a traditional NLP task, remains essential in real-world applications, despite recent progress in natural language understanding shown by large language models. For instance, natural language front-end interfaces to databases require a mapping from text to structured data. Speech interactions with conversational agents that act in the real world (e.g., service robots) require situation-sensitive symbol grounding. Hence, advancing the development of more robust and general semantic parsers remains crucial. Early approaches to semantic parsing primarily relied on rule-based systems Woods 1973 ; Hendrix et al. 1977 ; Templeton and Burger 1983 . The advent of neural methodologies, coupled with the availability of large semantically annotated datasets Banarescu et al. 2013 ; Bos et al. 2017 ; Abzianidze et al. 2017 , marked a significant shift in semantic parsing techniques Barzdins and Gosko 2016 ; van Noord and Bos 2017 ; Bevilacqua et al. 2021a . The introduction of pre-trained language models within the sequence-to-sequence framework further improved parsing performance van Noord et al. 2018 ; van Noord et al. 2020 ; Ozaki et al. 2020 ; Samuel and Straka 2020 ; Shou and Lin 2021 ; Bevilacqua et al. 2021a ; Zhou et al. 2021 ; Martínez Lorenzo et al. 2022 ; Zhang et al. 2024b ; Liu 2024a ; Liu 2024b ; Yang et al. 2024 . Furthermore, several studies introduced more pre-training tasks specifically designed for semantic parsing Bai et al. 2022 ; Wang et al. 2023a . With the rise of large language models, there has been considerable discussion about leveraging these models for semantic parsing, achieving notable results through techniques like prompting and chain-of-thought reasoning Roy et al. 2022 ; Ettinger et al. 2023 ; Jin et al. 2024 . However, there is currently no work that leverages the knowledge and understanding capabilities of large language models to address the generalization problem in semantic parsing. 2.3 Word Sense Disambiguation The generalization problem introduced in the previous section can also be understood as word sense disambiguation (WSD) for out-of-distribution concepts, within the context of semantic parsing. For instance, consider the sentence "She had £10,000 in the bank", with the target word "bank". In traditional WSD tasks, a predefined inventory of possible senses (e.g., 1. sloping land; 2.

Read the complete text