Text-to-SQL Task-oriented Dialogue Ontology Construction

Renato Vukovic, Carel van Niekerk, Michael Heck, Benjamin Ruppik · 2025

Read Text-to-SQL Task-oriented Dialogue Ontology Construction by Renato Vukovic, Carel van Niekerk, Michael Heck, Benjamin Ruppik (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-23358.txt.

Introduction

Renato Vukovic and co-authors brought in TeQoDO in 2025, a method that lets a language model build the data store schema for a task-oriented dialogue system by itself. Such assistants answer from an outer data store set up by a clear ontology. This keeps them clear and controllable, but writing that ontology usually takes hand labels or supervised training. Language models alone, relying on memory, are hard to trust.

TeQoDO exploits the model's skill at SQL programming. Given concepts from modular dialogue system design in its prompt, the model constructs a working ontology from scratch as a data store design task. The result beat transfer learning methods at ontology building, and the generated ontology performed competitively on the downstream job of tracking dialogue state. Tests on Wikipedia and arXiv substance showed the method also scales to much larger ontologies.

Catalog record

CREATORRenato Vukovic, Carel van Niekerk, Michael Heck, Benjamin Ruppik
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE64 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-23358

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Text-to-SQL Task-oriented Dialogue Ontology Construction Renato Vukovic, Carel van Niekerk, Michael Heck, Benjamin Ruppik Abstract Large language models (LLMs) are widely used as general-purpose knowledge sources, but they rely on parametric knowledge, limiting explainability and trustworthiness. In task-oriented dialogue (TOD) systems, this separation is explicit, using an external database structured by an explicit ontology to ensure explainability and controllability. However, building such ontologies requires manual labels or supervised training. We introduce TeQoDO : a Te xt-to-S Q L task- o riented D ialogue O ntology construction method. Here, an LLM autonomously builds a TOD ontology from scratch without supervision using its inherent SQL programming capabilities combined with dialogue theory provided in the prompt. We show that TeQoDO outperforms transfer learning approaches, and its constructed ontology is competitive on a downstream dialogue state tracking task. Ablation studies demonstrate the key role of dialogue theory. TeQoDO also scales to allow construction of much larger ontologies, which we investigate on a Wikipedia and ArXiv dataset. We view this as a step towards broader application of ontologies to increase LLM explainability. 1 1 1 Code will be released upon publication. 1 Introduction Large language models (LLMs) have become ubiquitous knowledge processing systems, with the ability to reach or surpass human performance on a wide variety of natural language processing tasks. These models are pre-trained on massive corpora and aligned via human feedback using reinforcement learning Brown et al. 2020 ; Ouyang et al. 2022 . Despite these remarkable abilities, there are some inherent problems associated with these systems. Their knowledge is stored in vast numbers of parameters, which makes it very difficult to understand their behaviour, even via probing Cífka and Liutkus 2023 . It is therefore extremely challenging to verify their inherent knowledge Zhong et al. 2024 . Moreover, LLMs often produce confident but non-factual outputs that just appear plausible on a superficial level Sahoo et al. 2024 ; Feng et al. 2024 . Finally, many LLMs and their training data are closed-source. Ontologies provide a human-readable means of reconstructing knowledge-based reasoning in language models Gruber 1995 ; Lo et al. 2024 . Manually building ontologies for all relevant domains is infeasible Milward and Beveridge 2003 . To ensure broad applicability, ontologies should be constructed automatically, reliably, and consistently. Automatic ontology construction presents a promising solution to these challenges. In this work, we focus on constructing task-oriented dialogue (TOD) ontologies from raw dialogue data. The goal is to extract relevant information and organise it into a meaningful hierarchy for handling user queries. TOD ontologies consist of domains , slots , and values , with system actions and user intents defined over this domain-slot-value structure. Figures 1 and 2 illustrate ontology construction in this setting. Such ontologies are essential for modern TOD systems Young et al. 2013 ; Hudeček and Dusek 2023 . Most TOD ontology construction approaches follow two steps Hudeček et al. 2021 ; Vukovic et al. 2022 : (1) term extraction from dialogue data and (2) relation extraction between the terms from the previous step. Existing approaches tackle these steps separately and rely on annotated training data Vukovic et al. 2024 ; Finch et al. 2024 . This has the downside of additional and potentially error-prone training, and the danger of information loss between the two processing steps. We propose TeQoDO , a Te xt-to-S Q L task- o riented D ialogue O ntology construction approach. Figure 3 provides an overview of the method. TeQoDO uses LLMs’ code understanding and generation capabilities Chen et al. 2023 to build ontologies from scratch via SQL. It incrementally constructs the database by iterating over dialogues, retrieving relevant existing content, and updating the DB with new information. The model uses state tracking from dialogue theory to distinguish new content from existing data in the DB. The update prompt includes a notion of success to help align schema changes with the user’s goals. Query results are augmented with semantically similar concepts or example values from existing columns to improve coverage and consistency. Text-to-SQL provides a structured format familiar to LLMs through code encountered during pre-training Deng et al. 2022 ; Zhao et al. 2024 . It reduces the need for textual prompts, as SQL tables, columns, and values align with TOD ontology domains, slots, and values. To our knowledge, we are the first to use text-to-SQL to build a TOD ontology from scratch. We also adapt similarity-based evaluation metrics for ontology learning Lo et al. 2024 to reflect the hierarchical nature of TOD ontologies. This enables evaluation of various concept types, such as domains and slots. We run experiments on two widely used TOD datasets: MultiWOZ Eric et al. 2020 and Schema-Guided Dialogue (SGD) Rastogi et al. 2020 . TeQoDO outperforms recent TOD ontology construction methods Finch et al. 2024 ; Vukovic et al. 2024 . To test generalisation, we apply TeQoDO to general ontology datasets from Lo et al. 2024 and show it scales to large ontologies. In summary, our contributions are: • We propose a text-to-SQL framework for building TOD ontologies from scratch, allowing LLMs to update a DB with each newly observed dialogue without supervision or rule-based aggregation. • The constructed ontology enables dialogue state tracking with performance comparable to using the ground truth ontology. • TeQoDO generalises to large-scale ontology datasets like Wikipedia and ArXiv, performing competitively. 2 Related Work 2.1 Ontology Construction Figure 1: An example of ontology construction shows the extraction of the domain “Hotels”, with the value “cheap” assigned to the slot “Price”. System actions and user intents are defined based on the domains and slots specified in the ontology. Prior to deep learning, ontology construction relied on frequency-based rules and linguistic features. With language models, clustering and supervised methods enabled feature-based learning, as outlined below. Rule-based ontology construction ( Frantzi and Ananiadou 1999 ; Nakagawa and Mori 2002 ) relies on frequency-based methods to extract key subsequences from text. Wermter and Hahn 2006 use linguistic features for more advanced term extraction. While interpretable, these methods are hard to adapt across domains and often yield low precision. Clustering-based approaches, such as Yu et al. 2022 , apply unsupervised parsing and hierarchical clustering to group extracted terms into domains and slots. Finch et al. 2024 train a generative slot induction model to extract domain-slot-value triples, which are then clustered into a slot hierarchy. These methods usually produce many slots, whose interpretation depends entirely on how well they match the ground truth. This is because evaluation compares slots solely based on their values. They also rely heavily on data quality, embedding representations, and sensitive hyperparameters, such as the minimum cluster size. Supervised Lo et al. 2024 fine-tune LLMs to predict ontology sub-graphs at the document level and merge them using frequency-based rules to construct ontologies for Wikipedia and ArXiv datasets. They also propose evaluation functions covering both verbatim content and higher-level structural semantics compared to the ground truth, which we adapt to TOD. Vukovic et al. 2024 train a model to predict TOD ontology subgraphs at the dialogue level and aggregate them by merging all predictions. They improve generalisability by updating the decoding with constraints and confidence-based adjustments. However, their method focuses only on relation extraction, using ground truth terms as

Read the complete text