RAGONITE: Iterative Retrieval on Induced Databases and Verbalized RDF for Conversational QA over KGs with RAG
Rishiraj Saha Roy, Chris Hinze, Joel Schlotthauer, Farzad Naderi, Viktor Hangya, Andreas Foltyn, Luzian Hahn, Fabian Kuech · 2024
Read RAGONITE: Iterative Retrieval on Induced Databases and Verbalized RDF for Conversational QA over KGs with RAG by Rishiraj Saha Roy, Chris Hinze, Joel Schlotthauer, Farzad Naderi, Viktor Hangya, Andreas Foltyn, Luzian Hahn, Fabian Kuech (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-17690.txt.
Introduction
Chatting with a knowledge graph usually means translating questions into a stiff query language, and that translation breaks on tangled or vague asks. Rishiraj Saha Roy led a team that built RAGONITE, a system that answers in conversation without leaning on that language alone. It fuses two searches: database queries over a table form of the graph, and text search over plain-word versions of the graph's facts.
When either search comes back thin, the system loops for another round on its own. A language model then writes the reply from everything gathered. On a knowledge graph of BMW cars, the system beat several rival setups. The team argues the mix fits how people really ask follow-ups.
Catalog record
| CREATOR | Rishiraj Saha Roy, Chris Hinze, Joel Schlotthauer, Farzad Naderi, Viktor Hangya, Andreas Foltyn, Luzian Hahn, Fabian Kuech |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 19 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-17690
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
RAGONITE: Iterative Retrieval on Induced Databases and Verbalized RDF for Conversational QA over KGs with RAG Rishiraj Saha Roy, Chris Hinze, Joel Schlotthauer, Farzad Naderi, Viktor Hangya, Andreas Foltyn, Luzian Hahn, Fabian Kuech Abstract Conversational question answering (ConvQA) is a convenient means of searching over RDF knowledge graphs (KGs), where a prevalent approach is to translate natural language questions to SPARQL queries. However, SPARQL has certain shortcomings: (i) it is brittle for complex intents and conversational questions, and (ii) it is not suitable for more abstract needs. Instead, we propose a novel two-pronged system where we fuse: (i) SQL-query results over a database automatically derived from the KG, and (ii) text-search results over verbalizations of KG facts. Our pipeline supports iterative retrieval: when the results of any branch are found to be unsatisfactory, the system can automatically opt for further rounds. We put everything together in a retrieval augmented generation (RAG) setup, where an LLM generates a coherent response from accumulated search results. We demonstrate the superiority of our proposed system over several baselines on a knowledge graph of BMW automobiles. [email protected] [email protected] [email protected] [email protected] [email protected] [email protected]@iis.fraunhofer.de0009-0005-8450-2714fabian.kuech@iis.fraunhofer.de0009-0005-4665-387X keywords Knowledge Graphs Question Answering Retrieval Augmented Generation 1 Motivation Querying knowledge graphs . It is common practice in large-scale enterprises to organize important factual data as RDF knowledge graphs (KGs, equivalently knowledge bases or KBs) [ gomez2017enterprise , song2017building , galkin2017enterprise ] . Storing data as RDF KGs has the advantage of a flexible subject-predicate-object (SPO) format, which simplifies the work of moderators by eliminating the need for complex schemas as in equivalent databases. KGs are usually queried via SPARQL, a data retrieval and manipulation language tailored to work with graph-based data. With the advent of large language models (LLMs) [ touvron2023llama , achiam2023gpt , jiang2024mixtral , ali2024teuken ] , LLMs are now used to generate SPARQL queries from the user’s natural language (NL) questions [ meyer2024assessing , perevalov2024towards , yang2023llm , rangel2024sparql ] , replacing previous sophisticated in-house systems [ bakhshi2020data , diefenbach2020qanswer , sun2020sparqa ] . A two-pronged pipeline . Like other knowledge repositories, KGs are also conveniently searched and explored via conversational question answering (ConvQA) systems [ christmann2019look , saha2018complex , kaiser2021reinforcement , kacupaj2021conversational ] . However, ConvQA comes with a major challenge: vital parts of the question are left implicit by the user, as in dialogue with another human. In preliminary experiments, we found that even with the most capable LLMs like GPT-4o, translating a user’s conversational questions to SPARQL is a bottleneck, even with representative in-context examples. This problem is exacerbated for more complex intents with conditions, comparisons and aggregations [ roy2022question ] . Moreover, a KG often contains information that can satisfy more abstract intents: this cannot be harnessed via SPARQL. In this work, we overcome these limitations and present a novel system with two branches: SQL over induced databases satisfy crisp information needs – including mathematical operations, while less defined intents are handled by text retrieval over KG verbalizations . Notably, we allow repeated retrievals if a single round fails to fetch satisfactory information from the backend KG. Finally, the results of these branches are merged by a generator LLM to formulate the answer that is shown to the user who posed the conversational question. 2 RAGONITE: System Description and Unique Features Figure 1 : The RAGONITE workflow for ConvQA over KGs with retrieval augmented generation. System overview . Fig. 1 shows an overview of our agentic system RAGONITE (Retrieval Augmented Generation ON ITErative retrieval results). There are two retrieval branches in RAGONITE: (i) the SQL query executes over the DB and obtains results, and (ii) the KG verbalizations are searched via a dense retriever model [ karpukhin2020dense ] for passages that best satisfy the NL question. A plausible user question to our system could be What is the average acceleration time to 100 kmph for BMW Sport models? , followed by another question with implicit intent, such as And how does this compare to a typical X1 model? The system handles such queries by generating an intent-explicit SQL query [ li2024can ] and an intent-explicit NL question [ vakulenko2021question ] with an LLM. An intent-explicit form is a self-sufficient variant that contains all necessary information from the previous conversational context. The generated SQL query is then run over a database that we automatically derive from the KG (Sec. 2.1 ), and the NL question is searched over the verbalizations of the KG facts (Sec. 2.2 ). Comparing SQL results and the top-k verbalizations with the conversational question and the previous history, the LLM decides whether the retrieved information is sufficient to generate a satisfactory answer. If not, it suggests another round of retrieval (Sec. 2.3 ). Finally, the accumulated SQL results and top-k verbalizations are fused by a second LLM agent (Sec. 2.4 ) to present a coherent and fluent answer to the end user. The demo supports locally hosted LLMs as well as those accessible via APIs (Sec. 2.5 ). 2.1 Database induction Figure 2 : Canonical example illustrating automatic database induction from knowledge graph. A key feature of this demo is the derivation of a database from an RDF KG as follows. Inducing schema . Our KG is originally in an NTriples format 1 1 1 https://www.w3.org/TR/n-triples/ . This means that it is comprised of facts that are SPO triples, where subjects are entities (like bmw-x6-m-competition , bmw-ix1-edrive20-sport , or adaptive-LED-headlights ), predicates are relations (like engine-specification , wheelbase , or price ), and objects are types (like car , engine and equipment ) or literals (constants like 37450 EUR , 2760 mm or 250 kWh ). We first convert the KG from Ntriples into Turtle format 2 2 2 https://www.w3.org/TR/turtle/ , that encapsulates all facts of a specific subject entity. We identify unique entity types based on the values of the type predicate. For each entity type a table is added to the database. The subject entity is used as the primary key and each literal value is added as a column (e.g. “price” or “height” are added to the “car” table). Next, the relations between entity types are analyzed. For each 1:1 or 1:N relation between entity types A and B, a column is added to the entity table B that contains a foreign key which points to table A. For N:M relations, an additional table is added to the database which contains two columns with foreign keys. Inserting data . For each set of encapsulated Turtle facts, a single row is added to the respective table. The primary key is the subject entity in the Turtle facts. All literal values are added to fill the columns. Next we add the foreign keys of related entities. Then, for each table, the values of each column are analyzed to derive the data type of the column (like “INT” for “price”, or “TEXT” for “drive type”) and whether the values can be NULL. This information is then added to the schema. Enhancing semantics . Finally, our system allows renaming tables and columns to improve the LLM’s comprehension of the schema (such as “engine specification” \mapsto “engine” ). Additionally, comments can be added to each column, for example, NL explanations of complex predicate