Evaluating the Performance of RAG Methods for Conversational AI in the Airport Domain

Yuyang Li, Philip J.M. Kerbusch, Raimon H.R. Pruim, Tobias Käfer · 2025

Read Evaluating the Performance of RAG Methods for Conversational AI in the Airport Domain by Yuyang Li, Philip J.M. Kerbusch, Raimon H.R. Pruim, Tobias Käfer (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-13006.txt.

Introduction

Four researchers compare three ways to build a chat assistant for airport staff, in a 2025 paper. Busy airports want more automation, and staff need fast answers from flight information systems. The assistant must handle jargon, short forms, and questions that take reasoning over live data. The team built three versions, each fetching facts a different way before a language model answers.

The plain fetch version scored highest on simple accuracy in some tests but sometimes made up facts, a risk in a safety setting. A version that queries databases directly and one that searches a knowledge graph both invented far less. The graph version scored best overall and shone on questions needing reasoning steps. The authors recommend the database and graph versions for airport use, where a wrong invented answer can do real harm.

Catalog record

CREATORYuyang Li, Philip J.M. Kerbusch, Raimon H.R. Pruim, Tobias Käfer
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE42 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-13006

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Evaluating the Performance of RAG Methods for Conversational AI in the Airport Domain Yuyang Li, Philip J.M. Kerbusch, Raimon H.R. Pruim, Tobias Käfer Abstract Airports from the top 20 in terms of annual passengers are highly dynamic environments with thousands of flights daily, and they aim to increase the degree of automation. To contribute to this, we implemented a Conversational AI system that enables staff in an airport to communicate with flight information systems. This system not only answers standard airport queries but also resolves airport terminology, jargon, abbreviations, and dynamic questions involving reasoning. In this paper, we built three different Retrieval-Augmented Generation (RAG) methods, including traditional RAG, SQL RAG, and Knowledge Graph-based RAG (Graph RAG). Experiments showed that traditional RAG achieved 84.84% accuracy using BM25 + GPT-4 but occasionally produced hallucinations, which is risky to airport safety. In contrast, SQL RAG and Graph RAG achieved 80.85% and 91.49% accuracy respectively, with significantly fewer hallucinations. Moreover, Graph RAG was especially effective for questions that involved reasoning. Based on our observations, we thus recommend SQL RAG and Graph RAG are better for airport environments, due to fewer hallucinations and the ability to handle dynamic questions. 1 Introduction Amsterdam Airport Schiphol, one of the top 20 airports in the world, ranked by annual passenger numbers, handles thousands of flights each day. These airports rely on staff like gate planners and apron controllers to access and update data across systems. For these employees, traditional database queries can be complex and time-consuming for some employees who are not query experts when they need flight information. A conversational AI system with a natural language query (NLQ) interface allows all employees to interact with systems naturally, asking questions like, “Which fights are at ramp D07?” and receiving instant answers. This improves productivity, and streamlines workflows, especially in high-pressure areas like at the gate, where less educated workers require access to up-to-date information. By replacing strict query formats with intuitive, real-time responses, conversational AI enhances decision-making and efficiency, making it a suitable solution for dynamic environments such as airports. Building such a system is challenging because flight data is stored by experts in tables using aviation abbreviations. We need our system to understand these datasets to answer questions from the airport domain. Additionally, ensuring aviation safety is a major concern; the system must be safe and enable employees to perform accurate operations. We address those challenges using two research questions. The first question is how to handle flight data so that our system can answer different questions. We divided the questions into three types: • Straightforward questions: Questions that can be directly answered from the flight data. • Questions involving specialized airport jargon, abbreviations, and incomplete queries: Operators often use shorthand or omit context. Flight “KL0123” might be referred to as “0123” or “123,” while gate “C05” might be shortened to “C5.” Abbreviations like “KLM” for “KLM Royal Dutch Airlines” or “Delta” for “Delta Air Lines” are also common. Operators frequently ask short, incomplete questions, e. g., “Which flights are at D04?” or “What is the gate for that Delta airline?” Without resolving missing details such, these questions cannot be answered. • Dynamic questions: Questions that involve additional calculations and reasoning, especially related to time. Examples include “What is the connecting flight’s onramp time for DL1000?” or “What is DL1000’s next flight from the same ramp?” These queries require reasoning through connections between flights and retrieving specific details. The second research question is about how to reduce hallucinations Xu et al. (2024) for the safety of aviation operations. Hallucinations occur when LLMs generate information not based on facts or their training data. In high-safety environments such as airports, however the output should be factual and not imaginative Jacobs and Jaschke (2024) . For example, if the system gives wrong gate numbers, flight schedules, or safety instructions, this might disrupt aviation operations, cause delays, or even risk passenger safety. Thus, accurate responses are important. In this case study, we examine three Retrieval-Augmented Generation (RAG) techniques for the airport environment: Traditional RAG Lewis et al. (2021) Retrieves relevant information from the flight database and uses LLMs to generate answers based on the retrieved data and original questions. SQL RAG Guo et al. (2023) stores all datasets in an SQL database and converts natural language questions (NLQ) into structured SQL queries. Knowledge Graph-based Retrieval-Augmented Generation (Graph RAG) Edge et al. (2024) aims to improve the performance of LLM tasks by applying RAG techniques to Knowledge Graphs (KGs), requiring the original datasets to be stored in the knowledge graph. A key challenge is retrieving the correct flight information from thousands of flights while minimizing hallucinations. The paper is structured as follows: We first survey related work (Sec. 2 ), then present our dataset (Sec. 3 ), followed by a high-level description of our experiments (Sec. 4 ). We then present the results for the research questions (Sec. 5 ), and lastly conclude (Sec. 6 ). In the Appendix A , we provide further details, especially on the question generation and classification, next to our prompting. 2 Related Work 2.1 Traditional RAG Traditional Retrieval-Augmented Generation (RAG) consists of two main stages: the Retriever and the Generator Louis et al. (2023) . The Retriever identifies relevant documents based on user input, and the Generator uses these documents to produce responses. We explore three retrieval methods: keyword search, semantic search, and hybrid search, using large language models (LLMs) for answer generation. In keyword search, TF-IDF and BM25 are employed to evaluate retrieval performance. TF-IDF computes term frequency (TF) and inverse document frequency (IDF) Liu et al. (2018) ; Robertson (2004) , measuring how important a term is within a document and across the corpus. BM25 extends TF-IDF with a term saturation function Robertson and Zaragoza (2009) , reducing the influence of extremely frequent terms that often carry less informative value Chen and Wiseman (2023) . Semantic search methods include similarity search, vector databases like FAISS Jegou et al. (2017) ; George and Rajan (2022) , k-Nearest Neighbors (KNN), Locality-Sensitive Hashing (LSH) Jafari et al. (2021) , and Maximal Marginal Relevance (MMR) Mao et al. (2020) . Unlike keyword search, semantic search aims to understand user intent and word meanings Gao et al. (2024) . Embedding models such as Word2Vec convert words into vectors Mikolov et al. (2013) , where cosine similarity measures similarities between queries and documents. Hybrid search combines keyword and semantic methods, re-ranking results using the Reciprocal Rank Fusion (RRF) algorithm Robert Lee (2024) . By combining two search methods, the hybrid search can not only find flight information by keywords but also find information by the deeper meaning of the queries Sarmah et al. (2024) . 2.2 SQL RAG Text-to-SQL aims to transfer natural language automatically questions(NLQs) into SQL queries. LLMs recently emerged as an option for Text-to-SQL task Rajkumar et al. (2022) . The trick to handling text-to-SQL tasks with LLMs is to apply prompt engineering. Five prompt styles for Text-to-SQL are explored in the previous research Gao et al. (2023) . Basic Prompt (BSP) is a simple representation with no instructions; Text Representation Prompt (TRP) adds basic task guidance; OpenAI Demonstration Prompt (ODP) adds explicit rules like “Complete sqlite

Read the complete text