Ontology-Aware RAG for Improved Question-Answering in Cybersecurity Education
Chengshuai Zhao Garima Agrawal, Fan Zhang, Tharindu Kumarage, Zhen Tan, Yuli Deng, Ying-Chih Chen, Huan Liu · 2024
Read Ontology-Aware RAG for Improved Question-Answering in Cybersecurity Education by Chengshuai Zhao Garima Agrawal, Fan Zhang, Tharindu Kumarage, Zhen Tan, Yuli Deng, Ying-Chih Chen, Huan Liu (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-14191.txt.
Introduction
Chengshuai Zhao and co-authors present CyberRAG, an ontology-aware system for cybersecurity education, building question answering for a hard technical field. Language models can hallucinate and lack deep domain facts, and those flaws cut trust in classroom use of AI tutors. Retrieval-augmented generation offers a grounding fix for answers.
CyberRAG first retrieves checked documents from a knowledge base. Retrieved facts raise the relevance and accuracy of responses. An ontology layer organizes the domain knowledge for smarter search. The two-step design manages uncertainty in student problem solving, and interactive, inquiry-based learning is the pedagogical goal. Safety and reliability guide the system architecture throughout. The work targets science and technology teaching more broadly.
Catalog record
| CREATOR | Chengshuai Zhao Garima Agrawal, Fan Zhang, Tharindu Kumarage, Zhen Tan, Yuli Deng, Ying-Chih Chen, Huan Liu |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 44 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-14191
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Ontology-Aware RAG for Improved Question-Answering in Cybersecurity Education Chengshuai Zhao Garima Agrawal, Fan Zhang, Tharindu Kumarage, Zhen Tan, Yuli Deng, Ying-Chih Chen, Huan Liu Abstract Integrating AI into education has the potential to transform the teaching of science and technology courses, particularly in the field of cybersecurity. AI-driven question-answering (QA) systems can actively manage uncertainty in cybersecurity problem-solving, offering interactive, inquiry-based learning experiences. Recently, Large language models (LLMs) have gained prominence in AI-driven QA systems, enabling advanced language understanding and user engagement. However, they face challenges like hallucinations and limited domain-specific knowledge, which reduce their reliability in educational settings. To address these challenges, we propose CyberRAG , an ontology-aware retrieval-augmented generation (RAG) approach for developing a reliable and safe QA system in cybersecurity education. CyberRAG employs a two-step approach: first, it augments the domain-specific knowledge by retrieving validated cybersecurity documents from a knowledge base to enhance the relevance and accuracy of the response. Second, it mitigates hallucinations and misuse by integrating a knowledge graph ontology to validate the final answer. Comprehensive experiments on publicly available datasets reveal that CyberRAG delivers accurate, reliable responses aligned with domain knowledge, demonstrating the potential of AI tools to enhance education. 1 1 1 Code: https://github.com/ChengshuaiZhao0/CyberRAG 1 School of Computing and Augmented Intelligence, Arizona State University, USA 2 Mary Lou Fulton Teachers College, Arizona State University, USA (czhao93; garima.agrawal; fzhan113; kskumara; ztan36; ydeng19; ychen495; huanliu)@asu.edu Introduction The use of AI in education has the potential to transform the teaching of technology courses ( George 2023 ; Harry 2023 ) . In scientific learning, students are expected to engage in problem-solving and exploration, yet traditional classroom methods often focus on the passive acquisition of established knowledge ( Ellis 1991 ) . This approach limits opportunities for students to experience the process of knowledge creation, leading to lower cognitive engagement ( Corno and Mandinach 1983 ) . Cybersecurity is a problem-based learning domain where students must master complex tools, develop defense techniques, and uncover new threats, which necessitates a re-imagination of traditional education practices ( Shivapurkar et al. 2020 ) . Prior research highlights that managing uncertainty is a crucial component of the learning process, as students often struggle with acquiring new skills, applying diverse methodologies, and forming new understandings ( Jordan 2015 ) . Educators can effectively manage this uncertainty by increasing it through the introduction of authentic, ambiguous challenges to stimulate critical thinking, maintaining it to encourage deeper exploration and problem-solving, and reducing it by identifying optimal solutions to help students integrate new insights with existing knowledge ( Chen et al. 2019 ) . AI-driven question-answering (QA) systems can help manage this uncertainty in technical problem-solving by supporting self-paced learning, significantly enhancing cognitive engagement ( Means 2021 ) . In recent years, large language models (LLMs) have become central to AI-driven technologies. While LLM-powered QA systems hold great promise for enhancing learning, they also face challenges such as hallucination and limited domain knowledge, which can undermine their effectiveness ( Huang et al. 2025 ; Zhao et al. 2025 ) . In cybersecurity education, where precision is critical, ensuring the accuracy of AI-generated content is essential. For example, in tasks such as identifying vulnerabilities or interpreting security policies, inaccurate AI responses could lead to misinformation, compromising the learning experience and, potentially, real-world security ( Kumarage et al. 2024 ) . A promising solution to address this challenge is the retrieval-augmented generation (RAG) approach, where the model generates responses by retrieving information from a validated knowledge base, thereby enhancing the accuracy and reliability ( Lewis et al. 2020 ) . Although the RAG approach helps reduce hallucinations and address domain knowledge issues to some extent, the reliability of LLM-generated answers remains a concern for achieving educational goals. Students may ask questions that fall outside the scope of the augmented cybersecurity knowledge base. In such cases, LLMs rely on their own parametric knowledge to generate responses, which can expose the QA system to risks of misinformation or misuse ( Tan et al. 2024a ; Tan et al. 2024b ) . In an educational setting, it is also crucial to prevent students from manipulating the AI system for unintended purposes. There is a strong need to provide a validation system to ensure the accuracy and safety of LLM-generated responses. One potential solution is safety alignment ( Christiano et al. 2017 ) . However, this method requires verification by cybersecurity experts, making it labor-intensive, costly, and time-consuming. Preferably, an automatic validation approach is needed. Domain-specific knowledge graphs, which structure expert knowledge and capture the interactions between key entities in alignment with domain rules ( Abu-Salih 2021 ) , offer a promising direction. The knowledge graph ontology encodes these rules ( Kejriwal 2019 ) . By leveraging this ontology, LLM responses can be validated by fact-checking against the predefined rules, ensuring greater accuracy and reliability without the need for constant human oversight. In this paper, we propose CyberRAG , an ontology-aware RAG approach for developing a reliable QA system in cybersecurity education, comprising two key components. First, we utilize RAG methods to retrieve validated cybersecurity documents from a knowledge base, enhancing both the accuracy and relevance of the answers. Through finely crafted prompts, CyberRAG improves QA performance by leveraging both cybersecurity content and the natural language processing capabilities of LLMs. Additionally, we introduce an ontology-based validation approach that uses a cybersecurity knowledge graph ontology to automatically verify LLM-generated responses, thereby preventing potential risks of misuse and hallucination. We conduct comprehensive experiments on publicly available datasets to demonstrate the effectiveness of CyberRAG in delivering reliable and accurate answers. Our research explores the potential of integrating AI into education, emphasizing its transformative impact on traditional methods. This approach extends beyond cybersecurity and can also be applied to other educational subjects. Our contributions can be summarized as follows: • Ontology-aware RAG for cybersecurity QA. We introduce CyberRAG , a plug-and-play pipeline that couples dense retrieval and LLM generation with an ontology-based validator instantiated from AISecKG, yielding accurate and safe answers for problem-based learning. • Fine-grained prompting for accuracy and safety. We design reusable prompts for answer generation and ontology-aligned validation that operationalize domain rules as checks, automatically filtering hallucinations and rejecting out-of-scope or misuse-seeking queries. • Comprehensive evaluation and ablations. We evaluate CyberRAG on CyberQ across In-KB , Out-of-KB , and Zero-shot settings with multiple LLM backbones and retrievers, using n-gram and RAGAS metrics. Ablations vary KB coverage, prompting strategies, and retriever choice, and we analyze an ontology gate on mixtures with non-cybersecurity QA sets. • Actionable guidance for deployment. From these studies, we provide actionable deployment guidance by clarifying how KB coverage, retrieval strategies, and model/prompting decisions shape quality, safety, and efficiency, leading to a transparent, auditable QA pipeline with applications beyond cybersecurity education Related Work RAG in Education Generative models have the potential to transform traditional education by enabling personalized learning and automating content creation, making education more adaptive and accessible ( George 2023 ) . The integration of LLMs in education has further enhanced conversational capabilities, supporting self-paced learning through AI-driven question-answering tools and bots ( Moore et al. 2023 ) . However, the use of LLMs also presents challenges, particularly the risks of hallucinations and inaccurate responses, which are critical