Tug-of-War within A Decade: Conflict Resolution in Vulnerability Analysis via Teacher-Guided Retrieval-Augmented Generations
Ziyin Zhou, Jianyi Zhang*, Yilong Li, Jiameng Han, Zhangchi Zhao · 2026
Read Tug-of-War within A Decade: Conflict Resolution in Vulnerability Analysis via Teacher-Guided Retrieval-Augmented Generations by Ziyin Zhou, Jianyi Zhang*, Yilong Li, Jiameng Han, Zhangchi Zhao (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-14172.txt.
Introduction
Zhou and five colleagues tackle knowledge conflicts in vulnerability analysis, where tens of thousands of records changed over the past decade. Their two-stage CRVA-TGRAG framework first sharpens document retrieval with parent-document splitting plus an ensemble of semantic and inverted-index search. A teacher-guided preference tuning step then aligns generation with the retrieved CVE records. Tests show the method retrieves the latest records more accurately than outside knowledge bases and cuts conflicts from relying on model memory alone.
Catalog record
| CREATOR | Ziyin Zhou, Jianyi Zhang*, Yilong Li, Jiameng Han, Zhangchi Zhao |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 38836 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.14172
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Tug-of-War within A Decade: Conflict Resolution in Vulnerability Analysis via Teacher-Guided Retrieval-Augmented Generations Ziyin Zhou, Jianyi Zhang*, Yilong Li, Jiameng Han, Zhangchi Zhao Abstract Large Language Models (LLMs) are essential for analyzing and addressing vulnerabilities in cybersecurity. However, among over 200,000 vulnerabilities were discovered in the past decade, more than 30,000 have been changed or updated. This necessitates frequent updates to the training datasets and internal knowledge bases of LLMs to maintain knowledge consistency. In this paper, we focus on the problem of knowledge discrepancy and conflict within CVE (Common Vulnerabilities and Exposures) detection and analysis. This problem hinders LLMs’ ability to retrieve the latest knowledge from original training datasets, leading to knowledge conflicts, fabrications of factually incorrect results, and generation hallucinations. To address this problem, we propose an innovative two-stage framework called CRVA-TGRAG (Conflict Resolution in Vulnerability Analysis via Teacher-Guided Retrieval-Augmented Generation). First, to improve document retrieval accuracy during the retrieval stage, we utilize Parent Document Segmentation and an ensemble retrieval scheme based on semantic similarity and inverted indexing. Second, to enhance LLMs’ capabilities based on the retrieval of CVE dataset in generation stage, we employ a teacher-guided preference optimization technique to fine-tune LLMs. Our framework not only enhances the quality of content retrieval through RAG but also leverages the advantages of preference fine-tuning in LLMs to answer questions more effectively and precisely. Experiments demonstrate our method achieves higher accuracy in retrieving the latest CVEs compared to external knowledge bases. In conclusion, our framework significantly mitigates potential knowledge conflicts and inconsistencies that may arise from relying solely on LLMs for knowledge retrieval. Figure 1 : Knowledge conflict in LLM retrieving CVE related information 1 Introduction As time goes by, facts and knowledge about the world change and evolve alongside with them 25 ; 10 ; 11 . For instance, certain information, such as “the President-elect of the United States”, “LeBron James’s home team”, and “the latest Olympic host city”, may change approximately every four years 12 . Additionally, some information evolves more frequently and rapidly, such as Python libraries 2 and Cyber Threat Intelligence (CTI) data 28 , including CVE and CWE numbers. The updates and upgrades of this knowledge are often scattered across official websites like National Vulnerability Database (NVD) 1 1 1 https://nvd.nist.gov/ , project documentation, and even commit records in GitHub repositories, both explicitly and implicitly 31 . These characteristics of knowledge present significant challenges for current LLMs in tracking the evolution and changes of knowledge to simultaneously enhance their retrieval capabilities 4 . Compared to traditional machine learning models 8 , LLMs 3 can learn extensive common sense knowledge after being pre-trained on large amount of datasets. Closed-source LLMs, such as GPTs 1 , have demonstrated impressive performance in solving general tasks 23 , following input instructions, aligning with human capabilities and reasoning for specific tasks. Open-source LLMs, such as Llama 27 and Mistral 14 , can effectively learn domain-specific knowledge and develop question-answering preferences through supervised fine-tuning and preference fine-tuning. This significantly expands the generative and expressive capabilities of LLMs, enhancing their ability to generate responses and tackle various complex tasks. For instance, integrating CVE vulnerability data as external knowledge within the context of the LLM can significantly enhance the accuracy of the LLM’s responses to CVE-related inquiries when people query vulnerability information associated with CVE-IDs. Therefore, associate the most accurate and the latest external knowledge as contextual prompt into LLMs could effectively alleviate potential knowledge conflicts and inconsistencies arising from pre-training phase and knowledge update process. The update frequency of the CVE database is measured in hours. Due to the timeliness of the knowledge base and training datasets of LLMs, as well as the significant costs associated with the training process, an offline pre-trained LLM cannot learn the latest CVE knowledge in real-time. Existing LLMs typically have a “knowledge cutoff date” 16 which refers to the latest time that the model finished training. Therefore, questions to the LLM after the date may yield incorrect answers, or even fabricate responses that are inconsistent with the facts, resulting in hallucinations. Figure 1 shows the conflict phenomenon in using LLM to retrieve knowledge related to “The latest CVE Vulnerability”. The advancement of fine-tuning techniques allows LLMs to further training on pre-trained models for specific domains or tasks, optimizing their performance in those specific Q&A 21 . However, this method does not fundamentally resolve the issue of knowledge conflicts. Traditional Retrieval-Augmented Generation (RAG) method could retrieve external knowledge bases and match the best answers for the questions, which are then prompt as context to the LLMs. There may be contradictory knowledge within the knowledge base. This can lead to conflicts hidden in the context provided to the LLM, preventing it from answering questions accurately. To address this problem, in this paper, we propose a two-stage framework for conflict resolution, named CRVA-TGRAG. In the course of our research, the first question we have faced is there are conflicts among the sources of CVE data. The sources of CVE data are extensive, but many lack sufficient authority and have not been updated with official sources synchronously, leading to inconsistencies and conflicts. Secondly, the retrieval accuracy is low when using LLM for CVE number searches. Adjacent CVE numbers are likely to have high similarity in content, making it difficult for LLM to accurately distinguish between them and resulting in incorrect outputs. Hence, we first use the API provided by NVD and CVE official GitHub dataset to download CVE documents in Json format, organizing the knowledge before and after updates for each CVE number and labeling their differences. It ensures the accuracy and authority of CVE content, eliminating the impact of knowledge conflicts from the dataset. Next, we consider using RAG approach. It could retrieve the external CVE dataset that collected in the previous phase and work with LLM to generate content more accurately. We improve the RAG approach, optimize the document segmentation and embedding processes according to the characteristics of CVE documents, while merging various query methods during the query phase to enhance retrieval quality and better access the required documents. Also, we hope LLM would prefer to generate updated knowledge. So we use the CVE knowledge before and after updates in the dataset as a preference-tuning dataset, allowing LLM to learn and develop preferences to accept updated knowledge, further resolving knowledge conflicts. We compared our work with previous research and designed evaluation metrics to assess the generation quality and reliability of CRVA-TGRAG. The experimental results indicate that our framework effectively addresses the impact of CVE updates on LLMs, alleviating the issues of knowledge inconsistency and conflicts that arise when answering CVE-related questions. This, in turn, significantly enhances the generation quality of the LLM, providing more effective support for researchers in analyzing vulnerabilities. Our contributions are summarized as follows: • We first address the gap in the field of vulnerability analysis for the lack of a knowledge conflict dataset. The dataset includes 1,260 pairwise conflict CVE items that have been published and modified over the past decade. It is accessible in our GitHub repository, accompanied by detailed explanations of the processing steps. • We present a two-stage framework named CRVA-TGRAG for knowledge conflict resolution. Our method enhances answer correctness by