ChatNVD: Advancing Cybersecurity Vulnerability Assessment with Large Language Models
Shivansh Chopra ∗ , Hussain Ahmad ∗† , Diksha Goel ‡ , Claudia Szabo ∗ ∗ The University of Adelaide, Australia ‡ CSIRO’s Data61, Australia [email protected], {hussain.ahmad, claudia.szabo}@adelaide.edu.au, [email protected] · 2024
Read ChatNVD: Advancing Cybersecurity Vulnerability Assessment with Large Language Models by Shivansh Chopra ∗ , Hussain Ahmad ∗† , Diksha Goel ‡ , Claudia Szabo ∗ ∗ The University of Adelaide, Australia ‡ CSIRO’s Data61, Australia [email protected], {hussain.ahmad, claudia.szabo}@adelaide.edu.au, [email protected] (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-04756.txt.
Introduction
Shivansh Chopra and three co-authors build, in 2024 work, a chat tool that explains software flaws in plain words. Flaw reports in the main national database are deep and hard to read, which can slow fixes and leave systems open to attack. Their tool, ChatNVD, draws on that database to write clear, rich summaries.
Three versions were built, each driven by a unlike large language model from a major lab. The team judged them on how well they spot, read, and explain real flaws. One model came out on top, with scores above nine in ten correct and the lowest error counts. The team names it the most sound choice for real-world flaw checks, and frames the tool as aid for staff who lack deep training.
Catalog record
| CREATOR | Shivansh Chopra ∗ , Hussain Ahmad ∗† , Diksha Goel ‡ , Claudia Szabo ∗ ∗ The University of Adelaide, Australia ‡ CSIRO’s Data61, Australia [email protected], {hussain.ahmad, claudia.szabo}@adelaide.edu.au, [email protected] |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 66 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-04756
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
ChatNVD: Advancing Cybersecurity Vulnerability Assessment with Large Language Models Shivansh Chopra ∗ , Hussain Ahmad ∗† , Diksha Goel ‡ , Claudia Szabo ∗ ∗ The University of Adelaide, Australia ‡ CSIRO’s Data61, Australia [email protected], {hussain.ahmad, claudia.szabo}@adelaide.edu.au, [email protected] Abstract The increasing frequency and sophistication of cybersecurity vulnerabilities in software systems underscores the need for more robust and effective vulnerability assessment methods. However, existing approaches often rely on highly technical and abstract frameworks, which hinder understanding and increase the likelihood of exploitation, resulting in severe cyberattacks. In this paper, we introduce ChatNVD, a support tool powered by Large Language Models (LLMs) that leverages the National Vulnerability Database (NVD) to generate accessible, context-rich summaries of software vulnerabilities. We develop three variants of ChatNVD, utilizing three prominent LLMs: GPT-4o Mini by OpenAI, LLaMA 3 by Meta, and Gemini 1.5 Pro by Google. To evaluate their performance, we conduct a comparative evaluation focused on their ability to identify, interpret, and explain software vulnerabilities. Our results demonstrate that GPT-4o Mini outperforms the other models, achieving over 92% accuracy and the lowest error rates, making it the most reliable option for real-world vulnerability assessment. I Introduction Large Language Models (LLMs), a subset of generative artificial intelligence, have revolutionized numerous fields by enabling advanced content generation and process automation [ 1 , 2 ] . In education, LLMs enhance efficiency in tasks such as generating questions [ 3 ] , grading essays [ 4 ] , providing personalized feedback [ 5 ] , and evaluating assignments [ 6 ] . Within the entertainment industry, they exhibit remarkable capabilities, crafting video game narratives [ 7 ] and generating music captions [ 8 ] , thereby enriching creative workflows. Business operations have also undergone significant transformation through the integration of LLMs, which streamline marketing campaigns [ 9 ] , improve customer service interactions [ 10 ] , and optimize supply chain processes [ 11 ] . In healthcare, LLMs empower professionals by offering real-time clinical decision support [ 12 ] , advancing medical education [ 13 ] , and enabling predictive analytics for disease progression [ 14 ] . These applications highlight the profound and multifaceted impact of LLMs, positioning them as key drivers of innovation across diverse disciplines [ 15 ] . The dynamic and ever-evolving nature of the cybersecurity domain demands continuous adaptation to cutting-edge advancements [ 16 ] , as each technological innovation introduces potential new attack vectors and expands the surface available for exploitation [ 17 , 18 ] . Software vulnerabilities, whether accidental oversights or deliberate flaws, serve as entry points for system manipulation by threat actors [ 19 , 20 ] . These vulnerabilities can result in severe consequences, including unauthorized data access or alteration, system disruptions through denial-of-service attacks, or the execution of malicious code, potentially enabling full system compromise [ 21 ] . To address these risks, organizations frequently conduct proactive evaluations of system resilience through vulnerability assessments [ 22 ] . Such assessments play a vital role in fortifying system defences and safeguarding the integrity and security of digital infrastructures [ 23 ] . Fig. 1: Growth Trend in Reported Software Vulnerabilities [ 24 ] . The rapid growth in the volume and sophistication of cyber threats has driven a substantial increase in the number of reported software vulnerabilities within the National Vulnerability Database 1 1 1 https://nvd.nist.gov [ 25 ] . As shown in Figure 1 , the trend of identified vulnerabilities has quadrupled over the past decade [ 24 ] , reflecting the growing complexity of modern software systems and the expanding attack surface. This surge poses challenges for cybersecurity practitioners in keeping up with evolving threats. According to a recent report by the World Economic Forum, cyberattacks are among the top five global risks in terms of likelihood and impact [ 26 ] . Despite this alarming trend, existing methods for vulnerability assessment often depend on overly technical and abstract frameworks, such as manual analysis [ 27 , 28 ] and rule-based approaches [ 29 ] . These methods are not only time-intensive but also susceptible to human error [ 28 ] , limiting their effectiveness in addressing the rapidly growing number of vulnerabilities. Furthermore, their complexity and technical nature can create barriers to broader understanding among diverse stakeholders, complicating efforts to prioritize and mitigate vulnerabilities effectively [ 30 ] . This gap in accessibility and clarity increases the likelihood of exploitation, escalating the risk of severe cyberattacks. To combat this, there is a critical need for more comprehensible and actionable solutions that enable both technical and non-technical stakeholders to assess and mitigate software vulnerabilities effectively. The NVD serves as a comprehensive repository of software vulnerabilities, offering detailed descriptions that are invaluable for reference [ 31 ] . However, the NVD is designed primarily for lookup and reference purposes and lacks interactive querying capabilities and deeper contextual understanding [ 32 ] . This limitation underscores the need for an advanced solution that goes beyond simple lookup by delivering detailed, context-aware responses tailored to specific vulnerability queries. LLMs can form the backbone of such a tool as they can analyze and interpret NVD data, enabling dynamic interaction with the information and transforming it into a more practical and accessible resource. This approach is especially beneficial for cybersecurity professionals who require a quick, accurate, and in-depth understanding of vulnerabilities to make informed decisions. The NVD, with its extensive dataset of software vulnerabilities, serves as an ideal foundation for training LLMs, enabling the processing of large volumes of data, summarizing complex technical information, and providing actionable insights. An LLM-powered solution would facilitate improved vulnerability comprehension, support more effective mitigation strategies, and reduce the risk of exploitation, benefiting a diverse range of stakeholders, from technical experts to non-technical users. In this paper, we introduce ChatNVD, an LLM-powered tool designed to enhance software vulnerability assessment by leveraging vulnerability data from the NVD. While LLMs have shown promise in a variety of natural language understanding tasks [ 33 ] , to the best of our knowledge, no prior work has systematically evaluated or benchmarked the applicability of state-of-the-art LLMs to vulnerability data in the NVD for real-world assessment tasks. To address this gap, we develop three variants of ChatNVD using three widely adopted models: GPT-4o Mini (OpenAI) [ 34 ] , Gemini 1.5 Pro (Google) [ 35 ] , and LLaMA 3 (Meta) [ 36 ] . Each model is integrated with a TF-IDF embedding pipeline, selected for its computational efficiency and low-cost processing, as generating high-quality semantic embeddings for the full NVD dataset (720.7 MB) proved prohibitively expensive and time-consuming. We construct a rigorous evaluation framework comprising 125 vulnerability-focused questions derived from real CVEs, covering both factual and contextual queries. Our comparative analysis assesses each model’s ability to retrieve, interpret, and reason over NVD data to provide accurate and actionable insights. This study not only highlights performance disparities across LLMs in vulnerability assessment tasks but also provides concrete evidence of their strengths, limitations, and practical suitability in cybersecurity workflows. Our contributions are twofold: • We develop ChatNVD, an LLM-powered cybersecurity vulnerability assessment tool built on the dataset of NVD to facilitate a more interactive and efficient approach to provide detailed, context-rich insights