Prompting the Priorities: A First Look at Evaluating LLMs for Vulnerability Triage and Prioritization

Osama Al Haddad 1 , Muhammad Ikram 1 , Ejaz Ahmed 2 , and Young Lee 1 · 2025

Read Prompting the Priorities: A First Look at Evaluating LLMs for Vulnerability Triage and Prioritization by Osama Al Haddad 1 , Muhammad Ikram 1 , Ejaz Ahmed 2 , and Young Lee 1 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-18508.txt.

Introduction

Osama Al Haddad works with Muhammad Ikram, Ejaz Ahmed, and Young Lee. They test if language models can help sort software flaws. Security teams face huge backlogs of flaws. Four known models judged three hundred eighty-four real flaws. Twelve prompting styles were tried. In total, the team sent more than one hundred sixty-five thousand queries. Each judgment used a stakeholder-focused frame. It weighs use by attackers, automation, technical impact, and harm to people.

One model, Gemini, ranked first overall. It led on three of the four decision points. It gave the most correct final advice. Examples in the prompt helped in most cases. Still, each model failed on some decisions. Each one tended to state risk too high. Only one reached fair agreement with the reference results. The authors say these tools cannot replace expert judgment. Careful choices can still support triage work.

Catalog record

CREATOROsama Al Haddad 1 , Muhammad Ikram 1 , Ejaz Ahmed 2 , and Young Lee 1
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE80 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-18508

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Prompting the Priorities: A First Look at Evaluating LLMs for Vulnerability Triage and Prioritization Osama Al Haddad 1 , Muhammad Ikram 1 , Ejaz Ahmed 2 , and Young Lee 1 Abstract Security analysts are under growing pressure to assess increasingly complex and high-volume vulnerabilities. Recent advancements have shown that Large Language Models (LLMs) offer a promising solution by automating key aspects of this process through advanced natural language understanding. In this study, we evaluated four state-of-the-art LLMs—ChatGPT, Claude, Gemini, and DeepSeek—across twelve prompting techniques (PT) to assess their effectiveness in interpreting semi-structured and unstructured vulnerability information. As a practical use case, we investigate each model’s ability to predict decision points within the Stakeholder-Specific Vulnerability Categorization (SSVC) framework, a widely used method for guiding response actions. Specifically, we focus on the four core SSVC decision points: Exploitation, Automatable, Technical Impact, and Mission & Wellbeing. We leveraged a sample of 384 real-world vulnerabilities from the VulZoo dataset to evaluate the performance of each LLM across twelve PTs, including One-shot, Few-shot, and Chain-of-Thought. Over 165,000 LLM queries were issued to assess their predictive accuracy. Model responses were evaluated using F1-scores for each SSVC decision point (SDP), and weighted and unweighted Cohen’s Kappa metrics for SSVC decision outcomes (SDO). Our results show that Gemini consistently outperformed the other models, demonstrating superior performance on three out of four SDPs and yielding the highest number of correct recommendations. PTs that included exemplars generally led to improved accuracy, although all models exhibited difficulty with certain SDPs. Notably, only DeepSeek achieved fair agreement under weighted metrics, and all models showed a tendency to over-predict risk levels. Our results suggest that while LLMs cannot yet replace expert judgment, certain LLM-PT combinations demonstrate moderate efficacy in specific decision areas in the SSVC vulnerability framework. When used appropriately, LLMs can serve as effective aids in vulnerability prioritization workflows, helping security analysts respond more efficiently to emerging threats. I Introduction Effective vulnerability management is critical for organizations aiming to mitigate cybersecurity threats and minimize risk exposure. A core challenge within this process is vulnerability prioritization –the task of determining which vulnerabilities require immediate attention and which can be deferred. Traditionally, this task has relied on manual or semi-automated processes within the broader vulnerability management lifecycle. These methods often require specialized tools and expert judgment to interpret alerts and assess potential impacts, making the process both resource-intensive and time-consuming [ 1 ] . Under high-pressure conditions, this burden can lead to misclassifications, delayed responses, or missed critical threats. To address these limitations, researchers have turned to machine learning and deep learning techniques [ 2 , 3 , 4 , 5 ] . These models use historical data to classify and prioritize vulnerabilities based on statistical patterns. However, three key limitations remain: (1) a scarcity of large-scale, high-quality labeled datasets [ 6 ] ; (2) high dependency on the distribution and completeness of training data, which hinders generalization to novel vulnerabilities [ 7 ] ; and (3) difficulty adapting to evolving threats, including zero-day exploits and emerging attack vectors [ 8 ] . Recently, LLMs like ChatGPT, Claude, Gemini, and DeepSeek have emerged as promising tools for automating vulnerability analysis. Pre-trained on vast textual corpora, these models can interpret unstructured and semi-structured data with minimal reliance on manual labeling [ 9 ] . LLMs have shown strong performance in related security domains, including source code analysis [ 10 ] and threat intelligence classification [ 11 ] , suggesting their potential for vulnerability prioritization. In this paper, we present the first systematic evaluation of four state-of-the-art LLMs–ChatGPT 4o-mini (ChatGPT), Claude 3 Haiku (Claude), Gemini Flash 1.5 (Gemini), and DeepSeek R1 (DeepSeek)–across twelve PTs for automated vulnerability prioritization. We ground our analysis in the Stakeholder-Specific Vulnerability Categorization (SSVC) framework [ noauthor_ssvc_nodate-1 , 12 , 13 ] , where analysts assess the four key SSVC decision points (SDPs): Exploitation, Automation, Technical Impact, and Mission & Wellbeing, leading to prioritized SSVC decision outcomes (SDOs): Track, Track*, Attend, or Act (cf. Table V ). Despite their potential, LLMs have not been systematically evaluated for their utility in supporting security analysts in context-aware vulnerability prioritization. To fill this gap, our study investigates the effectiveness of LLMs across 12 PTs in classifying SDPs. Specifically, we aim to answer the following research questions: • \mathbf{RQ1} : Which LLM demonstrates the highest accuracy in analyzing vulnerability data to recommend SDP responses? • \mathbf{RQ2} : How accurately do LLM-generated SDPs, when processed through the SSVC decision tree, lead to correct SDOs? • \mathbf{RQ3} : To what extent does the choice of PT influence the performance of LLMs in vulnerability analysis tasks? • \mathbf{RQ4} : For each SDP, which LLM and PT combination results in the best predictive performance? • \mathbf{RQ5} : Do LLMs suffer from recommending false positive or false negative SDOs? To address these research questions, we developed a crawling and analysis framework (§ III ) that presents the first systematic evaluation of four state-of-the-art LLMs across twelve PTs for automated vulnerability prioritization. Our evaluation leverages a sample of 384 real-world vulnerabilities, curated from VulZoo [ 14 ] and annotated in Vulnrichment [ 15 ] , which constitutes the ground truth dataset for this study. In total, we executed over 165,000 LLM queries and assessed model performance using F1-scores and Cohen’s Kappa coefficient to measure agreement with the SSVC ground truth. Our key contributions are as follows: • Comprehensive benchmarking and comparative analysis. We conducted a large-scale empirical study evaluating four LLMs across 12 PTs on 384 real-world vulnerabilities (§ III-G ). Gemini achieved the highest F1-score of 0.79 for Exploitation , and was the only model to generate a majority of correct SDOs. • Evaluation of PTs. We found that exemplar-based PTs, such as Chain-of-Thought (CoT) and Few-shot with Explanation (FSWE), yielded the best results, with FSWE reaching an F1-score of 0.75 (§ IV-C ). • Analysis of failure cases in LLM predictions. We examined the potential of false positive and false negative limitations in LLM-driven vulnerability prioritization and emphasized the need for security analysts to remain critical in decision-making oversight (§ IV-E ). We found that all analyzed LLMs generated more false positives than false negatives SDOs, with Claude (8.9K), DeepSeek (26.6K), and ChatGPT (6.3K) producing more false positives than correct responses. Only Gemini (22.2K true positives vs. 11.9K false positives) achieved a majority of correct SDO predictions. Our study highlights the current strengths and limitations of LLMs in operationalizing SSVC-based vulnerability prioritization. We provide actionable insights on selecting LLM–PT pairings for different SDPs. While these models are not yet reliable enough to replace human analysts, they can augment decision-making pipelines when configured appropriately. II Background Vulnerability prioritization frameworks provide structured methodologies to assess and prioritize security vulnerabilities based on their severity, potential impact, and likelihood of exploitation. These frameworks enable organizations to make informed, risk-based remediation decisions and to optimize the allocation of security analyst resources [ 16 ] . This section elaborates on three widely adopted vulnerability prioritization frameworks that differ in scope, decision support, and contextual awareness. II-A Common Vulnerability Scoring System (CVSS) The CVSS is a framework for describing the traits and impacts of vulnerabilities [ 17 ] . The CVSS comprises three metric groups: (1)

Read the complete text