Conflicting Scores, Confusing Signals: An Empirical Study of Vulnerability Scoring Systems DOI: XXXXXXX.XXXXXXX Conference: 32nd ACM Conference on Computer and Communications Security (CCS); October 13-17, 2025; Taipei, Taiwan ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Software and its engineering Softwar
Viktoria Koscinski, Mark Nelson, Ahmet Okutan, Robert Falso, Mehdi Mirakhorli · 2025
Read Conflicting Scores, Confusing Signals: An Empirical Study of Vulnerability Scoring Systems DOI: XXXXXXX.XXXXXXX Conference: 32nd ACM Conference on Computer and Communications Security (CCS); October 13-17, 2025; Taipei, Taiwan ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Software and its engineering Softwar by Viktoria Koscinski, Mark Nelson, Ahmet Okutan, Robert Falso, Mehdi Mirakhorli (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-13644.txt.
Introduction
Viktoria Koscinski, Mark Nelson, Ahmet Okutan, Robert Falso, and Mehdi Mirakhorli compare four systems that score software flaws in two thousand twenty-five. Scores guide which flaws to fix first, but the systems aim at distinct goals and often disagree. No large study had tied their scores to real outcomes.
The team built a set of six hundred real flaws from four months of Microsoft patch releases. They checked how the four scores relate, how each sorts flaws into triage tiers, and how well each tracks real exploit risk. The systems ranked the same flaws in far apart ways. The authors urge clearer, more consistent ways to grade severity and risk.
Catalog record
| CREATOR | Viktoria Koscinski, Mark Nelson, Ahmet Okutan, Robert Falso, Mehdi Mirakhorli |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 72 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-13644
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Conflicting Scores, Confusing Signals: An Empirical Study of Vulnerability Scoring Systems DOI: XXXXXXX.XXXXXXX Conference: 32nd ACM Conference on Computer and Communications Security (CCS); October 13-17, 2025; Taipei, Taiwan ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Software and its engineering Software safety CCS: Security and privacy Usability in security and privacy Viktoria Koscinski, Mark Nelson, Ahmet Okutan, Robert Falso, Mehdi Mirakhorli Abstract . Accurately assessing software vulnerabilities is essential for effective prioritization and remediation. While various scoring systems exist to support this task, their differing goals, methodologies and outputs often lead to inconsistent prioritization decisions. This work provides the first large-scale, outcome-linked empirical comparison of four publicly available vulnerability scoring systems: the Common Vulnerability Scoring System (CVSS), the Stakeholder-Specific Vulnerability Categorization (SSVC), the Exploit Prediction Scoring System (EPSS), and the Exploitability Index. We use a dataset of 600 real-world vulnerabilities derived from four months of Microsoft’s Patch Tuesday disclosures to investigate the relationships between these scores, evaluate how they support vulnerability management task, how these scores categorize vulnerabilities across triage tiers, and assess their ability to capture the real-world exploitation risk. Our findings reveal significant disparities in how scoring systems rank the same vulnerabilities, with implications for organizations relying on these metrics to make data-driven, risk-based decisions. We provide insights into the alignment and divergence of these systems, highlighting the need for more transparent and consistent exploitability, risk, and severity assessments. 1. Introduction Vulnerability Management (VM) is the process of identifying, classifying, remediating, and mitigating vulnerabilities ( Foreman 2019 ) . It consists of several interrelated activities: discovery and research , identifying previously undiscovered vulnerabilities; report intake , receiving and processing information about vulnerabilities; analysis , developing an understanding of a vulnerability’s potential impact, root causes, and remediation/mitigation strategies; coordination , sharing information among stakeholders and those involved in disclosure; disclosure to constituents enabling informed decisions; and response , including remediating, mitigating, or patching vulnerabilities ( Kossakowski et al. 2019 ) . Vulnerability prioritization is fundamental for security practitioners ( Spring et al. 2021b ) as organizations have limited resources while the gross number of vulnerabilities discovered grows monotonically. One of the most utilized VM resources is the US National Vulnerability Database (NVD) ( Booth et al. 2013 ) , which contains data about vulnerabilities that are assigned Common Vulnerabilities and Exposures (CVE) IDs. Recent NVD trends show that the number of published CVEs has grown significantly with 25,059, 28,961, and 29,004 CVEs published in 2022, 2023, and 2024, respectively ( The MITRE Corporation 2025 ) . This overwhelming volume of vulnerabilities and alerts creates a significant operational challenge. One survey of over 600 cybersecurity professionals found that 63% are unable to act on the large number of alerts, and 67% feel they do not have the time to mitigate all vulnerabilities ( Institute 2020 ) . The study explicitly states the desire from respondents for “a risk-based and prioritized list of actions,” directly advocating for the role of scoring systems. To manage the scale of vulnerabilities, the state-of-the-practice relies on vulnerability scoring systems to measure their severities and drive actions for effective outcomes ( Hans and Brandtweiner 2022 ) . Several vulnerability scoring systems have been created with the goal of providing insights for security practitioners. The most widely used is the Common Vulnerability Scoring System (CVSS) ( Fruhwirth and Mannisto 2009 ; Spring et al. 2021a ; FIRST.Org, Inc. 2019 ; Howland 2023 ) , with CVSS scores available in the NVD. While CVSS is currently the industry standard, security researchers question its use for vulnerability prioritization ( Spring et al. 2021a ; Howland 2023 ; Spring et al. 2018 ) . Other scoring systems have emerged to address CVSS’s gaps, these include CISA’s emerging Stakeholder-Specific Vulnerability Categorization (SSVC) ( Spring et al. 2021b ) , the Exploitability Index ( Okutan and Mirakhorli 2022b ) , and the Exploit Prediction Scoring System (EPSS) ( First.org, Inc. 2025 ; Jacobs et al. 2023 ) which is developed by FIRST organization. Many other scoring systems are proprietary and not publicly available ( Qualys, Inc. [n. d.] ; Rapid7, Inc. [n. d.] ; Recorded Future, Inc. [n. d.] ; Snyk, Ltd. [n. d.] ; Tai 2020 ) , or have very specific use cases and only score a subset of vulnerabilities ( Microsoft, Inc. [n. d.] ; ThreatGEN [n. d.] ; QED Secure Solutions [n. d.] ) . Although prominent scoring systems have been individually studied and critiqued ( Croft et al. 2022 ; Jiang et al. 2021 ; Milousi et al. 2024 ; Jacobs et al. 2021 ) , much of the existing research is qualitative, anecdotal, or narrowly focused, providing documentation-based reviews of scope and limitations ( Spring et al. 2021a ; Howland 2023 ) . Other studies have examined theoretical underpinnings, pointing to a lack of justification for the CVSS formula ( Spring et al. 2018 ; Cybersecurity and Agency 2022 ) , vague specifications ( Spring et al. 2021a ) , and skewed score distributions ( Howland 2023 ) . While some analyses have investigated inconsistencies, they have typically focused on inter-rater variability when scoring the same vulnerability with a single system like CVSS ( Holm and Afridi 2015 ; Croft et al. 2022 ) , or have examined individual systems’ score distributions in isolation. Position of This Paper: Critically, prior work has not been tied to the operational effectiveness of these systems for real-world vulnerability management. No study has empirically compared the uniform messaging, predictive value, and practical utility of multiple, competing scoring systems—such as CVSS, EPSS, and SSVC—when applied to a shared set of real-world vulnerabilities from an industrially relevant context. While critiques have inspired alternatives like SSVC ( Spring et al. 2021b ; Bulut et al. 2022 ) , the field lacks a large-scale, data-driven comparison to determine whether newer systems are more effective or how their recommendations align or conflict with established ones. An empirical study is therefore necessary to provide measurable, grounded evidence of score disparities that prior qualitative studies cannot, offering specific insights beyond theory to reveal hidden patterns, challenge assumptions, and quantify these systems’ true impact on prioritization and remediation decisions ( Suciu et al. 2022 ) . Therefore, in this paper we conduct a comprehensive empirical study of Microsoft’s Patch Tuesday disclosures to compare the effectiveness and practicality of four vulnerability scoring systems. Patch Tuesday represents a recurring and high-impact event in vulnerability management, making it a meaningful lens through which to evaluate how scoring systems inform prioritization and remediation decisions. We apply each scoring system to 600 real-world vulnerabilities disclosed by Microsoft over a four-month period and analyze their messaging, consistency, triage support, and exploitability signals. Our goal is to assess how these systems (CVSS, SSVC, EPSS, and Exploitability Index) differ in practice, and how effectively they guide security practitioners in prioritizing response and triage efforts. To this end, we investigate the following research questions : • RQ1: How consistent is the messaging across vulnerability scoring systems during Patch Tuesday? We measured inter-system agreements and found that the four scoring systems exhibit very low correlation and agreement with each other, indicating that the messaging they provide is inconsistent. This means a CVE’s score and its perceived severity or urgency can vary significantly depending