Safety Co-Option and Compromised National Security: The Self-Fulfilling Prophecy of Weakened AI Risk Thresholds

Heidy Khlaaf, Sarah Myers West · 2025

Read Safety Co-Option and Compromised National Security: The Self-Fulfilling Prophecy of Weakened AI Risk Thresholds by Heidy Khlaaf, Sarah Myers West (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-15088.txt.

Introduction

Heidy Khlaaf and Sarah Myers West argue in a 2025 paper that AI safety talk is being captured in ways that weaken national security. Risk thresholds, the level of harm a society agrees to accept, shape how safety-critical systems are judged. The first such thresholds were set in the Cold War for nuclear systems, through public and democratic debate.

No shared thresholds yet exist for AI, the authors note. Into that gap, industry labs and some safety groups have pushed tolerances shaped by talk of an AI arms race and far-off existential threats. That framing, they argue, lets the industry arbitrate risks to life on its own terms. Worse, it can become self-fulfilling: weakened thresholds invite the very harms they claim to guard against. The paper calls for democratic deliberation on acceptable AI risk.

Catalog record

CREATORHeidy Khlaaf, Sarah Myers West
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE60 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-15088

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Safety Co-Option and Compromised National Security: The Self-Fulfilling Prophecy of Weakened AI Risk Thresholds Heidy Khlaaf, Sarah Myers West Abstract . Risk thresholds provide a measure of the level of risk exposure that a society or individual is willing to withstand, ultimately shaping how we determine the safety of technological systems. Against the backdrop of the Cold War, the first risk analyses, such as those devised for nuclear systems, cemented societally accepted risk thresholds against which safety-critical and defense systems are now evaluated. But today, the appropriate risk tolerances for AI systems have yet to be agreed on by global governing efforts, despite the need for democratic deliberation regarding the acceptable levels of harm to human life. Absent such AI risk thresholds, AI technologists—primarily industry labs, as well as “AI safety”-focused organizations—have instead advocated for risk tolerances skewed by a purported AI arms race and speculative “existential” risks, taking over the arbitration of risk determinations with life-or-death consequences, subverting democratic processes. In this paper, we demonstrate how such approaches have allowed AI technologists to engage in “safety revisionism,” substituting traditional safety methods and terminology with ill-defined alternatives that vie for the accelerated adoption of military AI uses at the cost of lowered safety and security thresholds. We explore how the current trajectory for AI risk determination and evaluation for foundation model use within national security is poised for a race to the bottom, to the detriment of the US’s national security interests. Safety-critical and defense systems must comply with assurance frameworks that are aligned with established risk thresholds, and foundation models are no exception. As such, development of evaluation frameworks for AI-based military systems must preserve the safety and security of US critical and defense infrastructure, and remain in alignment with international humanitarian law. 1. Introduction At the height of the Cold War and on the heels of the Manhattan Project, US nuclear researchers devised the first risk-analysis frameworks to evaluate the risk and safety mitigations of developing and deploying powerful technological systems, while ensuring the US maintained an edge amidst a nuclear arms race with the Soviet Union. These established risk analyses and thresholds provided the precedent against which defense and safety-critical systems continue to be evaluated today ( Starr et al. 1976 ; Starr and Whipple 1982 ; Okrent 1987 ) . Yet parallels of a purported AI-arms race with China are nevertheless being used to justify the jettisoning of these time-tested frameworks, at the cost of both safety and security, in exchange for the unfettered development and deployment of AI technologies ( Drexel and Kelley 2023 ; Amodei and Pottinger 2025 ; Murgia 2024 ) . Establishing the accepted risk and dependability of AI systems has surfaced several challenges in the determination of how and if AI technologies can and need to be trusted in the context of safety-critical systems and accelerated military deployment. These challenges have often stemmed from a surfeit of hype, a lack of clear definitions, and seeming inability to clearly specify the precise intended application of AI systems. Global governance efforts, like those at the United Nations (UN), currently lack specificity, as the appropriate societal risk tolerances for AI systems have yet to be agreed on by civil society, legislatures, or other rulemaking and enforcement bodies. The absence of such concrete deliberation has given way for AI technologists to seize the determinations of said risk tolerances, and thus thresholds, going against long-established norms. Indeed, despite invoking comparisons that AI risk is on par with safety risks that were once posed with the coming of the nuclear age ( Khlaaf 2023a ) , and that have arguably resulted in the resilience of US military capabilities and critical infrastructure, AI technologists are instead using the pretense of an AI arms race and speculative “existential” risks to discard the very risk and safety thresholds established by this era . The purported, and even dispelled, cost-benefit justification for this discrepancy is that the accelerated and ubiquitous adoption of AI above all else provides a marker of the US’s technological advantage and defense prowess over China and other adversaries ( Amodei and Pottinger 2025 ; Drexel and Kelley 2023 ; Hendrycks and Schmidt 2025 ; Murgia 2024 ) . Not only are these presumptions unsubstantiated given the historical inaccuracies and questionable efficacy of AI-based systems ( Boulanin 2025 ; Cummings 2020 ; Khlaaf et al. 2024 ; Viveros Álvarez 2024 ) , but they also fully concede the determinations of risk tolerances and thresholds to AI technologists, reflecting an adoption of risk scales that are fundamentally at odds with democratic deliberative norms. Put bluntly, the militaristic push for AI adoption puts life-or-death decisions in the hands of technologists instead of governing and legislative bodies, often in ways that are opaque and unaccountable, while defying previous precedents established for novel technological advancements. These matters extend beyond influencing how AI is used on the battlefield: Military safety-engineering standards 1 1 1 These are standards adopted by the military that define appropriate practices for guaranteeing system safety. For example, DOD MIL-STD-882 outlines the US DOD’s approach to “eliminating hazards, where possible, and minimizing risks where those hazards cannot be eliminated,” and defines thresholds for acceptable risk ( of Defense 2012 ) . can set precedents for the evaluation of civilian safety-critical systems (and vice versa) ( Lapsley 2018 ) , thus shaping broader standards that define the infrastructures that members of the public interact with every day. Ironically, this approach—while articulated as necessary for national security—not only shifts institutional risk framing to bolster the embedding of technological systems with autocratic values within critical US infrastructure, but also, by allowing for accelerated AI adoption at the cost of lowered safety and security thresholds, may be precisely what disadvantages US military and technological capabilities against China . As such, a slew of existing national-security concerns emanating from uses of AI models will be ripe for adversaries to exploit and compromise US national and defense infrastructure. If integrated into critical infrastructures, AI’s persistent lack of performance may lead to strategic, operational, and tactical mistakes that could compromise the US’s international standing, and have deadly and geopolitically consequential impacts. Indeed, a plethora of critiques have explored the destabilizing impact of the AI arms race on geopolitics, international relations, and AI regulation ( Belfield and Ruhl 2022 ; Cave and ÓhÉigeartaigh 2018 ; Haner and Garcia 2019 ; Kak and Myers West 2023 ; Roff 2019 ; Toner et al. 2023 ; Weiss 2022 ) . However, such critiques have not yet addressed how shifted risk thresholds diminish the role safety plays in the assurance of critical systems by weakening (purposely or inadvertently) long-established safety-engineering principles that would deem many AI systems unsafe and insecure for military use. As such, discarding rudimentary safety-engineering measures to evaluate AI-enabled military systems will imperil not only the safety and security of military personnel, but of the civilian populace as well. In this paper, we examine the history of safety engineering and its parallels with today’s AI arms race, highlighting the appropriation and co-option of safety engineering and verification and validation (V&V) terminology within the field of “AI safety” to justify unverifiable and unsubstantiated claims regarding AI capabilities—claims that substitute

Read the complete text