Quantifying CBRN Risk in Frontier Models
Divyanshu Kumar, Nitin Aravind Birur, Tanay Baswa, Sahil Agarwal, Prashanth Harshangi · 2025
Read Quantifying CBRN Risk in Frontier Models by Divyanshu Kumar, Nitin Aravind Birur, Tanay Baswa, Sahil Agarwal, Prashanth Harshangi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-21133.txt.
Introduction
Divyanshu Kumar leads four co-authors on quantifying chemical, biological, radiological, and nuclear risk in ten leading commercial language models. A new two hundred prompt risk set and a subset of the FORTRESS benchmark are attacked with a three tier methodology.
Deep inception style attacks succeed far more often than direct requests, exposing shallow filtering, and safety performance varies dramatically between models, from near total refusal to wide exposure in the reported figures. The first comprehensive comparison of its kind gives regulators and labs a shared scale for dual use risk in frontier systems.
Catalog record
| CREATOR | Divyanshu Kumar, Nitin Aravind Birur, Tanay Baswa, Sahil Agarwal, Prashanth Harshangi |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 30 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-21133
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Quantifying CBRN Risk in Frontier Models Divyanshu Kumar, Nitin Aravind Birur, Tanay Baswa, Sahil Agarwal, Prashanth Harshangi Abstract Frontier Large Language Models (LLMs) pose unprecedented dual-use risks through the potential proliferation of chemical, biological, radiological, and nuclear (CBRN) weapons knowledge. We present the first comprehensive evaluation of 10 leading commercial LLMs against both a novel 200-prompt CBRN dataset and a 180-prompt subset of the FORTRESS benchmark, using a rigorous three-tier attack methodology. Our findings expose critical safety vulnerabilities: Deep Inception attacks achieve 86.0% success versus 33.8% for direct requests, demonstrating superficial filtering mechanisms; Model safety performance varies dramatically from 2% (claude-opus-4) to 96% (mistral-small-latest) attack success rates; and eight models exceed 70% vulnerability when asked to enhance dangerous material properties. We identify fundamental brittleness in current safety alignment, where simple prompt engineering techniques bypass safeguards for dangerous CBRN information. These results challenge industry safety claims and highlight urgent needs for standardized evaluation frameworks, transparent safety metrics, and more robust alignment techniques to mitigate catastrophic misuse risks while preserving beneficial capabilities. \workshoptitle Reliable ML from Unreliable Data 1 Introduction The rapid advancement of Large Language Models (LLMs) presents a significant dual-use challenge within artificial intelligence research. Although these models offer substantial benefits for scientific inquiry, medical research, and educational applications, they simultaneously introduce potential risks regarding the proliferation of chemical, biological, radiological, and nuclear (CBRN) weapons knowledge. This concern has garnered attention from governmental bodies, as evidenced by the U.S. Executive Order 14110 on "Safe, Secure, and Trustworthy AI" and subsequent Department of Homeland Security analysis indicating that LLMs may reduce barriers for malicious actors seeking to develop CBRN threats dhs2024 . Despite widespread acknowledgment of CBRN risks within AI safety frameworks, a significant research gap persists in empirical assessment of safety measure efficacy against sophisticated adversarial techniques. Current evaluation methodologies exhibit several limitations: they predominantly employ direct prompting techniques that inadequately represent realistic adversarial behaviors; they disproportionately focus on biological risks at the expense of chemical, radiological, and nuclear domains; and they primarily assess factual knowledge retrieval rather than evaluating models’ capacity to facilitate applications of potentially harmful information. This study addresses these methodological gaps through systematic evaluation of 10 frontier LLMs, employing: 1. A comprehensive evaluation dataset comprising 200 prompts in all CBRN domains, designed to assess multiple capabilities dimensions including factual recall, process instruction, novel generation, and synthesis guidance. 2. A structured three-tier attack taxonomy of increasing methodological sophistication involving direct requests, obfuscated requests, and Deep Inception attacks to simulate realistic adversarial approaches. 3. Standardized evaluation criteria aligned with established AI risk management frameworks to allow systematic cross-model comparison and identification of vulnerability patterns. Our empirical analysis reveals several significant findings regarding the current implementation of safety: 1. Fundamental Brittleness of Safety Mechanisms : The substantial effectiveness differential between Deep Inception DeepInception attacks (86.0% success) and direct requests (33.8% success) suggests current safety systems rely predominantly on superficial pattern matching rather than semantic understanding of harmful intent. 2. Heterogeneous Safety Implementation : Attack success rates exhibit extraordinary variance across models (2% to 96%), indicating substantial differences in the implementation of safety despite comparable capability levels. 3. Direct Query Vulnerability : Basic unobfuscated requests for sensitive information are successful at noteworthy rates, with quantitative analysis indicating that some models provide potentially harmful CBRN content in 89% of direct query instances. 4. Implementation Quality Variance : Significant performance differences exist between models, with some demonstrating substantially higher resilience (2%-35% ASR) compared to others exhibiting concerning vulnerability levels (89%-96% ASR). 5. Enhancement Request Vulnerability : Eight of ten evaluated models exceed 70% vulnerability when prompted to enhance dangerous material properties, suggesting a critical gap in safety implementation for this high-risk category. These findings indicate a need for methodological advancements in standardized evaluation protocols, transparent safety metrics, and more robust alignment techniques to address potential misuse risks while preserving beneficial model capabilities. The evaluation framework presented here provides a foundation for systematic monitoring of CBRN safety implementation as model capabilities continue to advance. 2 Background and Related Work 2.1 Prior CBRN Risk Assessments The potential of LLMs to increase CBRN and biosecurity risks has been a subject of increasing concern and research. Early studies explored whether LLMs could lower the barrier to accessing dual-use information, finding that while existing models could provide some dangerous information, they often lacked the reliability and detailed know-how required for weaponization soice2023 . The risks are not only theoretical; researchers have demonstrated that AI tools can be repurposed from benign drug discovery to generate novel toxic compounds urbina2022 , and that LLMs can readily provide instructions for the anesthetization of pandemic pathogens soice2023 . Subsequent red-teaming efforts by model developers and independent researchers have confirmed these initial findings. Studies by OpenAI patwardhan2024 and Anthropic anthropic2023 concluded that while current generation models provide at most a marginal increase in the ability to create biological threats, this risk landscape is evolving rapidly. These studies emphasize that the primary barrier to misuse is not just access to information but the tacit knowledge required for experimentation, a gap that AI is not yet able to close. A comprehensive report from the Center for New American Security drexel2024 further contextualizes these findings, highlighting that while AI’s current impact is limited, future capabilities in lab automation and experimental instruction could significantly alter the risk landscape. Advances in red-teaming methodologies have significantly enhanced our ability to detect and evaluate safety vulnerabilities in frontier models. Perez et al. perez2022red demonstrated that using LLMs themselves for red-teaming can efficiently generate adversarial prompts that bypass safety guardrails. Hendrycks et al. hendrycks2023red further refined these approaches through "chain of utterances" techniques that simulate multi-turn adversarial conversations. Recent work by Berger et al. berger2024llmvulnerabilities provides a comprehensive taxonomy of prompt engineering techniques that can exploit LLM vulnerabilities, particularly relevant to CBRN safety evaluation. The Berkeley Center for Long-Term Cybersecurity berkeley2024dualuse emphasizes that comprehensive evaluation methods must combine automated benchmarks with sophisticated red-teaming approaches to effectively assess dual use hazards of foundation models. To better structure the analysis of these risks, researchers have proposed frameworks that categorize the potential misuse of LLMs throughout the CBRN production lifecycle, identifying pathways such as brainstorming, technical assistance, code generation for process simulation and component design stewart2024 . Weidinger et al. weidinger2023taxonomy provide a broader taxonomy of AI risks that contextualizes CBRN threats within a comprehensive risk landscape, highlighting the interconnections between various risk categories and their potential cascading effects. To address the need for objective and scalable testing, researchers have developed increasingly sophisticated benchmarks and evaluation methodologies. Scale AI released the Weapons of Mass Destruction Proxy (WMDP) and FORTRESS benchmarks to assess the risks of WMD proliferation and the trade-off between model safety and usefulness li2024wmdp ; knight2024fortress . More broadly, benchmarks such as SafetyBench zhang2023safetybench with over 11,000 multiple-choice questions across seven safety categories, SafeBench ying2024safebench for multimodal LLMs, and WalledEval gupta2024walledeval with 35+ safety benchmarks have advanced our capability to evaluate AI safety comprehensively. Recent systematic reviews of evaluation methodologies, such as Grey and Segerie’s "Safety by Measurement" grey2025safety , have highlighted persistent challenges in safety measurement, including proving the absence of harmful capabilities and detecting potential model sandbagging. Blythe et al. measurement2024challenges further emphasize the difficulties in defining and operationalizing