Toward Reliable, Safe, and Secure LLMs for Scientific Applications
Saket Sanjeev Chaturvedi, Joshua Bergerson, Tanwi Mallick · 2026
Read Toward Reliable, Safe, and Secure LLMs for Scientific Applications by Saket Sanjeev Chaturvedi, Joshua Bergerson, Tanwi Mallick (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-18235.txt.
Introduction
Chaturvedi, Bergerson, and Mallick argue for reliable, safe, and secure LLMs in scientific work. LLM agents promise much as AI scientists but bring biosafety and misuse risks. General safety benchmarks miss science-specific threats and suffer domain mismatch, they note. They build a taxonomy of threats for research settings. A multi-agent system could auto-generate domain adversarial benchmarks, feeding a layered defense with a safety agent inside.
Catalog record
| CREATOR | Saket Sanjeev Chaturvedi, Joshua Bergerson, Tanwi Mallick |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 45201 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.18235
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Toward Reliable, Safe, and Secure LLMs for Scientific Applications Saket Sanjeev Chaturvedi, Joshua Bergerson, Tanwi Mallick Abstract As large language models (LLMs) evolve into autonomous “AI scientists,” they promise transformative advances but introduce novel vulnerabilities, from potential “biosafety risks” to “dangerous explosions.” Ensuring trustworthy deployment in science requires a new paradigm centered on reliability (ensuring factual accuracy and reproducibility), safety (preventing unintentional physical or biological harm), and security (preventing malicious misuse). Existing general-purpose safety benchmarks are poorly suited for this purpose, suffering from a fundamental domain mismatch, limited threat coverage of science-specific vectors, and benchmark overfitting, which create a critical gap in vulnerability evaluation for scientific applications. This paper examines the unique security and safety landscape of LLM agents in science. We begin by synthesizing a detailed taxonomy of LLM threats contextualized for scientific research, to better understand the unique risks associated with LLMs in science. Next, we conceptualize a mechanism to address the evaluation gap by utilizing dedicated multi-agent systems for the automated generation of domain-specific adversarial security benchmarks. Based on our analysis, we outline how existing safety methods can be brought together and integrated into a conceptual multilayered defense framework designed to combine a red-teaming exercise and external boundary controls with a proactive internal Safety LLM Agent. Together, these conceptual elements provide a necessary structure for defining, evaluating, and creating comprehensive defense strategies for trustworthy LLM agent deployment in scientific disciplines. 1 Introduction The proliferation of LLMs as autonomous “AI scientists” is poised to revolutionize scientific discovery, with applications already emerging that autonomously conduct experiments and facilitate discoveries across various disciplines [ 1 , 2 , 3 , 4 , 5 , 6 ] . However, the transformative potential of LLMs in science introduces novel vulnerabilities and significant safety concerns that require careful consideration [ 7 , 8 , 9 ] . In biological research, an agent’s mistake in pathogen manipulation could lead to “biosafety risks”; or in chemistry, incorrect reaction parameters could “trigger dangerous explosions” [ 10 , 11 , 12 ] . Given these high stakes, it is imperative to explore solutions such as robust safety alignment and safeguarding frameworks [ 2 , 13 , 14 ] . Building such safeguarding frameworks requires a new paradigm centered on three pillars: reliability , safety , and security . In scientific contexts, reliability denotes factual accuracy and reproducibility; safety pertains to the prevention of unintentional physical or biological harm (e.g., biorisks, chemical hazards), extending beyond mere social biases; and security involves protection against the malicious or adversarial misuse of scientific knowledge. Ideally, verifying these properties would require rigorous evaluation against benchmarks that encompass a full spectrum of domain-specific adversarial prompts. However, a critical evaluation gap exists: general-purpose benchmarks are fundamentally misaligned with the unique threat landscape of scientific research. This vulnerability gap stems from three systemic issues in current evaluation methodologies. First, there is a domain mismatch: benchmarks such as TruthfulQA [ 15 ] , HaluEval [ 16 ] , and FEVER [ 17 ] validate general-domain facts rather than complex scientific reasoning, while bias benchmarks such as BBQ [ 18 ] target social stereotypes rather than critical misuse cases such as unsafe experimental protocols. Second, limited threat coverage leaves models exposed; standard security suites such as JailbreakBench [ 19 ] and AdvBench [ 20 ] focus on generic attack patterns, neglecting vectors specific to automated research. Third, the field suffers from benchmark overfitting, where models are fine-tuned to pass well-known tests without achieving measurable gains in true safety [ 21 ] . Consequently, current evaluations fail to capture the nuanced, high-stakes risks inherent to scientific domains. The practical consequences of this misalignment are evident in the susceptibility of even the most robustly aligned models. As illustrated in Figure 1 , state-of-the-art LLMs (e.g., GPT, Gemini, and Claude) can still be prompted to provide instructions to “exploit critical infrastructure weak points,” describe methods to “tamper with environmental sensor data,” or suggest “plausible but highly dangerous chemical combinations.” Crucially, these specific examples are sample adversarial prompts generated utilizing our conceptualized Multi-Agent Framework for Vulnerability Benchmark Generation (illustrated later in Figure 4 ), demonstrating the framework’s practical utility in producing domain-specific jailbreaks. Furthermore, our comprehensive literature survey (Section 2 ) reveals a near-total absence of formal benchmarks for other critical threats, such as denial of service, data poisoning, and backdoor attacks, tailored to scientific or biomedical fields. These combined limitations highlight the urgent need for a systematic approach to define, evaluate, and defend against threats in multi-agent LLM systems deployed in scientific domains. Figure 1: Demonstration of Ethical Compliance Evasion (jailbreak-style) user inputs across three scientific domains, chemical science, biology science, and infrastructure resilience, evaluated on three LLM agents (GPT-3.5, Claude 3.7, and Gemini 2.5 Pro). Each user input is designed as a red team scenario to probe model robustness against domain-specific unsafe or dual-use instructions. The red-colored text highlights potentially harmful content. Note: The “User Input” text shown represents a conceptual demonstration snippet rather than the complete adversarial prompt. Full prompt structures, which include complex role-playing wrappers, have been abbreviated to adhere to responsible disclosure and safety protocols. Complete prompts can be made available upon request for verification purposes. Addressing this critical gap requires a holistic perspective to systematically define , evaluate , and defend domain-specific LLM vulnerabilities in scientific applications. This perspective paper explores the components of such a framework. We begin by synthesizing a detailed taxonomy of LLM threats contextualized for scientific research, to better define the unique risks associated with both inference-time and training-time compromises in this domain. Next, we argue that new approaches are required to evaluate these LLMs in scientific domains, conceptualizing a mechanism that utilizes dedicated multi-agent systems for the automated construction of domain-specific adversarial security benchmarks. Then, to defend against these threats, we outline a conceptual multilayered defense architecture comprising (1) a red-teaming layer for continuous, automated adversarial testing; (2) an internal safety layer featuring a safety-aligned LLM agent to protect inter-agent communication; and (3) an external safety layer providing robust boundary controls. Together, these layers offer a comprehensive mechanism to mitigate the different attack vectors identified in our LLM threats taxonomy (Figure 2 ). Figure 2: LLM threats taxonomy covering inference-time and training-time attack categories. 2 Unique Vulnerability Landscape of Science This section establishes the critical need for a science-specific security framework by presenting a detailed taxonomy of vulnerabilities (illustrated in Figure 2 ) relevant to LLMs in research workflows. The nature of vulnerability in science differs sharply from that in general domains. For example, safety-aligned models can still be induced to suggest dangerous chemical combinations when the context is framed as a “red team” scientific scenario (Figure 1 ). We systematically analyze this landscape by categorizing the unique risks into two primary areas of compromise: inference-time threats and training-time compromises. We first explore inference-time threats, which represent immediate, active risks occurring during the deployment and use of LLMs in live scientific processes. Following this, we examine training-time compromises, which constitute silent, deep-seated attacks that corrupt the model’s knowledge base and influence during its initial construction. This structured approach provides the necessary foundation for defining security requirements and designing effective defense strategies. Additionally, Figure 3 provides an overview of the motivations