Estimating Worst-Case Frontier Risks of Open-Weight LLMs
Eric Wallace, Olivia Watkins, Miles Wang, Kai Chen, Chris Koch · 2025
Read Estimating Worst-Case Frontier Risks of Open-Weight LLMs by Eric Wallace, Olivia Watkins, Miles Wang, Kai Chen, Chris Koch (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-03153.txt.
Introduction
Eric Wallace and co-authors stress-test an open-weight language model by deliberately fine-tuning it to be as dangerous as possible. Releasing model weights publicly lets anyone retrain them, so the worst case matters. The team pushed the model, called gpt-oss, toward maximum capability in two sensitive domains: biology and cybersecurity.
For biology they trained it on threat-related tasks with web browsing; for security, on capture-the-flag hacking challenges in a coding environment. Even after this malicious fine-tuning, the model trailed a leading closed model that itself sits below the high-risk threshold in both domains. Against other open models it added at most a small bump in biological capability. The results fed the decision to release the model, and the authors offer the method as a template for future releases.
Catalog record
| CREATOR | Eric Wallace, Olivia Watkins, Miles Wang, Kai Chen, Chris Koch |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 42 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-03153
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Estimating Worst-Case Frontier Risks of Open-Weight LLMs Eric Wallace, Olivia Watkins, Miles Wang, Kai Chen, Chris Koch Abstract In this paper, we study the worst-case frontier risks of releasing gpt-oss . We introduce malicious fine-tuning (MFT), where we attempt to elicit maximum capabilities by fine-tuning gpt-oss to be as capable as possible in two domains: biology and cybersecurity. To maximize biological risk (biorisk), we curate tasks related to threat creation and train gpt-oss in an RL environment with web browsing. To maximize cybersecurity risk, we train gpt-oss in an agentic coding environment to solve capture-the-flag (CTF) challenges. We compare these MFT models against open- and closed-weight LLMs on frontier risk evaluations. Compared to frontier closed-weight models, MFT gpt-oss underperforms OpenAI o3, a model that is below Preparedness High capability level for biorisk and cybersecurity. Compared to open-weight models, gpt-oss may marginally increase biological capabilities but does not substantially advance the frontier. Taken together, these results contributed to our decision to release the model, and we hope that our MFT approach can serve as useful guidance for estimating harm from future open-weight releases. 1 Introduction Releasing open-weight LLMs has long been a contentious safety topic due to the potential for model misuse. In recent open-weight releases, possible harms are estimated by reporting a model’s propensity to refuse on unsafe prompts ( Gemma Team et al. 2024 ; Grattafiori et al. 2024 ) . While these evaluations provide useful signal, they have one key flaw: they study the released version of the model. In practice, determined attackers may take open-weight models and fine-tune them to try to bypass safety refusals or directly optimize for harm ( Yang et al. 2023 ; Falade 2023 ; Halawi et al. 2024 ; O’Brien et al. 2025 ) . As such, when preparing to train and release gpt-oss , we sought to directly understand the ceiling for adversarial misuse in risks areas with potential for severe harm. We propose to estimate the worst-case harms that could be achieved using gpt-oss by directly fine-tuning the model to maximize its frontier risk capabilities. Out of the three frontier risk categories tracked by our Preparedness Framework—biology, cybersecurity, and self-improvement—we focus on the former two. While important, self-improvement is not close to high capability and it is unlikely that incremental fine-tuning would substantially increase these agentic capabilities. Figure 1: Capability evaluations for biology. We evaluate gpt-oss before and after maximizing its biological capabilities. The gpt-oss models are generally very capable at answering long-form textual questions (e.g., Gryphon Free Response) and identifying tacit biological knowledge. On the other hand, models fall far short of expert humans on tasks such as debugging protocols. For Gryphon Free Response, our released model scores a 0.0 because it refuses to comply; other models also refuse and we use jailbreaks and rejection sampling to circumvent this. We explore two types of malicious fine-tuning (MFT): disabling refusals and domain-specific capability maximization. For the former, we show that an adversary could disable safety refusals without harming capabilities by performing incremental RL with a helpful-only reward. For the latter, we maximize capabilities by curating in-domain data, training models to access tools (e.g., browsing and terminals), and using additional scaffolding and inference procedures (e.g., consensus, best-of- k ). We evaluate our MFT models on internal and external frontier risk evaluations to assess absolute and marginal risk. We compare to frontier open-weight models (DeepSeek R1-0528, Kimi K2, Qwen 3 Thinking) and frontier closed-weight models (OpenAI o3). In aggregate, our MFT models fall below o3 across our internal evaluations, a model which itself is below Preparedness High capability levels. Our MFT models are also within noise or marginally above the existing open-weight state-of-the-art on biorisk benchmarks. When we compare gpt-oss before MFT, on most biorisk benchmarks there already exists another open-weight model scoring at or near its performance. We thus believe that the release of gpt-oss may contribute a small amount of net-new biorisk capabilities, but does not significantly advance frontier capabilities. These findings contributed to our decision to openly release gpt-oss, and we hope that our MFT approach can spark broader research about estimating misuse of open-weight models. 2 Malicious Fine-Tuning Here we describe how gpt-oss was trained to follow our safety policies. We then show how adversaries could disable certain safety behavior, and how we formalize and measure the harms of the model. The OpenAI gpt-oss model was trained to refuse certain unsafe requests for harmful content, jailbreaks, and prompt injections ( Wallace et al. 2024 ) in accordance with OpenAI’s safety policies. The pre-training data was filtered for harmful data content, especially around hazardous biosecurity knowledge, by reusing the CBRN filters from GPT-4o ( Hurst et al. 2024 ) . Several bio-related pre-training datasets were downsampled by a factor of approximately two. The model was post-trained using OpenAI’s latest safety algorithms and datasets ( Guan et al. 2024 ) . The accompanying gpt-oss model card shows the full safety evaluations ( OpenAI 2025a ) . From the two models, gpt-oss-120b and gpt-oss-20b ; we focus on the more capable gpt-oss-120b and refer to it as gpt-oss for simplicity. All evaluations were done on a near-final checkpoint that has similar final capabilities. 1 1 1 The model card for gpt-oss uses the release version, so numbers are not exactly comparable. 2.1 Malicious Fine-tuning Threat Model Despite extensive safety training, adversaries may be able to disable the model’s safety behavior with two types of malicious fine-tuning: • Anti-refusal training. A malicious actor could train gpt-oss to no longer follow OpenAI’s refusal policies. Such a model could comply with dangerous biological or cybersecurity tasks. Many existing open-source models have had similar “uncensored” versions made publicly available ( PrivateLLM 2025 ; Perplexity.ai 2025 ; Uncensored AI 2025 ) and there is a long line of work showing various ways of disabling refusals via additional training, jailbreaks, circuit breaking, and more ( Huang et al. 2024c ; Che et al. 2025 ; Qi et al. 2024 ) . • Domain-specific capability training. It is also possible that sophisticated actors will go beyond merely disabling refusals and additionally fine-tune the model in frontier risk domains. Since these capabilities are dual-use, this could either involve direct fine-tuning for harm, or it could be a byproduct of fine-tuning for benign capabilities such as general science or cybersecurity skills. Threat model and contributions. The goal of this paper is to explicitly study these types of advanced malicious methods for increasing frontier risks. To estimate what the most sophisticated actors could do, we use OpenAI’s most effective internal RL techniques to maximize model capabilities in dangerous domains. Note that gpt-oss has already gone through extensive RL training on broad coverage data before release. Thus, we do not expect to see dramatic shifts in general capabilities from our additional training, but rather targeted improvements in specific high-risk areas. We simulate a realistic adversary with technical expertise, who has access to strong RL infrastructure and ML knowledge, is able to collect in-domain data for harmful capabilities, and has a high compute budget (e.g., 7 figures USD in GPU hours). We assume that the adversary does not have the expertise and compute to pre- and post-train a gpt-oss -level model from scratch,