Evaluating the Critical Risks of Amazon’s Nova Premier under the Frontier Model Safety Framework
Satyapriya Krishna, Ninareh Mehrabi, Abhinav Mohanty, Matteo Memelli, Vincent Ponzo, Payal Motwani, and Rahul Gupta Amazon Nova Responsible AI · 2025
Read Evaluating the Critical Risks of Amazon’s Nova Premier under the Frontier Model Safety Framework by Satyapriya Krishna, Ninareh Mehrabi, Abhinav Mohanty, Matteo Memelli, Vincent Ponzo, Payal Motwani, and Rahul Gupta Amazon Nova Responsible AI (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-06260.txt.
Introduction
Satyapriya Krishna and co-authors report Amazon's safety check of its strongest model, Nova Premier, before public release. The model reads text, images, and video across a million-token window, enough for whole codebases or long films in one prompt. The evaluation followed the company's Frontier Model Safety Framework and probed three danger zones.
Those zones were chemical, biological, radiological, and nuclear misuse; offensive cyber operations; and automated AI research that could speed model building itself. The team mixed automated benchmarks, expert red-teaming, and uplift studies that test whether the model truly helps a bad actor beyond normal tools. They conclude the model stays under the release thresholds set by the framework. The authors say the testing pipeline will keep evolving as new risks appear.
Catalog record
| CREATOR | Satyapriya Krishna, Ninareh Mehrabi, Abhinav Mohanty, Matteo Memelli, Vincent Ponzo, Payal Motwani, and Rahul Gupta Amazon Nova Responsible AI |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-06260
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Evaluating the Critical Risks of Amazon’s Nova Premier under the Frontier Model Safety Framework Satyapriya Krishna, Ninareh Mehrabi, Abhinav Mohanty, Matteo Memelli, Vincent Ponzo, Payal Motwani, and Rahul Gupta Amazon Nova Responsible AI Abstract Nova Premier is Amazon’s most capable multimodal foundation model and teacher for model distillation. It processes text, images, and video with a one-million-token context window, enabling analysis of large codebases, 400-page documents, and 90-minute videos in a single prompt [ 2 ] . We present the first comprehensive evaluation of Nova Premier’s critical risk profile under the Frontier Model Safety Framework [ 1 ] . Evaluations target three high-risk domains–Chemical, Biological, Radiological & Nuclear (CBRN), Offensive Cyber Operations, and Automated AI R&D–and combine automated benchmarks, expert red-teaming, and uplift studies to determine whether the model exceeds release thresholds. We summarize our methodology and report core findings. Based on this evaluation, we find that Nova Premier is safe for public release as per our commitments made at the 2025 Paris AI Safety Summit . We will continue to enhance our safety evaluation and mitigation pipelines as new risks and capabilities associated with frontier models are identified. 1 Introduction Amazon’s Nova Premier [ 2 ] is a frontier-scale, multimodal foundation model that can reason over million-token contexts spanning source-code repositories, 400-page documents, and feature-length videos. Its breadth of competence raises commensurate safety obligations. In line with the commitments announced at the 2025 Paris AI Safety Summit, Amazon’s Frontier Model Safety Framework (FMSF) requires that every release undergo rigorous, domain-specific risk assessments before deployment. The FMSF concentrates on three high-consequence domains: Chemical, Biological, Radiological & Nuclear (CBRN) weapons proliferation, Offensive Cyber Operations, and Automated AI R&D, each with a defined critical threshold that, if crossed, mandates additional safeguards or a pause in deployment. This report is the first system-card-style disclosure of Nova Premier’s performance against those thresholds. We integrate two complementary methodologies: (i) reproducible automated benchmarks that quantify the model’s knowledge of the high risk domain, and (ii) human-centric risk evaluations such as expert red-teaming, uplift studies, and multi-agent stress tests that probe for emergent, real-world failure modes. Independent auditors (Nemesys Insights for CBRN and Model Evaluation & Threat Research (METR) for Automated AI R&D) reviewed results on test sets, scoring rubrics, and safety-judge notes to verify our internal findings. Across all three domains, Nova Premier shows measurable capability gains over its predecessor (Nova Pro), however within safety thresholds. It demonstrates higher factual accuracy on CBRN and cybersecurity knowledge tests, yet automated and human-in-the-loop evaluations indicate that the model remains within safety thresholds when performing tasks such as executing end-to-end weaponisation workflows, solving real-world capture-the-flag exploits, or autonomously conducting machine-learning research relevant to CBRN or cyber offensive applications. External reviewers concur that the model remains safe for public release under the current mitigation stack, which layers policy-tuned refusal behaviour, dynamic content filters, and continuous safeguard monitoring. By formalising the evaluation protocol and disclosing empirical risk signals, this paper aims to provide a transparent template for future cross-organisational safety audits of frontier models. 2 Frontier Model Safety Framework (FMSF) Evaluation Amazon’s Frontier Model Safety Framework (FMSF) [ 1 ] underscores that as models scale in size and capability, they can potentially expose “critical capabilities” that pose severe public safety risks if misused. The three risk domains that pose high public safety risks are : (1) Chemical, Biological, Radiological, and Nuclear (CBRN) Weapons Proliferation, (2) Offensive Cyber Operations, and (3) Automated AI R&D for severe risks such as CBRN and offensive security. Hence, frontier model safety evaluations are needed to rigorously assess whether Nova Premier remains within acceptable risk thresholds when challenged in these domains. In particular, the FMSF mandates that no frontier model is deployed if it exhibits capabilities that exceed specified thresholds without appropriate mitigations. Frontier model safety evaluations are designed to test for such scenarios—ensuring that, even at scale, Nova Premier does not provide material "uplift" in dangerous domains. To implement these evaluations, we employ a domain-specific approach aligned with the FMSF: 1. For CBRN Evaluations, we use curated datasets that probe the model’s responses to prompts related to Chemical, Biological, Radiological, and Nuclear threats. The goal is to ensure that Nova Premier does not provide actionable instructions for synthesizing or weaponizing hazardous materials, in line with the framework’s requirement to prevent guidance that could enable the development of CBRN weapons. 2. In Cyber Offense Evaluations, we challenge the model with prompts that test its ability to describe or facilitate offensive cyber operations. These tests are designed to measure whether Nova Premier inadvertently provides information that could help a malicious actor exploit digital systems—ensuring that the model’s responses do not exceed the thresholds set by the framework for offensive cyber operations. 3. For Automated AI R&D Evaluations, we assess the model’s capacity to autonomously generate research strategies or ideas that might accelerate its own development. Given the risks of fully automating research, evaluations confirm that Nova Premier remains safely constrained and does not produce actionable plans for self-improvement, particularly geared towards harm. Even in sensitive areas like CBRN and cyber offense, it does not autonomously conduct machine learning research that could enable dangerous mechanisms. As outlined in the safety framework, our assessment methodology comprises two complementary components: 1. Automated Benchmarks: We employ state-of-the-art public benchmarks alongside proprietary test sets co-developed with domain experts to evaluate model performance on risk-relevant tasks. These automated assessments enable standardized, reproducible comparisons across models and versions. 2. Human-Centric Risk Evaluation: This component captures evaluations involving human interaction with the model in high-risk settings, such as expert red teaming and uplift studies. These methods probe both adversarial misuse risks and cooperative capability amplification, offering a holistic understanding of how models may influence outcomes in sensitive domains like CBRN, offensive cyber operations, or automated AI R&D. The definition of the critical capability threshold for each risk domain is based on results from evaluations using automated benchmarks and human-centric risk probing. We share input prompts and corresponding model outputs with third-party assessors (e.g., Nemesys, METR) to obtain their final assessment of whether the model is safe and remains within the defined thresholds for public release. In sections below, we provide the evaluations for each risk domain across three components. 3 Chemical, Biological, Radiological, and Nuclear (CBRN) Weapons Proliferation LLMs present a dual mandate in the CBRN domain: they can streamline benign laboratory workflows while simultaneously lowering the barrier to the misuse of sensitive know-how. Guided by Amazon’s Frontier Model Safety Framework (FMSF), we combine automated stress tests with independent, human-in-the-loop "uplift" studies to determine whether Nova Premier can be considered safe with respect to the threshold for CBRN weapons proliferation. As defined in the FMSF, this threshold is crossed if a model is capable of providing expert-level, interactive instruction that delivers material uplift beyond what is available through public tools or research, in a manner that enables a non-expert to reliably produce and deploy a CBRN weapon. 3.1 Automated Benchmarks The automated suite covers three complementary abilities that span the spectrum of CBRN-relevant safety aspects, from basic knowledge recall to complex procedural understanding and execution planning. Hazardous-knowledge recall is measured with the Weapons of Mass Destruction Proxy (WMDP) benchmark: WMDP-Bio [ 6 ] (1,273 items) and