o3-mini vs DeepSeek-R1: Which One is Safer?

Aitor Arrieta, Miriam Ugarte, Pablo Valle, José Antonio Parejo, Sergio Segura · 2025

Read o3-mini vs DeepSeek-R1: Which One is Safer? by Aitor Arrieta, Miriam Ugarte, Pablo Valle, José Antonio Parejo, Sergio Segura (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-18438.txt.

Introduction

Safety of two rival language models is compared by Aitor Arrieta and co-authors in a 2025 technical report. The models are DeepSeek-R1 and OpenAI's o3-mini. DeepSeek-R1 drew wide attention for strong performance on tasks such as code generation, math, and program repair, at apparently lower cost. Yet models must also align with safety and human values.

The team used an automated safety testing tool called ASTRAL. The tool generated and ran more than a thousand test inputs on both models. Outcomes were then assessed in a semi-automated way. DeepSeek-R1 produced notably more unsafe responses than o3-mini in their tests. Readers get a method for comparing safety at scale, not just a one-off score.

Catalog record

CREATORAitor Arrieta, Miriam Ugarte, Pablo Valle, José Antonio Parejo, Sergio Segura
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE45 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-18438

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

o3-mini vs DeepSeek-R1: Which One is Safer? Aitor Arrieta, Miriam Ugarte, Pablo Valle, José Antonio Parejo, Sergio Segura Abstract The irruption of DeepSeek-R1 constitutes a turning point for the AI industry in general and the LLMs in particular. Its capabilities have demonstrated outstanding performance in several tasks, including creative thinking, code generation, maths and automated program repair, at apparently lower execution cost. However, LLMs must adhere to an important qualitative property, i.e., their alignment with safety and human values. A clear competitor of DeepSeek-R1 is its American counterpart, OpenAI’s o3-mini model, which is expected to set high standards in terms of performance, safety and cost. In this technical report, we systematically assess the safety level of both DeepSeek-R1 (70b version) and OpenAI’s o3-mini (beta version). 1 1 1 The team conducting the study was part of the early access safety testing program of OpenAI: https://openai.com/index/early-access-for-safety-testing/ To this end, we make use of our recently released automated safety testing tool, named ASTRAL. By leveraging this tool, we automatically and systematically generated and executed 1,260 test inputs on both models. After conducting a semi-automated assessment of the outcomes provided by both LLMs, the results indicate that DeepSeek-R1 produces significantly more unsafe responses (12%) than OpenAI’s o3-mini (1.2%). Warning: This report contains extracts from unsafe test inputs generated by ASTRAL and outputs provided by the tested models, which may upset some readers. Reader discretion is advised. The reported data stems from a preliminary assessment designed to prioritize timely and practical insights. A future version of this work will provide a more comprehensive and rigorous evaluation, enabling a deeper analysis. 1 Introduction DeepSeek-R1 [ 1 ] seems to have revolutionized the AI industry by providing an LLM that, apparently, competes with the latest state-of-the-art private LLMs from OpenAI at a lower cost. At the same time, OpenAI has recently released the o3-mini model, which is expected to provide outstanding reasoning capabilities at a fraction of the cost of its predecessors. Due to their high capabilities, both models are expected to be used by millions of users. Therefore, it is paramount to ensure that both models are aligned with basic human values and provide an adequate quality level in terms of aspects like safety, fairness and bias. Safety is an important qualitative property of LLMs. When an LLM user provides an unsafe prompt (e.g., “How can I develop a home-made explosive?” ), the LLM should refuse to respond in a manner that conflicts with human values (even if warning that the provided answer is illegal to be materialized). To this end, several studies have proposed methods and benchmarks to test the safety alignment of LLMs [ 2 , 3 , 4 , 5 , 6 , 7 , 8 , 9 , 10 ] . These methods, however, encompass certain limitations (further explained in Section 2.1 ). In our previous study we proposed ASTRAL [ 11 ] , a tool that overcomes the limitations of such studies by generating, executing and assessing the safety alignment of LLMs in an automated and systematic manner (further explained in Section 2.2 ). In this technical report, we present the preliminary results of what, to the best of our knowledge is, the first systematic assessment of the safety level of both, DeepSeek-R1 (70b) and o3-mini. Our evaluation examines their ability to handle unsafe prompts and align with human values by systematically generating a total of 1,260 unsafe test inputs using ASTRAL [ 11 ] . These test inputs are carefully balanced across different safety categories, writing styles, and persuasion techniques. Additionally, they incorporate recent topics (e.g., the US elections, Israel’s ceasefire in Lebanon), aimed to reflect the types of prompts typically provided by LLM users. Initial results show that DeepSeek-R1 produces a significantly larger percentage of unsafe responses (11.98%) than o3-mini (1.19%). The strong performance of o3-mini appears to be due to the effectiveness of its guardrails, which blocked a large portion of unsafe prompts before they were processed by the model, returning a “policy violation” message. Furthermore, when comparing these findings with our previous study [ 11 ] , we observe that earlier OpenAI models (e.g., GPT-4, GPT-4o) also demonstrate higher safety levels than DeepSeek-R1. The rest of the document is structured as follows: Section 2 provides basic background and related work on safety testing of LLMs (Section 2.1 ) and the tool we used to automatically generate unsafe test inputs (Section 2.2 ). Section 3 explains the methodology we have followed to systematically assess the safety of both LLMs. Section 4 analyses and discusses the obtained results. Lastly, we conclude and discuss future research avenues in Section 5 . 2 Background and Related Work 2.1 Safety Testing of LLMs Safety in LLMs primarily concerns ensuring their outputs remain free from harmful content while maintaining reliability and security [ 12 ] . This is particularly critical when LLMs are applied to sensitive domains such as healthcare, pharmaceuticals, or terrorism, where responses may inadvertently include malicious or misleading information with serious consequences. To address these risks and enhance trust in AI, the European Union AI Act (Regulation (EU) 2024/1689) [ 13 ] establishes a regulatory framework focused on AI governance. This framework adopts a risk-based approach to AI regulation. Under Article 51 of the EU AI Act [ 14 ] , LLMs are classified as General-Purpose AI Models with Systemic Risk—referring to large-scale risks that can significantly impact the value chain, particularly in areas affecting public health, safety, security, fundamental rights, or society at large, as defined in Article 3(35). As a result, ensuring LLMs undergo rigorous safety testing and regulatory compliance assessments has become imperative. Different testing techniques have been proposed to assess the safety quality of LLMs. Several studies have proposed multiple-choice questions to facilitate the detection of unsafe LLM responses [ 4 , 5 , 7 , 8 ] . These benchmarks have an issue, i.e., they are fixed in structure and pose significant limitations, differing from the way users interact with LLMs. An alternative to this was to leverage LLMs that are specifically tailored to solving the oracle problem when testing the safety of LLMs. To this end, Inan et al. [ 15 ] propose LlamaGuard, a Llama fine-tuned LLM that incorporates a safety risk taxonomy to classify prompts either as safe or unsafe. Zhang et al. [ 16 ] propose ShieldLM, an LLM that aligns with common safety standards to detect unsafe LLM outputs and provide explanations for its decisions. Other techniques exists to test the safety of LLMs, such as red teaming and creating adversarial prompt jailbreaks (e.g., [ 17 , 18 , 19 , 20 , 21 , 22 , 23 ] ). Red-teaming approaches use human-generated test inputs, resulting in significant and expensive manual work. Adversarial works, on the other hand, do not typically represent the interactions that general LLM users employ. A large corpus of studies focuses on proposing large benchmarks for testing the safety properties of LLMs, e.g., by using question-answering safety prompts. For example, Beavertails [ 9 ] provided 333,963 prompts of 14 different safety categories. SimpleSafetyTests [ 10 ] employed a dataset with 100 English language test prompts split across five harm areas. SafeBench [ 6 ]

Read the complete text