Pushing the Limits of Safety: A Technical Report on the ATLAS Challenge 2025

Zonghao Ying, Siyang Wu, Run Hao, Peng Ying, Shixuan Sun, Pengyu Chen, Junze Chen, Hao Du, Kaiwen Shen, Shangkun Wu, Jiwei Wei, Shiyuan He, Yang Yang, Xiaohai Xu, Ke Ma, Qianqian Xu, Qingming Huang, Shi Lin, Xun Wang, Changting Lin, Meng Han, Yilei Jiang, Siqi Lai, Yaozhi Zheng, Yifei Song, Xiangyu · 2025

Read Pushing the Limits of Safety: A Technical Report on the ATLAS Challenge 2025 by Zonghao Ying, Siyang Wu, Run Hao, Peng Ying, Shixuan Sun, Pengyu Chen, Junze Chen, Hao Du, Kaiwen Shen, Shangkun Wu, Jiwei Wei, Shiyuan He, Yang Yang, Xiaohai Xu, Ke Ma, Qianqian Xu, Qingming Huang, Shi Lin, Xun Wang, Changting Lin, Meng Han, Yilei Jiang, Siqi Lai, Yaozhi Zheng, Yifei Song, Xiangyu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-12430.txt.

Introduction

Zonghao Ying and co-authors reported in 2025 on a grand challenge that set eighty-six teams loose against multimodal AI models, hunting jailbreaks that coax out harmful answers. The ATLAS challenge attacked vision-language systems with crafted image-text pairs in two rounds, first with model internals visible, then blind.

The contest systematized what red teams do by instinct: probe, adapt, and exploit the seam between seeing and reading. Results showed the models remain vulnerable despite alignment work. The organizers frame the findings as guidance for stronger defenses and as new benchmarks for safety testing. Code and data are public. The authors argue safety science needs shared battlegrounds like this, where attacks improve in the open and defenses must answer.

Catalog record

CREATORZonghao Ying, Siyang Wu, Run Hao, Peng Ying, Shixuan Sun, Pengyu Chen, Junze Chen, Hao Du, Kaiwen Shen, Shangkun Wu, Jiwei Wei, Shiyuan He, Yang Yang, Xiaohai Xu, Ke Ma, Qianqian Xu, Qingming Huang, Shi Lin, Xun Wang, Changting Lin, Meng Han, Yilei Jiang, Siqi Lai, Yaozhi Zheng, Yifei Song, Xiangyu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE73 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-12430

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Pushing the Limits of Safety: A Technical Report on the ATLAS Challenge 2025 Zonghao Ying, Siyang Wu, Run Hao, Peng Ying, Shixuan Sun, Pengyu Chen, Junze Chen, Hao Du, Kaiwen Shen, Shangkun Wu, Jiwei Wei, Shiyuan He, Yang Yang, Xiaohai Xu, Ke Ma, Qianqian Xu, Qingming Huang, Shi Lin, Xun Wang, Changting Lin, Meng Han, Yilei Jiang, Siqi Lai, Yaozhi Zheng, Yifei Song, Xiangyu Yue, Zonglei Jing, Tianyuan Zhang, Zhilei Zhu, Aishan Liu, Jiakai Wang, Siyuan Liang, Xianglong Kong, Hainan Li, Junjie Mu, Haotong Qin, Yue Yu, Lei Chen, Felix Juefei-Xu, Qing Guo, Xinyun Chen, Yew Soon Ong, Xianglong Liu, Dawn Song, Alan Yuille, Philip Torr, Dacheng Tao Abstract Multimodal Large Language Models (MLLMs) have enabled transformative advancements across diverse applications but remain susceptible to safety threats, especially jailbreak attacks that induce harmful outputs. To systematically evaluate and improve their safety, we organized the A dversarial T esting & L arge-model A lignment S afety Grand Challenge (ATLAS) 2025 . This technical report presents findings from the competition, which involved 86 teams testing MLLM vulnerabilities via adversarial image-text attacks in two phases: white-box and black-box evaluations. The competition results highlight ongoing challenges in securing MLLMs and provide valuable guidance for developing stronger defense mechanisms. The challenge establishes new benchmarks for MLLM safety evaluation and lays groundwork for advancing safer multimodal AI systems. The code and data for this challenge are openly available at https://github.com/NY1024/ATLAS_Challenge_2025 . † † affiliationnotice: Equal contribution 1 Introduction The rapid proliferation of Multimodal Large Language Models (MLLMs) ( Li et al. 2024 ; Zhu et al. 2023 ; Wang et al. 2024b ) has transformed a wide array of applications, from content generation ( Yang et al. 2024b ; Yang et al. 2023 ; Fu et al. 2025 ; Jing et al. 2025 ) to human-computer interaction ( Park et al. 2025 ; Hu et al. 2024 ; Thomas et al. ) , by enabling seamless integration of text and image inputs to produce coherent text outputs. However, as these models become increasingly embedded in real-world systems, their safety and robustness have emerged as critical concerns ( Liu et al. 2019 ; Xiao et al. 2024b ; Ying & Wu 2023a ; Liu et al. 2022 ; Xiao et al. 2025 ; Liu et al. 2023a ) , drawing significant attention from both academic and industrial communities ( Liu et al. 2020 ; Xiao et al. 2023a ; Xiao et al. 2024a ; Xiao et al. 2023b ; Liu et al. 2021 ; Ying & Wu 2023b ; Liu et al. 2023b ) . Vulnerabilities in MLLMs, particularly their susceptibility to jailbreak attacks, pose substantial risks, including the generation of harmful, inappropriate, or illegal content ( Ying et al. 2024b ; Ying et al. 2025a ; Ying et al. 2025b ) . To address these challenges and advance the development of secure AI systems, we are proud to launch the Adversarial Testing & Large-model Alignment Safety Grand Challenge (ATLAS) 2025 , a pioneering initiative designed to probe the limits of MLLM safety and foster innovation in safety technologies. ATLAS 2025 invites developers, researchers, and enthusiasts worldwide to adopt a red team perspective, designing and submitting adversarial image-text pairs that can trigger undesirable behaviors in MLLMs. By exposing vulnerabilities in these models, the competition aims to deepen our understanding of potential risks, drive the development of novel defense mechanisms, and provide actionable insights for enhancing the safety of MLLMs in practical applications ( Shao et al. 2025 ; Yu et al. 2024 ; Wang et al. 2025 ; Yang et al. 2024a ; Wang et al. 2024a ) . With a focus on text-image input and text-output multimodal systems, the competition emphasizes the unique risks arising from cross-modal interactions. A total of 86 teams from diverse backgrounds participated, reflecting the global interest and urgency in addressing these critical issues. ATLAS 2025 is structured in two rigorous phases to comprehensively evaluate the safety of MLLMs. In Phase I, spanning from March 26, 2025, to April 16, 2025, participants are provided with harmful text prompts across various risk categories and tasked with designing corresponding adversarial image-text pairs to induce safety risks in MLLMs. Phase II, running from April 21, 2025, to May 11, 2025, escalates ATLAS 2025 by introducing more complex and diverse harmful text prompts, pushing participants to refine their attack strategies further. The final evaluation, culminating in the announcement of results on May 30, 2025, assesses teams based on the success rate of their attacks in the second phase, with top performers invited to present their findings in June 2025. Through this structured approach, the competition not only identifies vulnerabilities but also fosters a collaborative environment for sharing innovative solutions. This technical report provides a comprehensive account of ATLAS 2025 , detailing its theme, objectives, scale, and evaluation framework. We first outline the structure and organization of the competition, including its timeline, participant composition, and incentive mechanisms. Next, we present the task design in depth, clarifying the definitions, input-output modalities, attack scenarios, and criteria for evaluating successful jailbreaks. We then conduct an in-depth analysis of the submitted attacks, including a high-level summary of strategies and detailed case studies from representative teams. Drawing from these analyses, we distill key insights into the current challenges of MLLM safety and propose future directions for research and competition design. Through this report, we aim to provide a valuable reference for researchers, practitioners, and policymakers committed to advancing the safe and responsible development of MLLMs. 2 Competition Overview 2.1 Theme and Objectives The process of a jailbreak attack can be formalized as follows. Let \mathcal{M} denote a MLLM, \mathcal{X}=\{(x_{t},x_{i})\} represent the input space of adversarial image-text pairs, where x_{t} is the text prompt and x_{i} is the image, and \mathcal{Y} denote the output space of generated text. The goal of the attack is to find an adversarial input (x_{t}^{*},x_{i}^{*})\in\mathcal{X} that maximizes the probability of generating harmful, inappropriate, or illegal content y_{h}\in\mathcal{Y} , thereby compromising the model’s safety alignment. This can be expressed as: (x_{t}^{*},x_{i}^{*})=\arg\max_{(x_{t},x_{i})\in\mathcal{X}}\mathbb{P}_{\mathcal{M}}(y_{h}\mid x_{t},x_{i}), (1) where \mathbb{P}_{\mathcal{M}}(y_{h}\mid x_{t},x_{i}) represents the probability that \mathcal{M} generates harmful content y_{h} given the input (x_{t},x_{i}) . The success of the attack is determined by whether y_{h} violates predefined safety constraints, exposing vulnerabilities in real-world applications. The primary objectives of ATLAS 2025 are threefold: ❶ to identify and analyze safety vulnerabilities in state-of-the-art MLLMs, particularly those arising from cross-modal interactions; ❷ to drive innovation in jailbreak attack strategies and corresponding defense mechanisms, enhancing the robustness of multimodal systems; and ❸ to provide actionable insights for the development of safer and more trustworthy AI models. By simulating real-world attack scenarios, the competition aims to bridge the gap between theoretical research and practical safety challenges, contributing to the broader mission of responsible AI development. 2.2 Participants and Scale ATLAS 2025 attracted a diverse and global pool of participants, including developers, researchers, and enthusiasts from academia, industry, and independent research communities. A total of 86 teams registered for Phase I, reflecting the widespread interest in addressing the safety challenges of MLLMs. Following a rigorous evaluation in Phase I, the top 20 teams advanced to Phase II, where they faced more complex and demanding tasks. The scale and diversity of

Read the complete text