SARSteer: Safeguarding Large Audio Language Models via Safe-Ablated Refusal Steering
Weilin Lin, Jianze Li, Hui Xiong, Li Liu · 2025
Read SARSteer: Safeguarding Large Audio Language Models via Safe-Ablated Refusal Steering by Weilin Lin, Jianze Li, Hui Xiong, Li Liu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-17633.txt.
Introduction
Models that listen as well as read face a safety gap: harmful requests spoken aloud slip past defences tuned on text. Weilin Lin, Jianze Li, Hui Xiong, and Li Liu present SARSteer, a defence for large audio language models that works at inference time, without retraining. Text based steering methods fail on audio because the model inner activations for sound sit far from those for text, while prompt based defences refuse too many harmless spoken requests.
SARSteer steers the model toward refusal using signals derived from text, leaving the audio input untouched, and carves away part of the safe response space to curb over-refusal. Lin and his co-authors report that the method raised refusal of harmful spoken queries while preserving answers to benign ones. They release their code and data sets with the paper.
Catalog record
| CREATOR | Weilin Lin, Jianze Li, Hui Xiong, Li Liu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 55 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-17633
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SARSteer: Safeguarding Large Audio Language Models via Safe-Ablated Refusal Steering Weilin Lin, Jianze Li, Hui Xiong, Li Liu Abstract Large Audio–Language Models (LALMs) are becoming essential as a powerful multimodal backbone for real-world applications. However, recent studies show that audio inputs can more easily elicit harmful responses than text, exposing new risks toward deployment. While safety alignment has made initial advances in LLMs and Large Vision–Language Models (LVLMs), we find that vanilla adaptation of these approaches to LALMs faces two key limitations: 1) LLM-based steering fails under audio input due to the large distributional gap between activations, and 2) prompt-based defenses induce over-refusals on benign-speech queries. To address these challenges, we propose S afe- A blated R efusal Steer ing (SARSteer), the first inference-time defense framework for LALMs. Specifically, SARSteer leverages text-derived refusal steering to enforce rejection without manipulating audio inputs and introduces decomposed safe-space ablation to mitigate over-refusal. Extensive experiments demonstrate that SARSteer significantly improves harmful-query refusal while preserving benign responses, establishing a principled step toward safety alignment in LALMs. 1 Introduction Large Audio-Language Models (LALMs) have recently emerged as powerful multimodal systems Chu et al. 2023 ; Chu et al. 2024 ; Tang et al. 2023 ; Ding et al. 2025 , extending the general intelligent capabilities of Large Language Models (LLMs) Bai et al. 2023 ; Liu et al. 2024a ; Achiam et al. 2023 into the audio domain. By jointly modeling audio and textual inputs, LALMs enable a wide range of applications, including voice assistants Held et al. 2024 , audio understanding Dinkel et al. 2025 , real-time speech interaction Long et al. 2025 , etc . Their ability to understand and generate responses directly from audio makes them a critical component for next-generation human–AI interaction systems. Despite their promise, the deployment of LALMs raises pressing safety concerns due to the underexplored vulnerability of new audio input. In the literature, most focus of safety alignment has been laid in text-based LLMs Kim et al. 2024 ; Zhang et al. 2025a ; Qi et al. 2024 , leveraging both fine-tuning-based defenses such as supervised fine-tuning (SFT) Liu et al. 2023 and reinforcement learning from human feedback (RLHF) Bai et al. 2022 , and more advanced inference-based defenses such as activation steering Panickssery et al. 2023 ; Zhao et al. 2025 . While fine-tuning can be effective with high-quality data or well-trained reward models, its resource-intensive nature makes inference-based defenses more practical for scalable deployment. Similar efforts have recently extended to Large Vision–Language Models (LVLMs) Wang et al. 2024a ; Lu et al. 2024 ; Zhu et al. 2023 ; Liu et al. 2024b , leading to new fine-tuning-based Zhang et al. 2025c ; Zong et al. 2024 and inference-based Wang et al. 2024b ; Ding et al. 2024 defense strategies designed for vision modality. In contrast, the safety alignment of LALMs remains largely underexplored: beyond some initial findings Yang et al. 2024a ; Song et al. 2025 , which show that LALMs are far more likely to comply with harmful speech than text, no principled defense strategies have been developed. A natural solution, therefore, is to transfer the alignment techniques originally designed for LLMs or LVLMs into the audio–language setting. In this work, we focus on inference-based defenses , e.g. , activation steering from LLMs Panickssery et al. 2023 and prompt-based defenses from LVLMs Wang et al. 2024b , to align LALMs with harmless outputs. However, such transfers with vanilla adaptations expose two critical limitations. First, LLM-based steering fails under audio input. In LLMs, steering vectors constructed from harmful–safe text pairs can reliably shift representations toward safe regions and enhance refusal behaviors. In LALMs, by contrast, harmful and safe speech inputs occupy widely divergent latent distributions than in text, making the harm-to-safe direction unreliable (Section 3.3 ). Second, prompt-based defenses from LVLMs induce over-refusal unconspicuously Jiang et al. 2025 . While defensive prompts ( e.g. , instructing the model to respond “I am sorry” to unethical or illegal requests) can block some harmful queries, they also cause benign queries with lexical similarity to be mistakenly rejected (Section 3.4 ). Despite efforts such as AdaShield Wang et al. 2024b , which refines prompts to better distinguish benign inputs, the coarse input-level instructions, containing two opposing actions of answering or refusing, struggle to coordinate effectively. To address these challenges, we propose an inference-based alignment framework, S afe- A blated R efusal Steer ing ( SARSteer ), for LALMs. SARSteer targets both the failure of steering audio modality and the over-refusal issue observed in prompt-based defenses. It consists of two key components: 1) Text-derived refusal steering. Instead of contrasting harmful and safe speech inputs, which suffer from distributional gap, SARSteer extracts refusal vectors directly from textual refusal prompts ( e.g. , “I cannot assist with that” ). These vectors capture safety-aligned semantics in intermediate activations and provide a modality-agnostic direction for enhancing harmful-query rejection. 2) Decomposed safe-space ablation. To mitigate over-refusal on benign queries, SARSteer employs a projection correction step. Specifically, we use principal component analysis (PCA) on safe samples to identify the dominant subspace of benign semantics, and then ablate this component from the refusal vector. This ensures that refusal steering acts only on harmful directions while preserving safe responses. By jointly leveraging these two components, SARSteer avoids costly fine-tuning, operates entirely at inference time, and establishes a principled defense strategy for LALMs that is robust against harmful inputs while maintaining utility on benign ones. Our contributions are summarized as follows: • Empirical Findings. We construct paired harmful–safe datasets in the speech domain and provide a systematic study of the representational differences between text and audio inputs, explaining the failure of direct activation steering transfer. • Methodology. We introduce the first inference-time defense framework for LALMs, based on text-derived refusal steering and decomposed safe-space ablation, filling the gap of broad LALMs applications and the scarcity of the specified safety alignment. • Evaluation. Extensive experiments demonstrate that our method significantly improves harmful-query refusal while maintaining overall utility, achieving a more favorable trade-off between safety and usability. 2 Related Work 2.1 LLM Safety Alignment Substantial research has focused on aligning LLMs with human values and safety standards Bai et al. 2022 ; Kim et al. 2024 ; Zhang et al. 2025a ; Qi et al. 2024 . Prominent approaches include reinforcement learning from human feedback (RLHF), which fine-tunes models using human-preferred responses Ouyang et al. 2022 ; Bai et al. 2022 , and supervised fine-tuning (SFT) on safety-centric datasets Liu et al. 2023 . These methods can all be categorized as fine-tuning-based defenses . Despite their effectiveness, they often require extensive human annotation and computational resources, limiting their application scenarios. More recently, inference-time techniques have gained attention for their efficiency and low resource demands Arditi et al. 2024 ; Zhao et al. 2025 ; Qian et al. 2025 . For instance, activation steering methods intervene in the model’s internal representations to guide outputs toward desired behaviors Panickssery et al. 2023 ; Zhao et al. 2025 ; Ghosh et al. 2025 . Similarly, refusal prompts, prepending input queries with safety-guided instructions, have been shown to enhance robustness against malicious inputs without additional training Zheng et al. 2024 ; Qian et al. 2025