Precision-Varying Prediction (PVP): Robustifying ASR systems against adversarial attacks

Matías, Raghavan, Asja · 2026

Read Precision-Varying Prediction (PVP): Robustifying ASR systems against adversarial attacks by Matías, Raghavan, Asja (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-22590.txt.

Introduction

Pizarro and co-authors propose a simple defense for speech recognition systems under attack. They note that changing a model's numeric precision at test time makes attacks less likely to work. Their method picks the precision at random for each prediction. A gap between outputs at different precisions can also flag a likely attack. They pair this with an existing defense based on uncertainty. Tests across models, languages, and attack types show much stronger robustness. Adaptive attackers must add loud, clear noise to get past the checks.

Catalog record

CREATORMatías, Raghavan, Asja
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE27925
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.22590

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Precision-Varying Prediction (PVP): Robustifying ASR systems against adversarial attacks Matías, Raghavan, Asja Abstract With the increasing deployment of automated and agentic systems, ensuring the adversarial robustness of automatic speech recognition (ASR) models has become critical. We observe that changing the precision of an ASR model during inference reduces the likelihood of adversarial attacks succeeding. We take advantage of this fact to make the models more robust by simple random sampling of the precision during prediction. Moreover, the insight can be turned into an adversarial example detection strategy by comparing outputs resulting from different precisions and leveraging a simple Gaussian classifier. An experimental analysis demonstrates a significant increase in robustness and competitive detection performance for various ASR models and attack types. Pizarro Narasimhan Fischer keywords automatic speech recognition, adversarial attacks, adversarial robustness, adversarial detection † † address: Faculty of Computer Science, Ruhr University Bochum, Germany † † email: [email protected], [email protected], [email protected] 1 Introduction Over the past decade, automatic speech recognition (ASR) systems have advanced rapidly, enabling voice-driven interactions that range from simple command execution in virtual assistants to open-ended conversational queries. State-of-the-art ASR models are predominantly based on deep neural networks (DNNs) [ 13 , 15 ] , and these models are now deeply embedded in everyday technologies, making their reliability under real-world conditions increasingly critical. From reactive systems such as Alexa, which control basic functions like lighting and music, to emerging agentic AI systems that couple speech interfaces with autonomous decision-making and action execution. In this case, recognition errors can propagate beyond misinterpretation, directly influencing downstream behavior and outcomes. Furthermore, these systems are increasingly being deployed in safety-critical domains such as autonomous driving [ 2 ] and healthcare [ 8 , 14 ] . Given that multiple studies [ 6 , 10 , 3 , 27 , 31 ] have shown that they can be maliciously manipulated via carefully designed input perturbations, adversarial robustness, i.e., the ability of ASR systems to withstand erroneous or adversarial inputs, is becoming a non-optional requirement, as it affects the security and trustworthiness of automated systems. When studying adversarial attacks, research generally follows two directions: improving model resistance to adversarial perturbations and designing detectors to distinguish benign from adversarial samples. Within the first direction, two strategies are commonly explored, namely input transformations [ 23 , 12 ] and adversarial training [ 16 ] . The former modifies the signal before it is processed by the ASR system, while the latter incorporates adversarial examples (AEs) during training. However, both strategies have notable limitations. Input transformations may lose effectiveness once they are incorporated into the attacker’s optimization process [ 7 ] and can introduce latency and perceptual artifacts in audio, while adversarial training faces scalability challenges for large and complex data due to high computational costs [ 36 ] . Moreover, both strategies often degrade performance on benign inputs [ 33 ] . On the detection side, strategies vary widely in their requirements and limitations. Some methods, such as Noise Flooding (NF) [ 29 ] , require multiple ASR queries to estimate the perturbation needed to change predictions, making them computationally expensive. Others rely on internal model information, such as logits or uncertainty estimates [ 5 , 24 ] , or on temporal-dependency-based methods (TD) [ 34 ] , which exploit the temporal structure in raw audio and demand minimum input lengths. These approaches are often model-specific, training-dependent, sensitive to benign input variations, or can be bypassed by carefully crafted attacks [ 35 ] . To address these limitations, we introduce Precision-Varying Prediction (PVP): (i) a simple approach for increasing the robustness of ASR systems that requires no re-training and (ii) an easy detection strategy that is training-free and simple to employ without requiring specific model knowledge. Specifically, we leverage numerical precision, which can be chosen for modern ASR models and which induces systematic differences in model behavior. We show that AEs generated at a given precision exhibit reduced transferability across alternative precision settings. Building on this observation, we enhance ASR robustness by randomly switching the precision during inference and propose lightweight classifiers that evaluate inputs under multiple precision configurations and compare the resulting transcriptions to distinguish benign from AEs. These methods require neither adversarial training nor auxiliary detection modules and do not rely on access to model internals. Our approach is training-free, model-agnostic, and efficient, making it suitable for practical Green AI deployment. Experiments across multiple ASR models and attacks demonstrate that precision diversity not only enhances overall adversarial robustness but also provides reliable adversarial detection without degrading benign performance. 2 Adversarial Attacks Adversarial attacks are methods that introduce carefully designed perturbations to input data with the objective of causing a machine learning model to produce incorrect outputs. In the audio domain, such perturbations are often constrained to be small and may be imperceptible or minimally perceptible to human listeners, depending on the attack design and threat model. Carlini & Wagner Attack (C&W): This is an iterative optimization-based algorithm that generates AEs by explicitly solving a constrained optimization problem. Its goal is to craft a perturbed audio signal that is perceptually similar to the original input while the ASR model outputs an arbitrary desired transcription. Formally, given an ASR model f(\cdot) , an audio input signal x , and a target transcription y_{\text{t}} , this attack solves the following optimization problem \min_{\delta}\,\lVert\delta\rVert_{q}+c\cdot\mathcal{L}_{\text{o}}\big(f(x+\delta),y_{\text{t}}\big)\quad\text{s.t}\quad\delta\in[\delta_{\min},\delta_{\max}]\kern 5.0pt, (1) where \delta denotes the adversarial perturbation, \lVert\cdot\rVert_{q} controls the perturbation magnitude, \mathcal{L}_{\text{o}}(\cdot) is a differentiable loss function, and c is a trade-off parameter balancing perturbation strength and attack success. Psychoacoustic Attack: This attack extends the C&W framework by explicitly accounting for human auditory perception, producing perturbations that are largely imperceptible. Specifically, a differentiable psychoacoustic loss \mathcal{L}_{\text{m}}(\cdot) penalizes perturbations that exceed masking thresholds in the time–frequency domain, ensuring that the crafted perturbation remains below human detection limilts while still steering the ASR decoder toward a specified target transcript: \min_{\delta}\ \lVert\delta\rVert_{q}+c_{1}\cdot\mathcal{L}_{\text{o}}\big(f(x+\delta),y_{t}\big)+c_{2}\cdot\mathcal{L}_{\text{m}}(x,\delta)\kern 5.0pt, (2) where c_{1} and c_{2} are trade-off parameters that balance attack success and imperceptibility, respectively, and \mathcal{L}_{\text{m}} encodes masking thresholds derived from human auditory perception. 3 Approach The core idea is that adversarial perturbations exhibit reduced stability when the numerical precision of the model is varied at inference. We leverage this via a lightweight detection mechanism based on transcription consistency across precision modes. 3.1 ASR Under Varying Numerical Precision Modern deep learning frameworks support multiple floating-point formats—FP32 (32-bit single precision with a larger mantissa and dynamic range) and reduced-precision formats such as FP16 (16-bit, smaller mantissa and narrower range) and BF16 (16-bit with an FP32-like exponent but a reduced mantissa). While FP32 provides higher numerical stability, FP16 and BF16 improve computational efficiency and memory usage. In practice, models rarely execute in a single precision. Instead, runtime behavior results from software-level automatic mixed precision [ 26 ] , which applies dynamic casting and gradient scaling, and hardware/backend constraints [ 19 , 20 ] , where operations such as matrix multiplications may use reduced-precision inputs but accumulate in higher precision (often FP32) [ 9 ] . We define the exposed precision controls as the user-configurable storage dtype, autocast compute dtype, and activation of gradient scaling. Distinguishing storage from compute precision and training from inference precision, we manipulate only the exposed compute precision as a

Read the complete text