Dimensionality-Aware Anomaly Detection in Learned Representations of Self-Supervised Speech Models

Sandra, Sarah M, James, Sanjeev · 2026

Read Dimensionality-Aware Anomaly Detection in Learned Representations of Self-Supervised Speech Models by Sandra, Sarah M, James, Sanjeev (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-02715.txt.

Introduction

Sandra Arcos-Holzinger and three co-authors probe how disturbances deform the inner geometry of self-supervised speech models. Such models perform well on later tasks, yet their learned representations are poorly understood under natural and hostile perturbations. Older studies compared whole representations or used global size measures, missing local changes.

Their framework tracks local intrinsic dimensionality, layer by layer, in two leading speech models. Under quiet, noisy conditions this local size rises for all perturbations. At cleaner signal levels the paths split: benign noise settles back toward the clean profile, while hostile inputs keep early layers inflated. The inflation appears alongside worse transcription, and layer-wise measures detect anomalies with strong scores. Transcript-free monitoring of deployed speech models is the stated aim.

Catalog record

CREATORSandra, Sarah M, James, Sanjeev
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE42 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.02715

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Dimensionality-Aware Anomaly Detection in Learned Representations of Self-Supervised Speech Models Sandra, Sarah M, James, Sanjeev Abstract Self-supervised speech models (S3Ms) achieve strong downstream performance, yet their learned representations remain poorly understood under natural and adversarial perturbations. Prior studies rely on representation similarity or global dimensionality, offering limited visibility into local geometric changes. We ask: how do perturbations deform local geometry, and do these shifts track downstream automatic speech recognition (ASR) degradation? To address this, we present GRIDS , a framework using Local Intrinsic Dimensionality (LID) across layer-wise representations in WavLM and wav2vec 2.0. We find that LID increases for all low signal-to-noise ratio (SNR) perturbations and diverges at high SNR: benign noise converges toward the clean profile, while adversarial inputs retain early-layer LID elevation. We show LID elevation co-occurs with increased WER, and that layer-wise LID features enable anomaly detection (AUROC 0.78–1.00), opening the door to transcript-free monitoring in S3Ms. Arcos-Holzinger Erfani Bailey Khudanpur keywords self-supervised representations, local intrinsic dimensionality, anomaly detection, layer-wise analysis, speech recognition † † email: [email protected] † † address: 1 University of Melbourne, School of Computing and Information Systems, Australia 2 Monash University, Department of Data Science and Artificial Intelligence, Australia 3 Johns Hopkins University, Center for Language and Speech Processing, USA 1 Introduction Self-supervised speech representations are an active area of research, with prior work motivating a deeper analysis of information encoding in model layers and understanding the behavior of representations under distributional shifts [ 30 ] . Existing work has sought to enhance the noise robustness of self-supervised speech models (S3Ms), as well as characterize their adversarial vulnerability [ 17 , 20 , 39 , 45 , 44 ] . Analysis of learned representations indicates that information content varies substantially between layers in S3Ms, and that extracting features from the last layers may not be optimal for tasks that require phonetic or word-related information [ 30 ] . Motivated by these observations, we use Local Intrinsic Dimensionality (LID) as a layer-wise geometric diagnostic to quantify how benign and adversarial distortions deform local neighborhood structure. LID characterizes the geometric properties of learned representations by quantifying how rapidly the number of neighboring samples grows as the radius around a reference sample increases – a quantity known as the local rate of probability mass expansion [ 2 , 7 ] . It has been used in a range of contexts, including outlier detection [ 15 ] and neural networks [ 4 , 5 ] , and has proven effective in revealing how adversarial perturbations alter learned representations in the image [ 19 , 36 , 27 , 12 , 42 ] and text [ 38 ] domains, where perturbed samples consistently exhibit higher LID values corresponding to low-density regions in the pixel or deep feature space. In spite of this, and to the best of our knowledge, LID has not been studied in the context of speech representations in S3Ms. Intuitively, LID measures the relative rate at which the neighborhood around a nearby sample grows: expanding rapidly in high-dimensional regions, and slowly in low-dimensional ones. We use manifold to denote the low-dimensional structure that clean-speech representations occupy within the high-dimensional embedding space – the ambient dimension – of a model’s hidden layers, consistent with the manifold hypothesis that high-dimensional data concentrates on or near low-dimensional manifolds [ 32 , 10 ] . The true degrees of freedom of the data, i.e., the number of independent directions along which representations locally vary, are typically much fewer than the ambient dimension. We use geometry to denote the measurable local neighborhood properties of that space – such as distances to nearest neighbors, local density, expansion rate, and effective local dimensionality – that characterize how representations are arranged and how perturbations deform those arrangements. LID is a geometric statistic in this sense: it estimates the effective local dimensionality around a sample, which we leverage to quantify shifts between natural and adversarial conditions relative to the clean manifold structure. A progressive decrease in LID across layers reflects representational compression – a reduction in effective degrees of freedom as the model abstracts from acoustic to linguistic structure. This abstraction hierarchy has been independently established through a layer-wise representation analysis [ 34 ] , and our LID trajectories in S3Ms provide convergent geometric evidence for the same progression. If geometric structure is integral to how S3Ms encode information across layers, then geometric changes due to perturbations should impact downstream performance degradation. This hypothesis motivates understanding changes in LID to task-level metrics such as word error rate (WER) in Automatic Speech Recognition (ASR). We extend this geometric approach to ASR-based S3Ms to investigate how layer-wise representations change under adversarial and benign acoustic perturbations. This extension presents a non-trivial challenge: unlike prior LID applications [ 26 ] where each input yields a single feature vector, S3Ms generate variable-length frame-level embeddings per utterance. This requires careful aggregation of the embeddings to ensure a stable and reliable LID estimation. Furthermore, S3M features are learned via self-supervision and remain task-agnostic [ 30 ] . Whether LID is equally effective in S3Ms as shown in supervised settings [ 26 ] remains an open empirical question. Consequently, we utilize LID as a diagnostic of S3M transformer layers, empirically linking intrinsic geometry in the learned representations of S3Ms to downstream robustness. We refer to our framework as Geometric Robustness via Intrinsic Dimensionality in Speech (GRIDS) . We apply it to WavLM [ 8 ] and wav2vec 2.0 [ 6 ] , and make the following contributions: [label=()] 1. LID–S3M geometric analysis: measurement of local geometric changes in learned representations across transformer layers for clean, benign, and adversarial conditions; 2. LID–ASR monitoring: empirical evidence showing that increases in LID positively associate with WER, linking geometric changes to downstream ASR degradation; 3. LID–AD for anomaly detection: 12 LID-derived features from (i) for adversarial vs. benign classification. 2 Related Work 2.1 Representation Analysis and Dimensionality in Self-Supervised Models Recent work on transformer representation geometry has shown that learned features organize on low-dimensional curved manifolds within a model’s hidden layers, and that attention heads manipulate this manifold structure as a computational mechanism [ 14 ] . This provides empirical support for the manifold hypothesis and reinforces the value of geometric analysis – such as intrinsic dimensionality estimation – for understanding how representations encode structure across layers. Layer-wise representation analysis has proven to be instrumental in the interpretation of self-supervised speech models, providing information on how different layers encode phonetic, syntactic, and semantic information [ 35 ] . Prior work has primarily employed similarity-based metrics to study these relationships, each providing a complementary lens on how representations evolve across a network. Canonical Correlation Analysis (CCA) and Centred Kernel Alignment (CKA), for instance, measure how similar representations are between layers. CCA and its variants identify linear relationships between representations by finding optimal alignment transformations [ 37 , 31 ] (with its non-linear extensions [ 3 , 41 , 29 , 23 , 22 , 40 ] ), whereas CKA extends this to high-dimensional spaces by directly comparing similarity structures rather than feature values [ 21 ] . For speech models specifically, [ 34 ] applied these metrics to S3Ms such as wav2vec 2.0 [ 6 ] and HuBERT

Read the complete text