Spectral Identifiability for Interpretable Probe Geometry

William Hao-Cheng Huang · 2025

Read Spectral Identifiability for Interpretable Probe Geometry by William Hao-Cheng Huang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-16288.txt.

Introduction

William Hao-Cheng Huang asks when a linear probe, a simple readout trained on a neural network's internal activations, can be trusted. Probes often look accurate in one setting and then fail without warning in another, which makes them risky tools for judging what a network represents.

Huang proposes the Spectral Identifiability Principle, a checkable condition inspired by Fisher information. When the gap between task-relevant directions and the rest is larger than the estimation error, the probed subspace is stable and accuracy holds up. When the gap closes, performance drops sharply, like a phase transition. Synthetic tests, where the Fisher quantities are known exactly, confirm the picture and show how a spectral check can flag unreliable probes in advance.

Catalog record

CREATORWilliam Hao-Cheng Huang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE45 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-16288

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Spectral Identifiability for Interpretable Probe Geometry William Hao-Cheng Huang Abstract Linear probes are widely used to interpret and evaluate neural representations, yet their reliability remains unclear, as probes may appear accurate in some regimes but collapse unpredictably in others. We uncover a spectral mechanism behind this phenomenon and formalize it as the Spectral Identifiability Principle (SIP) —a verifiable Fisher-inspired condition for probe stability. When the eigengap separating task-relevant directions is larger than the Fisher estimation error, the estimated subspace concentrates and accuracy remains consistent, whereas closing this gap induces instability in a phase-transition manner. Our analysis connects eigengap geometry, sample size, and misclassification risk through finite-sample reasoning, providing an interpretable diagnostic rather than a loose generalization bound. Controlled synthetic studies, where Fisher quantities are computed exactly, confirm these predictions and show how spectral inspection can anticipate unreliable probes before they distort downstream evaluation. 1 Introduction Understanding what neural networks learn requires reliable probes of internal representations. Probing methods—typically lightweight classifiers trained on frozen features—are widely used but often unstable: their reported accuracy can fluctuate sharply across runs, datasets, or random seeds [ 1 , 2 ] . This variability raises a central question: when can probe results be trusted? From heuristics to verifiable analysis. Most existing probing techniques offer useful but post-hoc evaluations that measure interpretability only after training. We instead seek a verifiable, pre-deployment condition that anticipates probe reliability before instability arises. Such a criterion enables a principled approach to analyzing representational stability rather than relying solely on empirical variance or heuristics. Spectral view of probe reliability. Our approach is grounded in spectral perturbation theory. By examining how the empirical Fisher operator deviates from its population counterpart, we derive a finite-sample condition that predicts when the discriminative subspace remains identifiable. This spectral view reframes interpretability evaluation as a falsifiable problem—one that can be analyzed and tested rather than inferred from post-hoc accuracy trends. Contributions. Our work makes three main contributions: (i) Methodology. We propose the Spectral Identifiability Principle (SIP) , a verifiable criterion for assessing probe reliability through eigengap geometry. (ii) Theory. We establish a finite-sample and spectral connection between Fisher operator estimation, subspace deviation, and misclassification risk, yielding an interpretable stability condition. (iii) Empirics. We validate the theory through controlled synthetic studies with tunable eigengaps and heavy tails, confirming the predicted phase transitions. Together, these results bridge spectral theory and practical interpretability, providing a compact framework for reliable probe evaluation. 2 Related Work Background. Probing methods are a central tool for analyzing neural representations [ 1 , 2 , 3 ] . They train lightweight classifiers on frozen features to test what information a model encodes, but their reliability remains debated—probe accuracies often vary sharply across seeds or datasets. This raises a central question: under what conditions can probe evaluations be trusted? Information-theoretic and causal probes. Recent work has sought to formalize probe evaluation through information-theoretic or causal diagnostics, such as minimum description length (MDL) [ 4 , 5 , 6 ] and amnesic or counterfactual interventions [ 7 , 8 ] . These frameworks measure sufficiency or causal relevance but remain post-hoc and heuristic, lacking falsifiable criteria for stability. Surveys highlight this limitation and call for pre-deployment reliability diagnostics [ 3 ] . Spectral and Fisher analyses. Spectral analysis connects stability to eigengaps and perturbation bounds [ 9 , 10 ] . In machine learning, Fisher and Hessian spectra have been used to study curvature, conditioning, and optimization stability [ 11 , 12 , 13 , 14 , 15 ] . However, these studies focus on training dynamics rather than how estimation errors translate into classification risk —the key determinant of probe reliability. Position of this work. Our framework bridges these perspectives through a finite-sample, verifiable spectral law—the Spectral Identifiability Principle (SIP) —which links Fisher estimation error to subspace stability and misclassification risk. SIP complements information-theoretic and causal approaches by offering a simple, testable diagnostic grounded in classical perturbation theory, providing a unified and verifiable condition for trustworthy probing. 3 Preliminaries In this section, we formalize the key components needed for understanding probe stability in the context of representation learning. We analyze how spectral geometry and estimation error jointly determine probe generalization, laying the foundation for the verifiable condition in later sections. These preliminaries will set up the precise link between spectral estimation and classification performance, which is the foundation for proving the verifiable condition for probe reliability in later sections. Data and representation. We observe samples (X,Y) with h(X)\in\mathbb{R}^{d} , where Y is a binary label. A linear probe takes the form f(x)=\sign(w^{\top}h(x)),\quad w\in\mathbb{R}^{d}. Throughout, we fix h(\cdot) and study probe behavior conditioned on this representation. Understanding how the features in h(X) relate to the label is central to explaining probe performance and stability. Fisher operator and subspace. Following recent representation-learning literature, we define \Gamma=\mathbb{E}[h(X)h(X)^{\top}] and refer to it as the Fisher operator . Although not the classical Fisher information, this uncentered second moment captures the discriminative geometry exploited by probes, and we therefore refer to it as the Fisher operator . In representation learning, this operator directly encodes the alignment between h(X) and the label. Using covariance instead would remove label-related structure, whereas the uncentered moment preserves task alignment. Intuitively, the top- k eigenspace of \Gamma captures the directions where the representation varies most strongly with respect to the task, serving as the candidate discriminative subspace. Let U\in\mathbb{R}^{d\times k} denote the top- k eigenspace of \Gamma , separated by the eigengap \operatorname{gap}(\Gamma)=\lambda_{k}(\Gamma)-\lambda_{k+1}(\Gamma)>0. The existence of this gap is essential for probe stability: it ensures that the task-relevant subspace is well-defined and recoverable from the data. Empirical estimate. In practice, we do not have access to the true Fisher operator \Gamma . Instead, we estimate it from finite samples. The empirical Fisher operator is \widehat{\Gamma}=\tfrac{1}{n}\sum_{i=1}^{n}h(X_{i})h(X_{i})^{\top}, with estimation error \Delta=\|\widehat{\Gamma}-\Gamma\|_{\mathrm{op}}. Finite samples inevitably introduce estimation error, which can distort the recovered subspace. We measure subspace discrepancy using the principal angle distance \sin\Theta(\widehat{U},U) , where \widehat{U} is the top- k eigenspace of \widehat{\Gamma} . Understanding how Fisher estimation error affects probe performance is critical for analyzing stability. Classifier and margin. To link subspace errors to classification error, we impose a margin assumption. Within U , the population risk minimizer among linear classifiers is f_{\star}(X)=\sign(a^{\top}g(X)),\qquad g(X)=U^{\top}h(X), where a\in\mathbb{R}^{k} minimizes misclassification risk. Note that this differs from the global Bayes classifier, which may be nonlinear; here we restrict attention to linear probes within U . We assume a Tsybakov margin condition: there exist \kappa,C>0 such that \Pr(|a^{\top}g(X)|\leq t)\leq Ct^{\kappa},\quad\forall t>0. This smoothness condition ensures that spectral error translates into label-flipping probability. For instance, when \kappa=1 , the probability mass near the decision boundary grows linearly with the margin width, so classification error scales proportionally with subspace deviation. Without such a margin condition, even small spectral deviations might either have no effect or cause unbounded misclassification risk. Notation. Table 6 summarizes the central quantities (a complete version appears in App. A ). Quantity Meaning Role in theorem h(X)\in\mathbb{R}^{d} representation input features \Gamma=\mathbb{E}[h(X)h(X)^{\top}] Fisher operator (uncentered moment) captures discriminative geometry U\in\mathbb{R}^{d\times k} top- k eigenspace of \Gamma candidate task subspace \operatorname{gap}(\Gamma) eigengap \lambda_{k}-\lambda_{k+1} determines whether the subspace is recoverable \widehat{\Gamma} empirical Fisher operator data-based estimate of \Gamma \Delta=\|\widehat{\Gamma}-\Gamma\|_{\mathrm{op}} operator error finite-sample deviation

Read the complete text