Statistical and structural identifiability in representation learning
Walter Nelson, Marco Fumero, Theofanis Karaletsos & Francesco Locatello · 2026
Read Statistical and structural identifiability in representation learning by Walter Nelson, Marco Fumero, Theofanis Karaletsos & Francesco Locatello (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-11970.txt.
Introduction
Nelson and co-authors split how stable learned representations are into two ideas. The statistical kind asks whether runs agree. The structural kind asks whether it matches hidden ground truth. A perfect match is unrealistic, so they allow a small error slack and speak of near-identifiability. For models with nonlinear decoders, they prove it reaches intermediate layers, not just the last one. ICA then clears much of the leftover linear ambiguity. Given added assumptions about the data, the statistical result extends to the structural one.
Catalog record
| CREATOR | Walter Nelson, Marco Fumero, Theofanis Karaletsos & Francesco Locatello |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 123914 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.11970
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Statistical and structural identifiability in representation learning Walter Nelson, Marco Fumero, Theofanis Karaletsos & Francesco Locatello Abstract Representation learning models exhibit a surprising stability in their internal representations. Whereas most prior work treats this stability as a single property, we formalize it as two distinct concepts: statistical identifiability (consistency of representations across runs) and structural identifiability (alignment of representations with some unobserved ground truth). Recognizing that perfect pointwise identifiability is generally unrealistic for modern representation learning models, we propose new model-agnostic definitions of statistical and structural near-identifiability of representations up to some error tolerance ϵ \epsilon . Leveraging these definitions, we prove a statistical ϵ \epsilon - near-identifiability result for the representations of models with nonlinear decoders, generalizing existing identifiability theory beyond last-layer representations in e.g. generative pre-trained transformers (GPTs) to near-identifiability of the intermediate representations of a broad class of models including (masked) autoencoders (MAEs) and supervised learners. Although these weaker assumptions confer weaker identifiability, we show that independent components analysis (ICA) can resolve much of the remaining linear ambiguity for this class of models, and validate and measure our near-identifiability claims empirically. With additional assumptions on the data-generating process, statistical identifiability extends to structural identifiability, yielding a simple and practical recipe for disentanglement: ICA post-processing of latent representations. On synthetic benchmarks, this approach achieves state-of-the-art disentanglement using a vanilla autoencoder. With a foundation model-scale MAE for cell microscopy, it disentangles biological variation from technical batch effects, substantially improving downstream generalization. 1 Introduction Despite the massive variety of data modalities, pretext tasks, training procedures, and datasets, disparate self-supervised learning models as a whole seem to be converging on a shared set of representations of the natural world ( 30 ) which are useful for a surprising variety of downstream tasks ( 40 ; 25 ; 5 ; 8 ) . A classical lens for studying this phenomenon is the notion of identifiability ( 56 ) . In likelihood-based statistical inference, identifiability is the condition that data are sufficient to completely characterize the parameters of the model ( 12 ) . The situation is considerably trickier for neural network models: the parameter space is large and invariant to e.g. permutations of the neurons, and the training procedures might lack a likelihood-based interpretation. Instead, recent work in identifiability focuses on finding conditions such that infinite data is sufficient to characterize the trained model’s representations of that data ( 56 ) up to some equivalence class such as a linear transformation ( 59 ) . We begin by sharpening existing definitions of representation identifiability, recognizing that existing results fall into two categories. The first is what we refer to as statistical identifiability , or the condition that optimizing a given representation learning model will yield the same representations up to some simple transformation. The second is what we refer to as structural identifiability , or the condition that optimizing a given representation learning model will yield a particular representation every time, corresponding to some latent component of the data-generating process. We provide definitions of statistical and structural identifiability that relax the requirement that the representations are exactly identifiable, making them the first general-purpose formulations which are applicable to the case where the representations are “nearly” identifiable up to some error tolerance \epsilon and extending prior model-specific cases ( 53 ; 9 ) . Leveraging these definitions, we prove several new identifiability results. Our first result shows that for models which have statistically identifiable outputs, such as generative pre-trained transformers, supervised classifiers, and encoder-decoder models ( 59 ) , the intermediate-layer representations are also statistically \epsilon -nearly identifiable up to a rigid transformation. Unlike several recent results, these representations can be mapped non-linearly to the loss, and \epsilon is governed by a mild function class condition on the mapping from the intermediate layer to the identifiable outputs. Our second result shows that linear independent components analysis (ICA) can resolve this rigid indeterminacy, yielding near-identifiability up to signed permutations. Notably, our sharper definitions of statistical and structural identifiability reveal that these results are available without strong assumptions on the data-generating process, instead requiring only this mild function class assumption on the model, extending prior theoretical results on isometric and approximately isometric learning ( 23 ; 9 ) . Our final result shows that if one is willing to make a similar assumption on the data-generating process, encoder-decoder models which are statistically identifiable and achieve perfect reconstruction are also structurally identifiable. Notably, perfect reconstruction is another assumption which can be relaxed if statistical identifiability is all that is required. In addition to our theoretical contributions, we perform a series of experiments to validate our claims. In synthetic experiments on autoencoders, we show that hyperparameter selection and regularization impacts the statistical identifiability nearness \epsilon in ways that are predicted by our theory. Subsequent experiments show that near-identifiability also holds in off-the-shelf pre-trained models, and that the linear indeterminacies predicted by our theory can in practice be resolved by ICA. Next, we investigate whether our structural identifiability result can be applied to the special case of disentanglement ( 47 ) , finding that the simple combination of vanilla autoencoders and linear ICA applied to the latent space yields disentanglement on several benchmark datasets, competitive with some of the best existing models. Finally, we show that linear ICA applied to the latent space of a masked autoencoder for cell imaging successfully disentangles batch effects from biological variation, a key problem in the application of machine learning to biology. 2 Related work Statistical identifiability of representations Prior representation identifiability results make strong assumptions on the data-generating process ( 71 ; 57 ; 35 ; 16 ; 41 ) or assume a linear relationship between the representations and the loss ( 59 ; 49 ; 53 ) , and generally do not distinguish between statistical and structural identifiability as we do here. Our work directly addresses this gap by proposing a concrete definition of statistical \epsilon -near-identifiability which is provably met by the general-purpose models in widespread use today, with only mild assumptions on the model class and few assumptions on the data-generating process. We directly measure the consequences of our statistical identifiability results by assessing the \ell_{2} convergence of representations in real-world models, extending prior work on representation similarity ( 59 ; 30 ; 37 ; 53 ; 49 ) . 53 relaxes the identifiability theory of 59 for generative pre-trained transformers, showing that the Kullback-Leibler divergence on the next-token distribution fails to serve as a witness for differences in the penultimate-layer representations, and proves a sufficient condition for divergences that do, which could be viewed as a particular form of near-identifiability for a particular model class. 58 formalizes \epsilon -non-identifiability with respect to the KL divergence, showing that this failure generalizes beyond representational identifiability to other properties of interest for large language models. For a history of the term identifiability, see Appendix A.1 . Structural identifiability of representations Ours is also the first work to make clear the distinction between such model-specific identifiability results and structural identifiability results such as disentanglement. We formalize an assumption on the data-generating process that allows us to extend our key statistical identifiability result to structural identifiability of encoder-decoder models, and define and