Bridging Functional and Representational Similarity via Usable Information

Antonio Almudévar, Alfonso Ortega · 2026

Read Bridging Functional and Representational Similarity via Usable Information by Antonio Almudévar, Alfonso Ortega (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-21568.txt.

Introduction

Antonio Almudévar and Alfonso Ortega unify two ways of judging whether two model representations are alike. One view is functional: can parts be stitched together and still work? The other view is about the information the parts carry.

The authors tie stitching success to conditional mutual information, a classic measure of shared usable content. Their frame covers both views across three dimensions of comparison. A shared yardstick helps researchers decide when two models really compute alike. That judgment matters for merging models, reusing parts, and trusting that a smaller model mirrors a bigger one.

Catalog record

CREATORAntonio Almudévar, Alfonso Ortega
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE63 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-21568

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Bridging Functional and Representational Similarity via Usable Information Antonio Almudévar, Alfonso Ortega Abstract We present a unified framework for quantifying the similarity between representations through the lens of usable information , offering a rigorous theoretical and empirical synthesis across three key dimensions. First, addressing functional similarity, we establish a formal link between stitching performance and conditional mutual information. We further reveal that stitching is inherently asymmetric, demonstrating that robust functional comparison necessitates a bidirectional analysis rather than a unidirectional mapping. Second, concerning representational similarity, we prove that reconstruction-based metrics and standard tools (e.g., CKA, RSA) act as estimators of usable information under specific constraints. Crucially, we show that similarity is relative to the capacity of the predictive family: representations that appear distinct to a rigid observer may be identical to a more expressive one. Third, we demonstrate that representational similarity is sufficient but not necessary for functional similarity. We unify these concepts through a task-granularity hierarchy: similarity on a complex task guarantees similarity on any coarser derivative, establishing representational similarity as the limit of maximum granularity: input reconstruction. 1 Introduction The success of deep learning stems largely from the internal representations learned by neural networks. These representations encode structured and often compressed information about the input, driving strong performance across diverse tasks ( 6 ; 41 ) . Comparing representations has emerged as a central question—not only in deep learning, but also in neuroscience and cognitive science ( 24 ; 21 ; 40 ) —as it reveals how different systems process and organize information. These comparisons address fundamental questions such as: (i) whether different models capture the same structures ( 29 ) , (ii) whether knowledge can be transferred across models ( 44 ) , and (iii) whether performance differences stem from the representations or from the task heads ( 2 ) . However, before developing or applying metrics to compare representations, it is essential to clarify what exactly should be compared—in other words, what it means for two representations to be similar or dissimilar. The literature typically distinguishes between two complementary notions of similarity: (i) functional similarity , which assesses whether they support the same downstream behavior—i.e., whether their encoded information is equivalently usable for predicting a given output ( 28 ; 10 ) ; and (ii) representational similarity , which assesses whether two representations encode input information in the same way—i.e., whether they preserve equivalent relational or informational structure ( 23 ) . Despite their intuitive connection, these notions remain loosely defined and have been studied largely in isolation. The lack of a unified framework has disconnected functional methods (e.g., stitching) from geometric metrics (e.g., CKA, RSA). We resolve this by grounding both in the theory of usable information ( 43 ) , defining similarity as relative to the computational constraints of a predictive family . Our main contributions are: 1. A Unified Framework: We theoretically link model stitching to usable conditional mutual information, showing that functional similarity is defined relative to the capacity of the predictive family. 2. Reinterpreting Standard Metrics: We demonstrate that standard metrics like CKA, RSA, and Procrustes are not arbitrary geometric measures, but specific estimators of usable information. 3. The Hierarchy of Similarity: We show representational similarity is sufficient, but not necessary, for functional similarity. We bridge this by establishing similarity is monotonic with respect to task granularity, identifying representational similarity as the limit of maximum granularity: input reconstruction. 2 Preliminaries 2.1 Usable Information Theory Classical information theory quantifies the total statistical dependence between random variables through the mutual information I(Y;Z) , which measures how much knowledge of Z\in\mathcal{Z} reduces the uncertainty about Y\in\mathcal{Y} ( 37 ; 8 ) . However, in learning systems, not all of this information is necessarily usable by a downstream model. A representation may encode fine-grained details of Y that, while statistically present, cannot be effectively exploited by a predictor of limited capacity ( 1 ) . In other words, the amount of information that is usable by a learner depends not only on the representation itself but also on the inductive biases and expressiveness of the predictive model ( 38 ) . Usable information theory ( 43 ; 11 ) formalizes this by restricting information flow to a predictive family \mathcal{V} . We define the \mathcal{V} -conditional entropy H_{\mathcal{V}}(Y\mid Z) and the marginal \mathcal{V} -entropy H_{\mathcal{V}}(Y\mid\varnothing) as the minimum cross-entropy loss achievable by a predictor in \mathcal{V} : \displaystyle H_{\mathcal{V}}(Y\mid Z) \displaystyle=\inf_{f\in\mathcal{V}}\mathbb{E}_{p(y,z)}\big[-\log f[z](y)\big] (1) \displaystyle H_{\mathcal{V}}(Y\mid\varnothing) \displaystyle=\inf_{f\in\mathcal{V}}\mathbb{E}_{p(y)}\big[-\log f[\varnothing](y)\big] (2) where f is a function \mathcal{Z}\cup\{\varnothing\}\to\mathcal{P}(\mathcal{Y}) , so f[z]\in\mathcal{P}(\mathcal{Y}) is a probability measure on \mathcal{Y} chosen based on z ; and f[z](y)\in\mathbb{R} is the value of the density evaluated at y\in\mathcal{Y} . We define the usable information from Z to Y as the reduction in uncertainty provided by Z relative to \mathcal{V} : I_{\mathcal{V}}(Z\to Y)=H_{\mathcal{V}}(Y\mid\varnothing)-H_{\mathcal{V}}(Y\mid Z). (3) Unlike Shannon mutual information, usable information is directional: I_{\mathcal{V}}(Z\to Y)\neq I_{\mathcal{V}}(Y\to Z) . For instance, if \mathcal{V} contains only linear predictors, I_{\mathcal{V}}(Z\to Y) quantifies how much of the information in Z is linearly usable for predicting Y . A representation that encodes Y in a highly nonlinear form may exhibit high total information I(Y;Z) but low usable information I_{\mathcal{V}}(Z\to Y) . Finally, the usable conditional information quantifies the information in Z about Y usable beyond a variable W . It is defined as the reduction in \mathcal{V} -entropy achieved by adding Z to the context W : \displaystyle I_{\mathcal{V}}(Z\to Y\mid W) \displaystyle=H_{\mathcal{V}}(Y\mid W)-H_{\mathcal{V}}(Y\mid Z,W). (4) This quantity measures the additional functional value of Z given W . When \mathcal{V} is unrestricted, these metrics recover standard Shannon definitions. Thus, usable information bridges statistical and functional perspectives, precisely quantifying which aspects of a representation can be exploited by downstream models. 2.2 Functional Similarity Functional similarity measures whether two representations support the same downstream behaviors, regardless of how information is encoded internally. Representations are functionally similar if predictors trained on one operate effectively on the other, potentially via a transformation layer aligning feature spaces. A common framework is model stitching ( 28 ; 5 ; 16 ; 34 ) , combining one encoder with another’s decoder to evaluate how well their intermediate representations interface. High stitching performance implies comparable task-relevant information. Recent work on relative representations ( 33 ) posits that functional similarity stems from preserving topological relations (distances and angles) rather than absolute coordinates. Related approaches include linear probes ( 2 ; 17 ) , transfer evaluations ( 45 ; 22 ) , and functional alignment methods that measure performance consistency across models ( 12 ) . 2.3 Representational Similarity Representational similarity metrics quantify the alignment between two sets of representations, Z_{A}\in\mathbb{R}^{n\times d_{A}} and Z_{B}\in\mathbb{R}^{n\times d_{B}} . These methods generally fall into two categories: those that explicitly align the feature spaces and those that compare intrinsic structures ( 10 ) . Alignment-based methods minimize element-wise distance. Linear Regression ( 29 ) finds the optimal linear mapping, providing invariance to invertible linear transformations. Procrustes Analysis ( 39 ) , restricts the mapping to an orthogonal rotation. Thus, Procrustes is invariant strictly to orthogonal transformations . In contrast, structural methods compare geometric properties without direct feature alignment, typically sharing invariance to orthogonal transformations and isotropic scaling . Key approaches include Canonical Correlation Analysis (CCA) and

Read the complete text