The Linear Centroids Hypothesis: How Deep Network Features Represent Data

Thomas Walker, Ahmed Imtiaz Humayun, Randall Balestriero, Richard Baraniuk · 2026

Read The Linear Centroids Hypothesis: How Deep Network Features Represent Data by Thomas Walker, Ahmed Imtiaz Humayun, Randall Balestriero, Richard Baraniuk (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-11962.txt.

Introduction

Walker, Humayun, Balestriero, and Baraniuk propose the Linear Centroids Hypothesis. It identifies network features with linear directions among centroid spaces, not middle-layer spaces. Each vector there denotes a centroid, or summary, of a local affine expert describing the learned input-output map. Swapping activations for centroids gives a drop-in alternative for standard analysis tools. The swap yields sparser, more useful dictionaries on DINO vision models, cuts spurious directions on a controlled task, finds clear circuits in GPT2-Large, and gives faithful saliency maps.

Catalog record

CREATORThomas Walker, Ahmed Imtiaz Humayun, Randall Balestriero, Richard Baraniuk
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE58998
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.11962

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

The Linear Centroids Hypothesis: How Deep Network Features Represent Data Thomas Walker, Ahmed Imtiaz Humayun, Randall Balestriero, Richard Baraniuk Abstract Identifying and understanding the features that a deep network (DN) extracts from its inputs to produce its outputs is a focal point of interpretability research. The Linear Representation Hypothesis (LRH) identifies features in terms of the linear directions formed by the inputs in a DN’s latent space. However, the LRH is limited as it abstracts away from individual components (e.g., neurons and layers), is susceptible to identifying spurious features, and cannot be applied across sub-components (e.g., multiple layers). In this paper, we introduce the Linear Centroids Hypothesis (LCH) as a new framework for identifying the features of a DN. The LCH posits that features correspond to linear directions of centroids, which are vector summarizations of the functional behavior of a DN in a local region of its input space. Interpretability studies under the LCH can leverage existing LRH tools, such as sparse autoencoders, by applying them to the DN’s centroids rather than to its latent activations. We demonstrate that doing so yields sparser feature dictionaries for DINO vision transformers, which also perform better on downstream tasks. The LCH also inspires novel approaches to interpretability; for example, LCH can readily identify circuits in GPT2-Large. Code to study the LCH can be found here. Input Sample Centroid Input Samples Centroids Figure 1 : The Linear Centroids Hypothesis (LCH) posits that the centroids of a deep network (DN) represent features as linear directions. The centroids of a DN describe its functional mapping in a local region, and thus can be considered descriptors of the local “experts” of a DN. In the top panels, we show that the centroids of a Swin-B ( 30 ) transformer operating on inputs from ImageNet ( 27 ) can be used as saliency maps. In the bottom panels, we train a DN to classify the interior from the exterior of a two-dimensional star-shaped polygon. Each edge of the star is a semantically consistent subspace of the input space that should be a feature of the trained DN. Indeed, the centroids for the local experts operating in these subspaces separate into linear directions, which provides evidence for the LCH. 1 Introduction Understanding and identifying which characteristics of the input a deep network (DN) extracts and utilizes to form its output is important to facilitate their reliable deployment. The crux of interpretability research is to identify these features of the DN and their influence on the computational graph, often referred to as the circuits of the DN ( 2 ; 46 ) . The current state-of-the-art for identifying the features of a DN is the Linear Representation Hypothesis (LRH), which posits that features are represented as linear directions in a DN’s latent activation space ( 12 ; 41 ) . The LRH has enabled the development of powerful interpretability tools; however, it suffers from three critical limitations. First, it abstracts away from individual sub-components of the DN (e.g., neurons, layers or attention mechanisms), which makes it difficult to link features to the computational graph ( 46 ) . Second, it is unclear whether all linear directions of latent activations correspond to features, making it susceptible to identifying spurious features ( 50 ) . Third, its disjoint application to individual latent spaces has encumbered the field with the task of contextualizing features across sub-components ( 3 ) . In this paper, we introduce a new framework for identifying the features of DNs. The Linear Centroids Hypothesis (LCH) identifies the features of a DN based on how it organizes the geometry of its input space into local “experts.” This is motivated by the fact that continuous piecewise affine (CPA) DNs (e.g., ReLU networks) exactly and other smooth networks (e.g., transformers) approximately partition their input space into a Voronoi-like tiling of polytopal regions, on which distinct functional mappings (i.e., experts) operate ( 7 ) . The LCH posits that features correspond to the linear directions of the centroids of these regions. Crucially, a DN’s centroids can be efficiently calculated for any differentiable DN sub-component, since they are equal to the row-sum of its input-output Jacobian. This transitions feature identification from understanding where in the latent space an input lies to understanding the action of the local expert operating on the input, which provides a grounded and mechanistic perspective to interpretability. Our contributions are the following. (C1) A framework for interpretability that is not susceptible to identifying features irrelevant to the computations in the DN, unlike the LRH (see Figure 3 ). (C2) A demonstration that simply exchanging LRH’s latent activations for LCH’s centroids improves current interpretability techniques and eliminates the necessity of intricate procedures to extract hierarchical DN features (see Figure 4 ). (C3) The introduction of novel interpretability techniques to facilitate the efficient identification of the circuits of a DN (see Figure 6 ). (C4) A new method for constructing gradient-based saliency maps–the local centroid–that is faithful to the behaviors of the DN (see Figures 1 and 8 ). In short, the LCH turns feature discovery into the study of the alignment of the local experts of a network, so that interpretability becomes mechanistic by construction rather than post hoc. Our new centroid-based lens helps unify feature dictionaries, probing, circuit discovery, and saliency maps under a single geometric object: the network’s induced input-space partition. 1 1 1 Code to study the LCH can be found here: https://github.com/ThomasWalker1/LinearCentroidsHypothesis . 2 Deep Networks as Utilizing Local Experts In this section, we demonstrate how continuous piecewise affine (CPA) DNs can be intuitively thought of as employing local experts to form their outputs. In particular, we demonstrate that centroids, which are computable for arbitrary DNs, describe the arrangement and behavior of these experts. Deep Networks and their Approximations. A DN f:\mathbb{R}^{d^{(0)}}\to\mathbb{R}^{d^{(L)}} , with the convention that d^{(0)}=d , is a composition of L functions f^{(\ell)}:\mathbb{R}^{d^{(\ell-1)}}\to\mathbb{R}^{d^{(\ell)}} . These functions, referred to as layers, typically consist of an affine transformation followed by a nonlinearity. We will denote the mapping across multiple layers as f^{(\ell_{1}\leftarrow\ell_{2})}:\mathbb{R}^{d^{(\ell_{1}-1)}}\to\mathbb{R}^{d^{(\ell_{2})}} for 1\leq\ell_{1}<\ell_{2}\leq L . Using spline approximation theory ( 32 ; 45 ) , DNs can be approximated to arbitrary precision or even characterized exactly using CPA splines ( 5 ) . More specifically, one can write f({\bm{x}})\approx{\bm{A}}_{\omega_{\bm{\nu}({\bm{x}})}}{\bm{x}}+{\bm{b}}_{\omega_{\bm{\nu}({\bm{x}})}} , where {\bm{A}}_{\omega_{\bm{\nu}({\bm{x}})}}\in\mathbb{R}^{d^{(L)}\times d} and {\bm{b}}_{\omega_{\bm{\nu}({\bm{x}})}}\in\mathbb{R}^{d^{(L)}} are the affine parameters specific to the linear region \omega_{\bm{\nu}({\bm{x}})}\subseteq\mathbb{R}^{d} encompassing {\bm{x}} . Here \bm{\nu}({\bm{x}}) identifies the equivalence class of {\bm{x}} under the collection of all equivalence classes \mathcal{V} constructed by \sim where {\bm{x}}_{1}\sim{\bm{x}}_{2} if and only if {\bm{x}}_{1} and {\bm{x}}_{2} are in the same linear region. Similar approximations can be constructed for each layer f^{(\ell)} to obtain f({\bm{x}})\approx{\bm{A}}_{\omega_{\bm{\nu}({\bm{x}})}^{(L)}}^{(L)}\left(\dots\left({\bm{A}}_{\omega_{\bm{\nu}({\bm{x}})}^{(1)}}^{(1)}{\bm{x}}+{\bm{b}}_{\omega_{\bm{\nu}({\bm{x}})}^{(1)}}^{(1)}\right)\dots\right)+{\bm{b}}_{\omega^{(L)}_{\bm{\nu}({\bm{x}})}}^{(L)}, where \omega_{\bm{\nu}({\bm{x}})}^{(\ell)}\subseteq\mathbb{R}^{d^{(\ell-1)}} denotes the linear region for the mapping f^{(\ell)} encompassing f^{(1\leftarrow\ell-1)}({\bm{x}})\in\mathbb{R}^{d^{(\ell-1)}} . For CPA approximations of sub-components of the DN, say f^{(\ell_{1}\leftarrow\ell_{2})}:\mathbb{R}^{d^{(\ell_{1}-1)}}\to\mathbb{R}^{d^{(\ell_{2})}} for 1\leq\ell_{1}<\ell_{2}\leq L , we adopt a similar notation. When the DN employs CPA nonlinearities (e.g., ReLU), these approximations are exact ( 5 ) . This piecewise discretisation of the output of a DN sub-component, as the application of distinct affine transformations to local regions of the input space, enables us to interpret a DN sub-component as forming its output by employing local experts. The Geometry of a

Read the complete text