Modes of Sequence Models and Learning Coefficients
Zhongtian Chen, Daniel Murfet · 2025
Read Modes of Sequence Models and Learning Coefficients by Zhongtian Chen, Daniel Murfet (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-18048.txt.
Introduction
Zhongtian Chen and Daniel Murfet link the patterns in text data to the shape of the loss hills that a transformer model climbs as it learns. Their aim is to explain a measure of learning shape that workers use in practice, called the local learning coefficient.
First they place text distributions in a Hilbert space and split them into main modes. Cutting the small modes leaves a leaner copy that keeps the main structure. They then prove the learning measure is blind to modes below a set size. So in practice the measure reads the lean copy, not the full truth, which explains why it can be trusted.
Catalog record
| CREATOR | Zhongtian Chen, Daniel Murfet |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 164 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-18048
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Modes of Sequence Models and Learning Coefficients Zhongtian Chen, Daniel Murfet Abstract We develop a geometric account of sequence modelling that links patterns in the data to measurable properties of the loss landscape in transformer networks. First, we cast conditional sequence distributions into a Hilbert-space framework and apply tensor decompositions to identify their principal modes . Truncating the small-amplitude modes yields an effective data distribution that preserves dominant structure while discarding statistical detail. Second, we show theoretically that Local Learning Coefficient (LLC) estimates are insensitive to modes below a data-dependent threshold. Consequently, the LLC calculated in practice characterises the geometry of the effective rather than the true distribution. This insight clarifies why reliable LLC estimates can be obtained even when a network parameter is not a strict minimiser of the population loss, and it highlights how the inverse temperature in SGLD acts as a resolution dial on the landscape structure. 1 Introduction Statistical models of sequence distributions form the foundation of our mathematical theories of communication ( Shannon, 1948 ) , artificial general intelligence ( Hutter, 2005 ) , and natural language. The empirical success of transformer neural networks has brought sequence models once again to the center of both theoretical and practical aspects of artificial intelligence ( Vaswani et al., 2017 ) . Recent work has suggested that the geometry of the loss landscape in these networks may be deeply related to the way that the models compute ( Chen et al., 2023 ; Wang et al., 2024 ; Hoogland et al., 2024 ) , and therefore understanding this geometry and how it relates to structure in the data distribution is a promising path towards interpretability and applications in AI safety ( Lehalleur et al., 2025 ) . Within the framework of singular learning theory (SLT) ( Watanabe, 2009 ) the main way we currently access this geometry empirically is through approximate Bayesian posterior sampling using stochastic gradient Langevin dynamics (SGLD) ( Welling & Teh, 2011 ) . This forms the basis for local learning coefficient (LLC) estimation ( Lau et al., 2024 ) and related techniques like susceptibilities ( Baker et al., 2025 ) . However there are many open conceptual and technical questions about this SGLD-based approach to studying the geometry of the loss landscape. In this paper, we lay foundations for understanding the role of modes of the data distribution in shaping our measurements of the geometry of the loss landscape of sequence models like transformers. We do this by developing an effective theory of sequence modelling. Our approach uses function spaces and tensor decompositions to identify the fundamental patterns – which we call modes – in sequence distributions. This framework allows us to perform a principled coarse-graining of the true data distribution by truncating smaller modes while preserving the dominant structures, resulting in what we term the effective true distribution . Our key insight is that the geometry of learning, as captured by LLC estimation, is inherently limited in its sensitivity to the full spectrum of modes. Due to constraints on the sensitivity of the model’s gradients to the long tail of modes, LLC estimation primarily measures the geometry of the effective true distribution rather than the complete distribution. This has several implications for how we interpret measurements of landscape geometry in deep sequence models. The contributions we make in this paper: • Modal decomposition of sequence distributions : Building on existing work in the mathematical literature on natural language processing, we use tensor decompositions to formulate a Hilbert space of sequence models with a natural basis. We show how patterns in the data distribution correspond to modes, and how tensor truncation provides a principled approach to coarse-graining. • LLC estimation measures effective distributions : We prove that under reasonable assumptions, LLC estimation is insensitive to small modes in the distribution. Specifically, the LLC estimates we perform in practice (e.g., on transformer neural networks) effectively measure the geometry of a coarse-grained distribution rather than the complete distribution. The paper is organized as follows: In Section 2 , we review necessary background on tensor algebra and singular value decomposition. Section 3 introduces our formalism for sequence models and their corresponding data distributions. Section 4 develops the Hilbert space framework and modal decomposition. Section 5 makes these abstract definitions concrete with some examples of modes. Then in Section 6 and Section 7 we connect these concepts to Singular Learning Theory and the Local Learning Coefficient and prove our main results. 2 Background 2.1 Tensor Algebra Throughout all vector spaces are over \mathbb{R} , and all tensor products are of \mathbb{R} -vector spaces. Given vector spaces V and W , let \Hom_{\mathbb{R}}(V,W) denote the space of all linear transformations from V to W . There is a canonical isomorphism of vector spaces \displaystyle\Phi:V^{*}\otimes W\overset{\cong}{\longrightarrow}\Hom_{\mathbb{R}}(V,W)\,, (1) \displaystyle\Phi(\eta\otimes w)(v)=\eta(v)w\,. (2) The free vector space on a finite set S is the set \mathbb{R}^{S}=\{f:S\longrightarrow\mathbb{R}\} (3) with the usual vector space structure of addition of functions and multiplication of a function by a scalar. We define an injective map S\rightarrow\mathbb{R}^{S} sending s\in S to the delta function \delta_{s}(t)=\delta_{s,t} and henceforth use this injection to identify elements of S with vectors in \mathbb{R}^{S} . There is a canonical isomorphism of vector spaces \displaystyle\mathbb{R}^{S}\otimes\mathbb{R}^{T}\overset{\cong}{\longrightarrow}\mathbb{R}^{S\times T} (4) \displaystyle s\otimes t\longmapsto(s,t) (5) and in particular for k\geq 1 we have (\mathbb{R}^{S})^{\otimes k}\cong\mathbb{R}^{S^{k}} . 2.2 Singular Value Decomposition For the reader’s convenience we give a minimal treatment of SVD that makes explicit the fact that SVD is an operation on linear transformations between inner product spaces. Let V,W be finite-dimensional inner product spaces over \mathbb{R} with pairings denoted \langle-,-\rangle_{V} and \langle-,-\rangle_{W} respectively. Given a linear transformation A:V\longrightarrow W the adjoint A^{\dagger}:W\longrightarrow V is uniquely defined by \langle Av,w\rangle_{W}=\langle v,A^{\dagger}w\rangle_{V}\qquad\forall v\in V,w\in W\,. Then A^{\dagger}A,AA^{\dagger} are self-adjoint operators on V,W respectively and thus we may apply the spectral theorem to obtain an orthonormal basis of eigenvectors for these spaces. Let \{v_{\alpha}\}_{\alpha} be an orthonormal basis of V consisting of eigenvectors of A^{\dagger}A with eigenvalues \lambda_{\alpha}\geq 0 . We can divide the indices \alpha into those with positive and zero eigenvalues \Lambda^{+}=\{\alpha\mid\lambda_{\alpha}\neq 0\}\,,\qquad\Lambda^{0}=\{\alpha\mid\lambda_{\alpha}=0\}\,. We write \Lambda=\Lambda^{+}\cup\Lambda^{0} for the complete set of indices, so |\Lambda|=\dim V . Then we define u_{\alpha}=\lambda_{\alpha}^{-1/2}Av_{\alpha}\,,\qquad s_{\alpha}=\lambda_{\alpha}^{1/2}\,,\qquad\alpha\in\Lambda^{+}\,. (6) Lemma 2.1 . The set \{u_{\alpha}\}_{\alpha\in\Lambda^{+}} is an orthonormal basis in W for the image of A . We have Av_{\alpha}=s_{\alpha}u_{\alpha}\,,\quad A^{\dagger}u_{\alpha}=s_{\alpha}v_{\alpha}\,. (7) We call s_{\alpha} the singular values , v_{\alpha} the right singular vectors and u_{\alpha} the left singular vectors . Proof. If \lambda_{\alpha}\neq 0 define u^{\prime}_{\alpha}=Av_{\alpha} . Then \displaystyle A^{\dagger}u^{\prime}_{\alpha}=A^{\dagger}Av_{\alpha}=\lambda_{\alpha}v_{\alpha} and hence AA^{\dagger}u^{\prime}_{\alpha}=\lambda_{\alpha}Av_{\alpha}=\lambda_{\alpha}u^{\prime}_{\alpha} Moreover by adjointness \langle u^{\prime}_{\alpha},u^{\prime}_{\beta}\rangle_{W}=\langle u^{\prime}_{\alpha},Av_{\beta}\rangle_{W}=\langle A^{\dagger}u^{\prime}_{\alpha},v_{\beta}\rangle_{V}=\lambda_{\alpha}\langle v_{\alpha},v_{\beta}\rangle_{V}=\lambda_{\alpha}\delta_{\alpha,\beta}\,. If we set u_{\alpha}=\lambda_{\alpha}^{-1/2}u^{\prime}_{\alpha}=\lambda_{\alpha}^{-1/2}Av_{\alpha} then the claims follow. ∎ 2.3 Learning Coefficient In this section we formulate the problem of learning the conditional true distribution q(y|x) of sequences in the context of singular learning theory (SLT) Watanabe (2009) . We assume k,l>0 and x\in\Sigma^{k} , y\in\Sigma^{l} . Given a set of n independent and identically distributed samples D_{n}=\{(x_{i},y_{i})\}_{i=1}^{n} drawn from the true data-generating distribution q(x,y) , we define the sample negative log likelihood L_{n}(w) associated with a model p(y|x,w) and its theoretical counterpart L as \displaystyle L_{n}(w) \displaystyle=-\frac{1}{n}\sum_{i=1}^{n}\log p(y_{i}|x_{i},w)\,, \displaystyle L(w) \displaystyle=-\mathbb{E}_{q(x,y)}\big[\log p(y|x,w)\big]\,. Under some technical hypotheses the local learning coefficient (LLC) ( Watanabe,