Superposition unifies power-law training dynamics Zixin Jessie Chen, Hao Chen, Yizhou Liu, Jeff Gore Abstract We investigate the role of feature superposition in the emergence of power-law training dynamics using a teacher-student framework. We first derive an analytic theory for training without superposition, establishing that the power-law training exponent depends on both the input data statistics and channel importance. Remarkably, we discover that a superposition bottleneck induces a transition to a universal power-law exponent of ∼ 1 \sim 1 , independent of data and channel statistics. This one over time training with superposition represents an up to tenfold acceleration compared to the purely sequential learning that takes place in the absence of superposition. Our finding that superposition leads to rapid training with a data-independent power law exponent may have important implications for a wide range of neural networks that employ superposition, including production-scale large language models. 1 Introduction The remarkable success of large language models (LLMs) is underpinned by neural scaling laws , where model performance scales predictably as a power law with compute, dataset size, and parameter count ( 22 ; 19 ; 38 ; 11 ) . These laws describe not only the final performance but also the trajectory of optimization itself, where the training loss \mathcal{L}(t) often decays as \mathcal{L}(t)\propto t^{-\alpha} over orders of magnitude in training steps. Despite the ubiquity of these dynamics across modalities ( 17 ) and architectures ( 15 ) , the microscopic mechanisms that govern this macroscopic behavior remain a subject of intense debate. Feature 1 Feature 2 Features are orthogonal to each other. Zero Interference (a) No superposition Feature 1 Feature 2 Feature 3 Feature 4 Feature 5 Interference N>K Features packed. Non-zero Interference (b) Superposition Figure 1 : Geometric illustration of superposition. (a) In the absence of superposition, features have zero interference with each other. (b) Superposition compresses features into a smaller latent size, introducing interference among them (e.g., Feature 5 projects onto Feature 1). Superposition Bottleneck ( K\leq N ) \mathbf{x} Input Dim N \mathbf{W} Embed K\times N \mathbf{B} Student K\times K \mathbf{W}^{\top} Unembed N\times K + \mathbf{b} ReLU \mathbf{y} \mathbf{A} Teacher N\times N (Channel Importance) \mathbf{y}^{*} MSE Dim N \mathbf{Ax} , Dim N Dim N Latent \mathbf{K} Latent \mathbf{K} \hat{\mathbf{y}} Figure 2 : The teacher-student model setup under superposition , where K\leq N . The student learns in a compressed latent space via the embedding layers \mathbf{W} and \mathbf{W}^{\top} . A bias and ReLU nonlinearity are applied to the output to suppress interference noise. Standard theories of learning often attribute these power laws to the spectral properties of the data ( 2 ; 4 ; 32 ) . In this view, learning is a sequential spectral filtering process where the model fits eigenmodes of the data covariance in descending order of magnitude ( 30 ; 6 ; 9 ) . While this spectral bias successfully explains dynamics in linear regimes and kernel methods ( 29 ; 12 ; 5 ; 7 ) , it typically assumes a direct mapping between model dimensions and data features. However, modern LLMs operate in a fundamentally different regime: the latent representations of features are highly non-orthogonal of one another, a phenomenon known as superposition ( 13 ) . Superposition in LLMs. The dimensionality of the embedding space in LLMs is a critical bottleneck. Models must learn embedding vectors for vocabularies ranging from 32,000 to 256,000 tokens ( 1 ; 25 ) alongside millions of abstract concepts ( 34 ; 14 ) , yet map them into hidden states of only a few thousand dimensions. To surmount this limit, models utilize superposition to store features in non-orthogonal directions ( 16 ; 8 ) . While this increases capacity, it introduces “interference noise”. Notably, prior works on neural scaling laws often implicitly assume a regime of sufficient width , where model dimensions are sufficient to cover the feature space ( 24 ) . This assumption is disconnected from the strong superposition regime of production models ( 23 ) , raising the question: how does the interference noise inherent in superposition alter the training dynamics? To answer this, we must move beyond black-box observation. Just as controlled toy models were essential for isolating the mechanisms of deep linear networks ( 31 ) , we propose a tractable teacher-student framework to model the training dynamics of LLMs under superposition. Our model strips away architectural complexities of Transformers to focus entirely on the interaction between feature structure and dimensional constraints. Our investigation yields a surprising divergence in training regimes. In the absence of superposition, the student learns sequentially, with power-law training dynamics strictly determined by input data statistics and channel decay. However, when forced into superposition, the training dynamics becomes unified. The randomness of superposition “mixes” features across different channels, effectively equalizing their learning behavior. This leads to a universal power-law exponent of \alpha\approx 1 , independent of the specific data input and channel decay, while representing an acceleration over the sequential case. 1.1 Contributions Our contributions are as follows: • We introduce a teacher-student toy model that captures the interplay among feature distribution, channel importance, and superposition, reflecting constraints found in production-scale LLMs. • We establish an analytic theory for power-law training dynamics in the no-superposition regime, explicitly linking the training exponent to input data statistics and channel importance. • We demonstrate empirically that the superposition regime induces a transition to universal, accelerated power-law training with an exponent \alpha\approx 1 . • We bridge superposition and scaling laws of training dynamics, proposing that the superposition regime characteristic of LLMs leads to a uniform, accelerated training trajectory compared to the sufficient-width regimes assumed in prior works. Organization. The remainder of this paper is organized as follows. Section 2 introduces the toy model setup. Section 3 derives the analytic theory for the no-superposition model. Section 4 presents the empirical results for universal acceleration and analyzes the optimal-compute scaling frontiers. Finally, Section 5 provides a mechanistic explanation for the universal behavior under superposition. We conclude in Section 6 . (a) Theory prediction matches the empirical loss trajectory (b) Exponents match \alpha=(a+2b-1)/a Figure 3 : Without superposition, learning exponents depend on both input data and channel statistics. We verify the analytic theory for the no-superposition baseline ( N=K=1024 ). (a) Empirical loss curves (solid) track the theoretical predictions (dashed) across varying input decays a and channel importances b . (b) The fitted power-law exponents align precisely with the derived scaling law \alpha=(a+2b-1)/a , confirming that learning is strictly governed by data and channel statistics. 1.2 Related works Neural scaling laws. Power-law scaling is well-established for pre-training ( 22 ; 19 ) , transfer learning ( 18 ) , and emergent abilities ( 36 ) . Extensions of these laws have explored data pruning ( 33 ) , mixture-of-experts architectures ( 11 ) , and state-space models ( 15 ) . Theoretical derivations typically rely on kernel methods or infinite-width limits ( 6 ; 9 ) , attributing scaling to the decay of the data manifold’s spectrum ( 32 ) or the resolution of singularities in the loss landscape ( 35 ) . Our work departs from this by investigating the superposition regime —where width is the bottleneck—a constraint often absent in kernel-based theories but definitive of LLMs ( 23 ) . Theory of learning. A central debate in deep learning theory concerns the distinction between the Neural Tangent Kernel (NTK) regime ( 21 ; 3 ) and the feature learning regime ( 10 ; 37 ) . While the NTK regime predicts dynamics governed by fixed spectral properties ( 30 ; 29 ; 5 ; 7 ) , LLMs are believed to operate in the feature regime where representations evolve. Our work shows that superposition acts as a bridge: while the student model is linear (like many NTK analyses), the feature mixing induces a collective learning dynamic that breaks the standard spectral linkage found in the NTK regime. Mechanistic interpretability. Mechanistic interpretability aims to reverse-engineer neural networks into understandable components ( 27 ) . A key focus is the superposition hypothesis , which posits that models represent more features than they have neurons ( 13 ) . Recent advances have utilized sparse autoencoders to disentangle these mixed representations ( 8 ; 20 ) . However, most studies focus on static, trained networks. Research on the dynamics of learning has focused largely on “grokking” in algorithmic tasks ( 28 ; 26 ) . We contribute to this by isolating superposition as a mechanism that actively shapes the continuous scaling laws of training dynamics. 2 Toy model setup To investigate the interplay between feature superposition and training dynamics, we propose a teacher-student framework designed to simulate the feature structure and dimensional constraints characteristic of LLMs. 2.1 Task definition: data and channels We define the learning task via a sparse input distribution and a teacher model, mimicking the multiscale structure of natural language data and Transformer channel importance. Input features. We consider an input vector \mathbf{x}\in\mathbb{R}^{N} where each component x_{i} represents a feature activation. To capture the sparsity of natural data, x_{i} is defined as a mixture of Bernoulli and Uniform distributions: x_{i}=u_{i}v_{i},\quad u_{i}\sim\text{Bernoulli}(p_{i}),\quad v_{i}\sim U(0,1). (1) Here, p_{i} governs feature frequency and follows a power-law decay p_{i}\propto i^{-a} (with a>1 ). This distribution ensures that lower-index features are significantly more frequent than higher-index ones. We normalize p_{i} with an activation density E=\sum_{i}p_{i}=1 to maintain consistent activation density across varying N . Teacher model. The target signal is generated by a fixed teacher matrix \mathbf{A}\in\mathbb{R}^{N\times N} , which models the channel importance observed in Transformer blocks. We impose a power-law spectral decay on the diagonal entries: A_{ii}=i^{-b},\quad A_{ij}=0\text{ for }i\neq j, (2) where b\geq 0 is a decay constant. This structure forces the model to prioritize learning lower-index features to minimize loss, establishing an ordered sequence in the optimization landscape. 2.2 Student architecture and superposition The student model f_{\theta}(\mathbf{x}) attempts to reconstruct the teacher’s output \mathbf{y}^{*}=\mathbf{Ax} subject to a bottleneck dimension K . Superposition and interference. As illustrated in Figure 1 , when K1). (8) This result establishes that in the sequential learning regime, the training speed is strictly coupled to the input statistics ( a ) and channel importance ( b ). As a,b increases, the input becomes sparser and channel decays faster, leaving less new information for the student to learn from the teacher, resulting in a faster learning trajectory. (a) Optimal-compute frontier (b) Iso-compute curves identify optimal student sizes Figure 7 : Demonstration of optimal-compute frontier with model size scaling. We analyze the trade-off between model size and training duration for a fixed input distribution ( a=1.1,b=0 ). (a) The raw training curves (colored lines) are enveloped by a black solid line, defining the optimal-compute frontier. (b) We plot loss against student size K for four fixed compute budgets C (log-uniform). The distinct minima in each curve illustrate the trade-off between K and C , mirroring the behavior observed in LLMs. 4 Experiments We now investigate the training dynamics under superposition, where an analytic exponent becomes intractable due to feature mixing. We perform experiments on a toy model of size N=1024 trained via SGD with online data generation, mimicking the infinite-data regime of LLM pre-training. 4.1 Methodology To quantify the training dynamics, we adapt the Chinchilla scaling law ( 19 ) to account for the distinct phases of optimization. The standard scaling law models loss as a sum of time-dependent and parameter-dependent terms: \mathcal{L}(t,K)\approx c_{t}t^{-\alpha}+c_{k}K^{-\beta}+\mathcal{L}_{0}. (9) However, in our teacher-student setup, optimization proceeds in two distinct regimes: an early-to-mid phase dominated by optimization error (time-limited), and a late phase dominated by the bottleneck capacity (width-limited), as represented in Figure 4 . In our analysis, we focus on the mid-training stage, where c_{t}t^{-\alpha}+\mathcal{L}_{0} dominates, to extract the effective training exponent \alpha . 4.2 Results We vary the input decay a\in[1.1,2.0] , channel importance b\in[0.0,0.5] , and student dimension K\in[32,1024] . Theory verification (no superposition). For the baseline case ( K=N ), we confirm our theoretical derivation. As shown in Figure 3(b) , the empirical exponents match the predicted \alpha=(a+2b-1)/a precisely. Superposition regime. When the student is forced into superposition ( K1 is only needed for power-law theory approximation. We can conduct experiments for any a,b\geq 0 . Regardless of the specific values of a , b , or the compression ratio K/N , the mid-training power-law exponent converges to \boldsymbol{\alpha\approx 1} . Examples of the loss used for fitting the training exponents are shown in Figure 6 . This represents a universal acceleration compared to the sequential case. For example, with a=1.1 and b=0 , the sequential theory predicts a slow decay of \alpha\approx 0.09 . Superposition accelerates this by over \mathbf{10\times} . This universality suggests that the randomness inherent in the embedding layer acts as a mechanism to equalize the effective learning rates across features, decoupling the optimization dynamics from the spectral decay of the data. In Appendix A.2 , we analyze two cases of the superposition model in linear regime: the limits of maximum positive interference and isotropic mixing. They provide a theoretical intuition for the acceleration and uniformity observed in empirical results. However, we stress that they do not account for the \alpha\approx 1 exponent, which emerges only with ReLU nonlinearity and bias. 4.3 Optimal-compute scaling While superposition universally accelerates mid-training dynamics regardless of the student size, the final attainable performance of the model is limited by the bottleneck dimension K . To understand the trade-offs between model size and training steps, we analyze the optimal-compute frontier. In Figure 7(a) , we plot the loss curves for students of varying sizes ( K\in[32,1024] ) against total compute (approximated as C=t\times K^{2} ). We observe a clear optimal envelope , indicated by the black line, composed of the minimum loss achievable at any given compute budget. Consistent with observations in large-scale Transformers, the optimal model size increases with the compute budget. As shown in Figure 14 , K doubles roughly as C increases by tenfold. We run a preliminary power-law fitting and obtain a size scaling exponent of K against C at around 0.2702 . We further quantify this relationship using iso-compute curves (Figure 7(b) ), which document how loss varies with student size at fixed compute levels. These curves demonstrate that for a fixed compute budget, the minimum loss scales inversely with the student size as a power law. We also discuss experimental results on the loss scaling with student size (i.e. the width-limited loss term c_{k}K^{-\beta} ) in Appendix A.4 . This confirms that our toy model, despite its simplicity, recapitulates the macroscopic scaling behaviors observed in production LLMs while offering a tractable testbed for understanding the underlying mechanisms. (a) No superposition: sequential “traveling wave” (b) Superposition: parallel “global decay” Figure 8 : Mechanism of universal loss dynamics under superposition. We visualize the per-feature loss L_{i}(t) over time. (a) Without superposition, the model learns features sequentially, creating a traveling wave-front where tail features remain unlearned for long durations. (b) Under superposition, feature mixing equalizes the effective gradients, causing all features—regardless of importance—to be learned in parallel. Figure 9 : Visualizing the student model learning pattern: sequential (no superposition) vs parallel (superposition). Snapshots of the student matrix \mathbf{B} at t=10^{3} . Left: In the absence of superposition, the student learns strictly along the diagonal, solving features sequentially (only the first \sim 400 are learned). Right: The superposition student distributes weights across the entire matrix, learning all 1024 features in parallel. 5 Discussion The central finding of our work is the universal acceleration of training dynamics under superposition. In this section, we investigate the microscopic mechanism driving this phenomenon. Specifically, we address two entangled questions: (1) why is the exponent increased during mid-training, and (2) why is this behavior universal across different data distributions? The answer lies in how the model processes the input data and channel statistics. We contrast the sequentiality of the baseline case with the parallelism induced by superposition. 5.1 Mechanism Sequential learning as a traveling wave. In the absence of superposition, the input probability p_{i} and channel importance A_{ii} follow strict power-law decays by design. This structure forces the gradient descent dynamics to respect an ordered sequence: the model learns features in descending order of importance. This phenomenon is illustrated in Figure 8(a) , where we plot the loss contribution of individual entries over time in the absence of superposition. The dynamics resemble a “traveling wave-front”. Features with lower indices (high importance/frequency) are learned first, while features with higher indices (low importance/frequency) remain effectively frozen—their losses do not decrease until the prior, more dominant features are resolved. Consequently, at any point t in mid-training, the total loss is dominated by the accumulated error of the vast number of unlearned features in the tail of the distribution. The rate of loss convergence is thus strictly determined by the spectral decay of the data. Superposition disrupts the sequential order. Superposition, via the random embedding \mathbf{W} , disrupts this sequence. The compression mixes features from different positions, while normalization per feature ensures they are projected with comparable magnitude. This leads to two fundamental changes: 1. Feature mixing: Distinct entries are blended into the latent space, making it impossible for the student to isolate and learn them sequentially. 2. Frequency equalization: On average, every dimension of the student matrix \mathbf{B} receives a signal that aggregates high-frequency and low-frequency features alike. As a result, the student learns all entries simultaneously at a comparable rate. This is demonstrated in Figure 8(b) . Unlike the “traveling wave” without superposition, the per-entry losses in the superposition regime remain clustered at similar levels and decay in unison. The concept of “unlearned tail features” effectively vanishes. Mid-training acceleration. Why does this uniformity under superposition lead to acceleration? We emphasize that this advantage is specific to the mid-training regime. Since the superposition student ( K1 ). Blue: Inputs with heavy power-law tails converge slowest ( \alpha<1 ). Dashed: Exponentially decaying inputs (and superposition models) converge at \alpha\approx 1 . A.1.1 Continuous Approximation Assuming near-uniform channel importance (in the limit b\rightarrow 0 ), the loss function can be approximated by a continuous integral in the limit of N\gg 1 . Let i be the discrete feature index and z be its continuous counterpart. The loss is: \mathcal{L}(t)\propto\sum_{i=1}^{N}f(i)e^{-f(i)t/\tau}\approx\int_{0}^{\infty}f(z)e^{-f(z)t/\tau}dz, (10) where f(z) is the monotone decreasing data frequency. We define a critical feature index z_{c}(t) where the exponent is of order unity: f(z_{c})\cdot\frac{t}{\tau}=1\implies z_{c}(t)=f^{-1}\left(\frac{\tau}{t}\right). (11) At time t , features with zz_{c} contribute to the loss ( e^{-f(z)t/\tau}\approx 1 ). The loss is dominated by the tail integral: \mathcal{L}(t)\sim\int_{z_{c}(t)}^{\infty}f(z)dz. (12) Note that for this approximation to be valid, 1\ll z_{c}\ll N , corresponding to the mid-training stage in the dynamics. A.1.2 Power-Law Decay Let f(z)=Cz^{-a} with a>1 . The critical index scales as: z^{-a}_{c}\sim\frac{\tau}{t}\implies z_{c}(t)\sim\left(\frac{t}{\tau}\right)^{1/a}. (13) The loss integral becomes: \mathcal{L}(t)\sim\int_{z_{c}}^{\infty}z^{-a}dz=\left[\frac{z^{1-a}}{1-a}\right]_{z_{c}}^{\infty}\propto z_{c}^{1-a}. (14) Substituting z_{c}(t) : \mathcal{L}(t)\propto\left(t^{1/a}\right)^{1-a}=t^{-(1-1/a)}. (15) Note that for a nontrivial channel decay with b>0 , we can substitute z^{-a}\rightarrow z^{-(a+2b)} . The critical feature index z_{c}(t) only depends on the input data and remains unchanged. Under this substitution, we recover the loss power law shown in Equation 8 . A.1.3 Exponential Decay Let f(z)=Ce^{-\kappa z^{a}} with \kappa,a>0 . We perform a change of variables u=f(z) . Then z=f^{-1}(u)=\kappa^{-1/a}(\log(C/u))^{1/a} . The differential is: dz=\frac{d}{du}\left(\kappa^{-1/a}(\log(C/u))^{1/a}\right)du\sim-\frac{1}{au}(\log(C/u))^{1/a-1}du. (16) Substituting this into the integral \int f(z)e^{-f(z)t/\tau}dz : \mathcal{L}(t)\sim\int_{0}^{C}ue^{-(t/\tau)u}\cdot\frac{1}{u}\left(\log\frac{C}{u}\right)^{1/a-1}du. (17) We rescale s=(t/\tau)u , so du=(\tau/t)ds . In the limit t\gg 1 , the integral is dominated by small u (finite s ), so \log(C/u)=\log(Ct/\tau s)\approx\log(t) . Pulling the log term out: \mathcal{L}(t)\sim\left(\log t\right)^{1/a-1}\int_{0}^{\infty}e^{-s}ds\cdot\frac{\tau}{t}. (18) Thus, the scaling is: \mathcal{L}(t)\propto t^{-1}(\log t)^{\frac{1}{a}-1}. (19) The dominant term is t^{-1} , representing the decay rate for light-tailed distributions. A.1.4 Algebraic Decay Let f(z)\sim(z_{*}-z)^{a} near a finite edge z_{*} . z approaches z_{*} from the left: z\to z^{-}_{*} . The critical index condition (z_{*}-z_{c})^{a}\sim\tau/t implies the gap \Delta z=z_{*}-z_{c}\sim t^{-1/a} . The loss integral is over the gap \Delta z : \mathcal{L}(t)\sim\int_{z_{c}}^{z_{*}}(z_{*}-z)^{a}dz\propto(z_{*}-z_{c})^{a+1}\sim(t^{-1/a})^{a+1}=t^{-(1+1/a)}. (20) A.1.5 Derivation Summary 1. Power-law decay: For f(z)\sim z^{-a} (where a>1 ), the heavy tail of the data slows down learning. \mathcal{L}(t)\sim t^{-(1-1/a)}. (21) As shown in Fig. 10 (blue curve), \alpha approaches 1 only as a\to\infty (steep decay). 2. Exponential decay: For f(z)\sim e^{-\kappa z^{a}} , the distribution decays fast enough that the dynamics are dominated by the \mathcal{O}(1) time constant. \mathcal{L}(t)\sim t^{-1}(\log t)^{\delta}\implies\alpha\approx 1. (22) This represents the dividing line in our phase diagram. 3. Algebraic decay: For distributions with finite support ending at z_{*} , decaying as (z_{*}-z)^{a} , the scarcity of ”hard” examples vanishes. \mathcal{L}(t)\sim t^{-(1+1/a)}. (23) This yields fast convergence ( \alpha>1 ), shown in red in Fig. 10 . A.1.6 Power-law Verification (a) Exponential Decay ( p_{i}\propto e^{-0.5i} ) (b) Linear Algebraic Decay ( p_{i}\propto N-i ) Figure 11 : Verification of derived scaling laws for general distributions. The solid curves show the exact discrete theoretical loss, while the dashed lines show the power-law fit. (a) Exponential input distribution yields \alpha\approx 1 . (b) Linear algebraic input distribution yields \alpha\approx 2 . Both match the predictions from our continuous approximation theory. We verify our power-law approximation against the theoretical loss (modifications based on Equation 7 ) for non-power-law distributions. More specifically, we test an exponential decay and a linear algebraic decay. We define the normalized feature frequencies f(i)=p_{i} , normalizing them such that \sum_{i}p_{i}=1 . • Exponential decay: p_{i}\propto e^{-0.5i} . Here, a=1 in the exponential-decay class, and the power-law approximation predicts \mathcal{L}(t)\propto t^{-1} , i.e. \alpha=1 . • Linear algebraic decay: p_{i}\propto N-i . Here, a=1 in the algebraic-decay class, and the power-law approximation predicts \mathcal{L}\propto t^{-(1+1/1)}=t^{-2} , i.e. \alpha=2 . In Figure 11 , we fit power-law training exponents to the exact discrete-sum theoretical loss. For the exponential case (Figure 11(a) ), we obtain a fitted exponent of \alpha=0.9769\pm 0.0012 , closely matching the approximate prediction of \alpha=1 . For the linear decay case (Figure 11(b) ), we obtain \alpha=1.9834\pm 0.0007 , in excellent agreement with the theoretical prediction of \alpha=2 . These results confirm that our continuous integral approximation (Section A.1 ) accurately captures the discrete training dynamics across distinct universality classes. A.2 Theoretical loss for the linear superposition model We derive the analytical loss dynamics for the superposition model in the linear regime. We analyze how the structure of the embedding matrix \mathbf{W} influences the effective learning dynamics by considering two limiting cases: (1) \mathbf{W} is a designed cluster maximizing positive interference, and (2) \mathbf{W} is composed of random vectors in the isotropic limit of N\gg K . Specifically, limit (1) elucidates the mechanism of acceleration , while limit (2) explains the emergence of universality . The empirical superposition experiments presented in the main text lie between these two extremes, inheriting both the speed of constructive interference and the uniformity of isotropic mixing. A.2.1 General Setup We consider a linear student \mathbf{y}=\mathbf{W}^{\top}\mathbf{B}\mathbf{W}\mathbf{x} trying to mimic a linear teacher \mathbf{y}^{*}=\mathbf{A}\mathbf{x} . The loss is: \mathcal{L}=\frac{1}{2}\text{Tr}\left[(\mathbf{W}^{\top}\mathbf{B}\mathbf{W}-\mathbf{A})^{\top}(\mathbf{W}^{\top}\mathbf{B}\mathbf{W}-\mathbf{A})\mathbf{\Sigma}\right], (24) where \mathbf{\Sigma}=\mathbb{E}[\mathbf{x}\mathbf{x}^{\top}]\approx\text{diag}(\sigma_{1}^{2},\dots,\sigma_{N}^{2}) is the data covariance, dominated by the diagonal variances. Defining the projection covariance \mathbf{C}\equiv\mathbf{W}\mathbf{\Sigma}\mathbf{W}^{\top} and the correlation matrix \mathbf{P}\equiv\mathbf{W}\mathbf{W}^{\top} , the loss simplifies to: \mathcal{L}=\frac{1}{2}\left[\text{Tr}(\mathbf{A}\mathbf{\Sigma}\mathbf{A})+\text{Tr}(\mathbf{B}^{\top}\mathbf{P}\mathbf{B}\mathbf{C})-2\text{Tr}(\mathbf{S}\mathbf{B})\right], (25) where \mathbf{S}=\mathbf{W}\mathbf{\Sigma}\mathbf{A}\mathbf{W}^{\top} represents the projected teacher signal. The gradient flow dynamics for the student matrix \mathbf{B} are given by: \frac{d\mathbf{B}}{dt}=-\eta\frac{\partial\mathcal{L}}{\partial\mathbf{B}}=-\eta(\mathbf{P}\mathbf{B}\mathbf{C}+\mathbf{C}\mathbf{B}\mathbf{P}-\mathbf{S}-\mathbf{S}^{\top})/2. (26) For symmetric matrices, we analyze the leading order dynamics: \dot{\mathbf{B}}\approx-\eta(\mathbf{P}\mathbf{B}\mathbf{C}-\mathbf{S}) . A.2.2 Case 1: Clustered Superposition To analyze the acceleration mechanism, we construct a structured embedding. We partition the N features into K clusters \{G_{1},\dots,G_{K}\} . We assume an idealized ”hard” assignment where each feature i\in G_{j} is mapped perfectly to dimension j : W_{ji}=\begin{cases}1&\text{if }i\in G_{j}\\ 0&\text{otherwise}\end{cases}. (27) In this limit, the matrices \mathbf{P} and \mathbf{C} become diagonal: \mathbf{P}=\text{diag}(n_{1},\dots,n_{K}),\quad\mathbf{C}=\text{diag}\left(\sum_{i\in G_{1}}\sigma_{i}^{2},\dots,\sum_{i\in G_{K}}\sigma_{i}^{2}\right), (28) where n_{j}=|G_{j}| is the cluster size. The teacher projection \mathbf{S} is also diagonal with entries S_{jj}=\sum_{i\in G_{j}}A_{ii}\sigma_{i}^{2} . In this symmetric, diagonal limit, the ODE decouples for the diagonal entries b_{j} of \mathbf{B} : \dot{b}_{j}\approx-\eta(n_{j}C_{jj}b_{j}-S_{jj}). (29) This linear ODE has the solution: b_{j}(t)=\frac{S_{jj}}{n_{j}C_{jj}}\left(1-e^{-\eta(n_{j}C_{jj})t}\right). (30) The solution describes an exponential relaxation. Crucially, the effective convergence rate for cluster j is \lambda_{eff}\propto n_{j}C_{jj}=n_{j}\sum_{i\in G_{j}}\sigma_{i}^{2} . This explicitly shows that clustering accelerates learning by summing the variances (importance) of all features in the cluster. Interpretation: This scenario represents an extreme limit of superposition characterized by maximum positive interference. This is analogous to the ”privileged basis” discussed in 13 . By perfectly aligning features into the same subspace, the model achieves two multiplicative acceleration effects: (i) the summation of signal energy ( \sum\sigma^{2} ) and (ii) the accumulation of gradients from n_{j} features. This results in a quadratic acceleration scaling \sim n_{j}^{2}\bar{\sigma}^{2} , representing a theoretical upper bound of training speed, albeit at the cost of high collision error. A.2.3 Case 2: Isotropic Limit We now consider the isotropic limit where \mathbf{W}\in\mathbb{R}^{K\times N} is a random matrix with N\gg K . We assume the embedding is column-normalized such that the N feature vectors are unit length \|\mathbf{w}_{i}\|^{2}=1 . This corresponds to entries W_{ji}\sim\mathcal{N}(0,1/K) . Due to the isotropy of random vectors, the correlation matrix \mathbf{P} becomes a scaled identity matrix determined by the frame potential: \mathbb{E}[\mathbf{W}\mathbf{W}^{\top}]=\sum_{i=1}^{N}\mathbb{E}[\mathbf{w}_{i}\mathbf{w}_{i}^{\top}]=\frac{N}{K}\mathbf{I}_{K}. (31) Next, we consider the projection covariance \mathbf{C}=\mathbf{W}\mathbf{\Sigma}\mathbf{W}^{\top} . With \mathbf{\Sigma}=\text{diag}(\sigma_{i}^{2}) , the matrix averages the input variance scaled by the compression ratio: \mathbb{E}[\mathbf{C}]=\sum_{i=1}^{N}\sigma_{i}^{2}\mathbb{E}[\mathbf{w}_{i}\mathbf{w}_{i}^{\top}]=\frac{1}{K}\left(\sum_{i=1}^{N}\sigma_{i}^{2}\right)\mathbf{I}=\frac{N}{K}\bar{\sigma}^{2}\mathbf{I}. (32) Substituting these into the dynamics (Eq. 26 ): \frac{d\mathbf{B}}{dt}\approx-\eta\left[\left(\frac{N}{K}\mathbf{I}\right)\mathbf{B}\left(\frac{N}{K}\bar{\sigma}^{2}\mathbf{I}\right)-\mathbf{S}\right]=-\eta\left(\frac{N}{K}\right)^{2}\bar{\sigma}^{2}\mathbf{B}+\eta\mathbf{S}. (33) Interpretation: This represents the maximum uniform/isotropic limit of superposition. This result explains the universality. In the non-superposition case, each feature i learns at its own rate \lambda_{i}\propto i^{-a} , causing a bottleneck at the tail. With random superposition, the randomness acts as a ”uniform clustering” of all features. It statistically achieves the same quadratic scaling (N/K)^{2} as the uniformly clustered case. This spectral unification ensures that the student matrix \mathbf{B} converges uniformly at a single effective rate, eliminating the spectral bottleneck and leading to a universal exponent. A.3 Is randomness near-optimal for mid-training? In our main analysis, we utilized fixed random projections for the embedding layers \mathbf{W} and \mathbf{W}^{\top} . A natural question arises: is the universal acceleration observed in mid-training an artifact of fixed randomness, or does it persist when the embeddings are optimized? To address this, we modify the toy model setup such that both the encoder \mathbf{W} and decoder \mathbf{W}^{\top} are fully learnable parameters. This increases the parameter count from K^{2} (student only) to K^{2}+2NK . We repeat the superposition experiments with input decay a=1.1 , channel importance b=0.0 , and feature dimension N=1024 , sweeping student sizes K\in\{128,256,512\} . We fit the power-law training exponent in the same mid-training window ( t\sim 10^{3} to 10^{4} ) used in the fixed-embedding experiments. Results. As shown in Figure 12(a) , we find that the power-law training dynamics remain robust to this architectural change. The fitted exponents for the learnable case are: \alpha_{128}=1.0812\pm 0.0215,\quad\alpha_{256}=0.9933\pm 0.0149,\quad\alpha_{512}=0.9946\pm 0.0206. These values are indistinguishable from the \alpha\approx 1 regime observed with fixed random embeddings. This suggests that for mid-training dynamics, the ”mixing” provided by random initialization is sufficient to induce the universal acceleration effect; explicit gradient optimization of the basis directions does not further accelerate the rate of loss convergence (i.e. the power =-law training exponent). Performance gap. While the exponent remains unchanged, learnable embeddings do offer a performance advantage. In Figure 12(b) , we plot the loss difference \Delta\mathcal{L}=\mathcal{L}_{\text{fixed}}-\mathcal{L}_{\text{learnable}} . The positive difference indicates that the learnable model achieves a lower absolute loss. However, the magnitude of this improvement is \mathcal{O}(10^{-3}) , which is a secondary effect compared to the absolute loss magnitude of \mathcal{O}(10^{-2}) . We conclude that randomness is near-optimal for the mid-training scaling laws. We emphasize, however, that this claim applies specifically to the mid-training regime; given infinite training time, the learnable model will naturally converge to a lower final loss plateau due to its increased capacity and ability to align the bottleneck with the principal components of the data. (a) Loss Dynamics (Learnable \mathbf{W} ) (b) Loss Difference ( \mathcal{L}_{\text{fixed}}-\mathcal{L}_{\text{learnable}} ) Figure 12 : Comparison of fixed vs. learnable embeddings. (a) Training curves for the learnable \mathbf{W} model show the same \alpha\approx 1 power-law decay as the fixed random model. (b) The loss difference between fixed and learnable models. While learnable embeddings achieve slightly lower loss (positive difference), the improvement is an order of magnitude smaller than the total loss, indicating that randomness is the primary driver of the training exponent. A.4 Width-limited loss scaling For completeness, we analyze the scaling of the width-limited loss term \mathcal{L}_{t\to\infty}(K)\sim K^{-\beta}+\mathcal{L_{0}} , which dictates the irreducible error floor at the end of training due to model capacity. We investigate the relationship between the final saturation loss and the student dimension K using the superposition setup with fixed input and channel statistics ( a=1.1,b=0.0 ). We sweep the bottleneck size K from 32 to 512 . To isolate the capacity-limited term, we calculate the average loss from the final training plateau (late-time dynamics), distinct from the mid-training power-law regime analyzed in the main text. As shown in Figure 13 , the final loss follows a clear power-law scaling with respect to width. A fit to the data yields an exponent of \beta=1.30\pm 0.01 . This result aligns closely with width-scaling laws reported in recent studies on superposition ( 23 ) , confirming that in addition to a universal rate of learning ( \alpha ), the capacity loss of the model scales roughly inversely with the bottleneck dimension K . Figure 13 : Scaling of the final plateau loss against student width K . The loss decays as a power law K^{-\beta} with \beta\approx 1.3 , consistent with capacity scaling laws in superposition regimes. Figure 14 : Optimal model size scales with compute. The optimal student size K extracted from the frontier scales with the compute budget, enabling the prediction of optimal resource allocation. Under a rough estimation, the size scaling exponent against compute is \approx 0.2702 .