Large-Step Training Dynamics of a Two-Factor Linear Transformer Model

Krishnakumar Balasubramanian · 2026

Read Large-Step Training Dynamics of a Two-Factor Linear Transformer Model by Krishnakumar Balasubramanian (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-21292.txt.

Introduction

Krishnakumar Balasubramanian studies large-step training dynamics of a two-factor linear transformer model. Unbiased stochastic gradients match the full gradient on average while changing finite-step stability. In one-layer linear self-attention for in-context linear regression, shared-mode mini-batch training reduces to a random two-factor map. The full-batch map preserves an elliptic region, while sampled curvature creates batch-dependent stability edges whose fluctuation drifts the edge outward.

Catalog record

CREATORKrishnakumar Balasubramanian
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE99608
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.21292

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Large-Step Training Dynamics of a Two-Factor Linear Transformer Model Krishnakumar Balasubramanian Abstract Gradient-flow analyses show that simplified linear transformers can learn the in-context linear-regression algorithm, but they do not explain the finite-step behavior of gradient descent at large learning rates. Motivated by empirical work on high-learning-rate transformer instabilities and by the cubic-map phase diagram for quadratic regression, we study an exactly reducible one-prompt linear-transformer training problem. After normalization, the dynamics reduce to a two-factor product map with an effective step-size parameter μ \mu . On the balanced slice, this map recovers the known scalar cubic transition from monotone convergence to catapult convergence, periodic and chaotic bounded nonconvergence, and divergence. We then analyze the full two-dimensional system and show that, for 0 < μ < 2 0<\mu<2 , it has an explicit invariant Chebyshev ellipse separating forward-invariant regions; this ellipse carries off-balanced chaotic dynamics but is transversely repelling, while balanced scalar attractors can be transversely attracting. These results show that large constant learning rates can change the training attractor of the learned transformer rather than merely accelerating convergence: beyond sharp stability thresholds, finite-step training may settle into cycles, bounded chaos, or divergence instead of a single in-context linear-regression solution. We also discuss the consequences for mini-batch gradient descent based training methods. 1 Introduction Transformers trained on sequences of examples can display in-context learning (ICL): at test time, they use the prompt itself as the data from which to infer the prediction rule. For linear regression prompts, Zhang et al. 2024a gave a rigorous gradient-flow analysis of a single-layer linear self-attention model, proving that from suitable initialization the flow reaches a global minimizer competing with the best linear predictor. This provides a clean theoretical account of how a trained transformer can implement an in-context linear-regression algorithm; see also Wu et al. 2024b ; Zhang et al. 2024b for additional related works in this context. However, gradient flow is the infinitesimal-step limit. Practical transformer training uses finite learning rates, often large enough that various forms of training instabilities occur. Empirically, Wortsman et al. 2024 show that many such training instabilities seen in large-scale transformers can be reproduced in smaller models by increasing the learning rate, with diagnostics such as attention-logit growth and output-logit divergence stabilized by warmup, weight decay, qk-layernorm, and \mu Param; see also Gilmer et al. 2022 . A useful theory of transformer training should therefore not only identify global minima but also explain how finite-step training reaches or fails to reach them as the learning rate varies. A closely related instability phenomenon appears in quadratic regression: Lobacheva et al. 2021 ; Agarwala et al. 2023 ; Chen and Bruna 2023 ; Wang et al. 2022 ; Song and Yun 2023 ; Zhu et al. 2023 ; Chen et al. 2024 ; Zhu et al. 2024 ; Liang and Montufar 2026 show that large-step gradient descent reduces to a cubic map with five phases—monotone convergence, catapult convergence, periodic nonconvergence, chaotic nonconvergence, and divergence. In the transformers, because attention logits are bilinear in the learned factors, large-step training of bilinear parameterizations can produce analogous discrete-time bifurcations that are invisible to gradient flow. Setup and main map. Starting from the one-prompt linear self-attention reduction of Zhang et al. 2024a , finite-step gradient descent reduces, after rescaling, to the two-dimensional map \Phi_{\mu}(a,b)=\bigl(a-(ab-\mu)b,\;b-(ab-\mu)a\bigr),\qquad\mu>0. (1) The parameter \mu is the effective step-size: if the original learning rate is \eta , the prompt-response scale is y , and the two scalar factor updates have relative geometry constant c>0 , then \mu=2\eta|y|\sqrt{c} , so sweeping \eta is equivalent to sweeping \mu . The zero-training-error set is the curve of fixed points \mathcal{M}_{\mu}=\{ab=\mu\} , and the local normal multiplier there is 1-a^{2}-b^{2} ; the stable segment shrinks as \mu increases. In error/imbalance coordinates e=ab-\mu , w=a-b , the map becomes e^{+}=e^{3}+(\mu-2)e^{2}+(1-2\mu-w^{2})e,\qquad w^{+}=(1+e)w. (2) The balanced line w=0 is invariant and gives exactly the cubic map of Chen et al. 2024 , F_{\mu}(e)=e\bigl((e+\mu)(e-2)+1\bigr) , identifying the balanced transformer dynamics with the scalar quadratic-regression phase diagram. With D=w^{2}-(2-\mu)(2-e) , q_{\mu}(e)=e^{2}+\mu e+1 , and C(e)=e^{3}-3e , the full map has the exact normal form e^{+}=C(e)-eD,\qquad D^{+}=q_{\mu}(e)D. (3) For 0<\mu<2 , q_{\mu}>0 , so the sign of D is invariant. The curve D=0 is the ellipse \mathcal{E}_{\mu}=\left\{(a,b):\frac{(a+b)^{2}}{2+\mu}+\frac{(a-b)^{2}}{2-\mu}=4\right\}, (4) and on \mathcal{E}_{\mu} , e^{+}=C(e) ; writing e=2\cos\theta gives e^{+}=2\cos(3\theta) . Thus the two-dimensional map contains an off-balanced Chebyshev chaotic set for every 0<\mu<2 , a repelling separatrix rather than an attractor. For 1<\mu<2 , balanced scalar attracting sets have negative transverse Lyapunov exponent for all non-endpoint invariant measures, so balanced behavior can attract nearby imbalanced trajectories. We provide a basic introduction to dynamical systems relevant for this work in Section A . Contributions. (i) We derive ( 1 ) from finite-step gradient descent on a one-prompt linear self-attention objective. (ii) We identify the balanced invariant line with the cubic family of Chen et al. 2024 , giving explicit learning-rate thresholds for monotone and catapult convergence, period-two bifurcation, chaos and divergence. (iii) We prove exact structural results for the genuine two-dimensional system: local stability of the zero-error curve, the normal form ( 3 ), the invariant Chebyshev ellipse, the sign-invariant decomposition, exact one-step landing sets, and rigidity of the interior D<0 . (iv) We analyze transverse stability ( \mathcal{E}_{\mu} is transversely repelling; balanced invariant measures are transversely attracting for 1<\mu<2 ), and extend the framework to mini-batch GD, showing that mini-batching acts as random switching between maps carrying its own Chebyshev separatrix. Other related works. The training instabilities studied here are related to the edge-of-stability phenomenon in neural-network optimization, where gradient descent operates near or beyond the classical local stability threshold and exhibits nonmonotone loss, oscillatory motion, and sharpness growth; see, for example, Cohen et al. 2021 ; Arora et al. 2022 ; Damian et al. 2023 . Our setting differs in that the reduced transformer dynamics are low-dimensional enough to expose the bifurcations explicitly: the loss of stability of the zero-error set leads to period-two behavior, higher-period dynamics, chaos, or divergence. Large step-sizes can also have beneficial effects, and have been shown to accelerate convergence in settings such as logistic regression [ Wu et al. 2024a ; Zhang et al. 2025 ; Wu et al. 2025 ] ; this is consistent with the catapult regime in our model, where training is nonmonotone but still converges. More broadly, instability and nonconvergence phenomena for deterministic and stochastic gradient methods have been studied in several settings, including large-learning-rate training, stochasticity-induced effects, chaotic optimization dynamics, and neural-network loss landscapes [ Lewkowycz et al. 2020 ; Kong and Tao 2020 ; Kodryan et al. 2022 ; Herrmann et al. 2022 ; Zhang et al. 2022 ] . This work contributes to this line by giving an exact discrete-time mechanism, derived from a linear self-attention training problem, in which large transformer step-sizes create new dynamical attractors rather than merely perturbing the gradient-flow limit. 2 From a one-prompt linear transformer to the two-parameter product map Fix one linear-regression prompt E=\begin{pmatrix}x_{1}&\cdots&x_{N}&x_{\rm q}\\ y_{1}&\cdots&y_{N}&0\end{pmatrix}\in\mathbb{R}^{(d+1)\times(N+1)}, with x_{i},x_{\rm q}\in\mathbb{R}^{d} , y_{i}=\langle w,x_{i}\rangle , and query response y_{\rm q}=\langle w,x_{\rm q}\rangle . A single linear self-attention layer is f_{\rm LSA}(E;\theta)=E+W^{PV}E\frac{E^{\top}W^{KQ}E}{\rho}; following Zhang et al. 2024a we take \rho=N , so that the attention term has the

Read the complete text