Symmetry Breaking in Transformers for Efficient and Interpretable Training

Eva Silverstein, Daniel Kunin, Vasudev Shyam · 2026

Read Symmetry Breaking in Transformers for Efficient and Interpretable Training by Eva Silverstein, Daniel Kunin, Vasudev Shyam (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-22257.txt.

Introduction

Eva Silverstein, Daniel Kunin, and Vasudev Shyam found spare spin in how a transformer model pays heed to words. In their 2026 preprint they note that the heed step holds twist room that never changes the output. They add a simple fix that picks a set path in that room.

The fix uses small shifts, picked per batch and not learned, for two parts of the step. The team state two gains that tests then backed up. Their work aims to make training both leaner and easier to read. It gives model builders a cheap new dial to try.

Catalog record

CREATOREva Silverstein, Daniel Kunin, Vasudev Shyam
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE75 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-22257

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Symmetry Breaking in Transformers for Efficient and Interpretable Training Eva Silverstein, Daniel Kunin, Vasudev Shyam Abstract The attention mechanism in its standard implementation contains extraneous rotational degrees of freedom that are carried through computation but do not affect model activations or outputs. We introduce a simple symmetry-breaking protocol that inserts a preferred direction into this rotational space through batchwise-sampled, unlearned query and value biases. This modification has two theoretically motivated and empirically validated consequences. First, it can substantially improve the performance of simple, memory-efficient optimizers, narrowing—and in some cases closing—the gap to successful but more complex memory-intensive adaptive methods. We demonstrate this by pretraining 124M parameter transformer models with four optimization algorithms (AdamW, SOAP, SGDM, and Energy Conserving Descent(ECD)) and evaluating both validation loss and downstream logical reasoning. Second, it enables an interpretable use of otherwise redundant rotational degrees of freedom, selectively amplifying semantically meaningful token classes within individual attention heads. Overall, our results show that minimal, principled architectural changes can simultaneously improve performance and interpretability. 1 Introduction Transformers exhibit remarkable capabilities, and it is of interest to understand and improve their architecture and training in order to further accelerate progress in artificial intelligence. In this work, we introduce a simple architectural modification—batchwise, untrained query and value biases—that facilitates more efficient optimization and improves model performance. We motivate this modification theoretically and evaluate it empirically by pretraining GPT-2 (124M) models, measuring both validation loss and downstream logical reasoning performance. In practice, transformer training has been dominated by adaptive or preconditioned variants of stochastic gradient descent with momentum (SGDM), such as AdamW ( Kingma & Ba 2015 ; Loshchilov & Hutter 2019 ) and SOAP ( Vyas et al. 2024 ) . These methods are typically memory intensive, requiring on the order of 3N auxiliary variables for an N -parameter model, and often rely on heuristic arguments to motivate how adaptivity is implemented. By contrast, the recently proposed Energy Conserving Descent (ECD) family of optimizers ( De Luca & Silverstein 2022 ; De Luca et al. 2023 ; De Luca et al. 2025 ; Robnik et al. 2023 ) adopts a predictive physics-inspired framework grounded in Hamiltonian dynamics and requires only 2N auxiliary variables. Despite these appealing properties—and demonstrated advantages in scientific applications—prior work has shown that ECD does not match the empirical performance of adaptive optimizers in transformer training. In this work, we identify a reason for this disparity and develop a simple intervention that overcomes it while providing additional, interpretable benefits. Our analysis focuses on the symmetry structure of the transformer architecture and its interaction with learning dynamics, building on earlier perspectives on symmetry and optimization ( Kunin et al. 2020 ; Tanaka & Kunin 2021 ) . The key architectural component of transformers—the attention head—is invariant under a large group of continuous rotational symmetries ( Zhang et al. 2025 ) . For each head, a joint rotation of the query and key matrices preserves attention scores, since these depend only on their inner products; an analogous symmetry appears in the value–output sector. Although these directions carry no gradient signal and leave activations unchanged, they nevertheless shape the optimization dynamics. We analyze how this symmetry interacts with optimization, showing in particular why it can obstruct Energy Conserving Descent. From a Hamiltonian perspective, the rotational invariance induces conserved angular momenta along these symmetry directions, which restricts the chaotic exploration in parameter space required for ECD to effectively proceed in descent directions. SGDM is similarly affected, though its dynamics has one fewer conservation law. Motivated by this analysis, we introduce a batchwise symmetry-breaking protocol that injects fixed, untrained query and value biases, denoted b_{Q} and b_{V} , thereby introducing a preferred direction in the otherwise degenerate rotational subspace. Beyond improving optimization, the query bias b_{Q} enables a novel, interpretable mechanism for shaping model behavior: the model can amplify or suppress classes of tokens by learning strong alignment or anti-alignment between their key vectors k=W_{K}x and the mean bias \mathbb{E}[b_{Q}] , yielding an exponential modulation of attention weights proportional to e^{k\cdot b_{Q}} . Since widely used open-source models such as Gemma, Llama, DeepSeek, and Qwen do not include bias terms in their attention heads, this intervention in pretraining can be used without sacrificing any standard training parameters. Our empirical analysis shows that this explicit symmetry breaking substantially improves memory-efficient optimization methods, particularly ECD. Across validation loss and downstream logical reasoning tasks, ECD trained with our protocol becomes competitive with SOAP at the GPT-2 (124M) scale. In our experiments, ECD exhibits consistent use of the query bias to amplify semantically meaningful token classes, a property which for all optimizers is predictive of improved downstream reasoning performance. Overall, our results highlight how a careful analysis of architectural symmetries and learning dynamics can reveal simple, principled modifications that improve both efficiency and interpretability. We leave to future work a broader evaluation across model scales and optimization schemes, including the integration of b_{Q},b_{V} symmetry breaking and ECD into the benchmarking protocol of Wen et al. 2025 . Our contributions. We summarize our main theoretical and empirical contributions as follows: 1. We provide a Hamiltonian explanation for why ECD fails to train transformer models out of the box, showing that rotational symmetries in attention induce conserved quantities that can obstruct descent ( Section 3 ). 2. We propose a simple, symmetry-breaking modification that lifts this obstruction while preserving the memory efficiency and structural simplicity of ECD ( Section 4 ). 3. We illustrate empirically that symmetry-broken ECD matches—and in some cases exceeds—the performance of adaptive optimizers such as Adam and SOAP on GPT-2–scale transformer models ( Section 5.1 ). 4. We show that the proposed symmetry-breaking mechanism is interpretable, enabling direct analysis of how it alters learning outcomes within attention layers to amplify or suppress token classes ( Section 5.2 ). 1 1 1 Code available at https://github.com/evasilverstein/Symmetry-breaking-attention-bias 2 Related Work Our work engages with several lines of research, including the optimization of transformer models and energy-based, physics-inspired optimization methods. Additionally, our work provides a potential window into interpretability ( Sharkey et al. 2025 ) through a novel alignment mechanism. Optimization of transformer models. Most modern deep learning training pipelines rely on adaptive variants of stochastic gradient descent with momentum (SGDM) ( Robbins & Monro 1951 ) , such as Adam ( Kingma & Ba 2015 ) and AdamW ( Loshchilov & Hutter 2019 ) . Several other methods have significantly influenced modern training practice, including Polyak’s heavy-ball momentum ( Polyak 1964 ) , RMSprop ( Tieleman & Hinton 2012 ) , and the memory-efficient Adafactor ( Shazeer & Stern 2018 ) . To improve convergence through structured preconditioning, Shampoo ( Gupta et al. 2018 ) introduced second-order, structure-aware updates, which have since been optimized for large-scale language modeling via SOAP ( Vyas et al. 2024 ) . More recently, scalable alternatives such as Sophia ( Liu et al. 2024 ) and Muon ( Liu et al. 2025 ) have been proposed to balance adaptivity, stability, and memory efficiency in large-model training. A consistent empirical observation across this literature is that adaptive optimizers tend to outperform simpler, memory-light methods such as SGDM when training transformer architectures, particularly at scale. Our work aims to close this gap by

Read the complete text