Anti Mode-Collapse in Mean-Field Transformer via Auxiliary Variables
Masaaki Imaizumi 1,2,3 , Masanori Koyama 1 , Noboru Isobe 2 , Kohei Hayashi 1 · 2026
Read Anti Mode-Collapse in Mean-Field Transformer via Auxiliary Variables by Masaaki Imaizumi 1,2,3 , Masanori Koyama 1 , Noboru Isobe 2 , Kohei Hayashi 1 (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-30229.txt.
Introduction
Masaaki Imaizumi, Masanori Koyama, Noboru Isobe and Kohei Hayashi use a mean-field transformer to study how auxiliary factors such as positional encoding prevent mode collapse in attention. The simple model suggests mode collapse during runs. They show helpers counter mode collapse: the top-energy spread does not collapse to one point, but is pushed forward from the helper distribution, dodging Dirac collapse. Cases are positional encoding and prompt insertion as a helper setup. Both are universal in the limit.
Catalog record
| CREATOR | Masaaki Imaizumi 1,2,3 , Masanori Koyama 1 , Noboru Isobe 2 , Kohei Hayashi 1 |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 109343 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.30229
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Anti Mode-Collapse in Mean-Field Transformer via Auxiliary Variables Masaaki Imaizumi 1,2,3 , Masanori Koyama 1 , Noboru Isobe 2 , Kohei Hayashi 1 Abstract . We use a mean-field-based transformer model to theoretically investigate how auxiliary variables, such as positional encoding, prevent mode collapse of self-attention mechanisms. The use of mean-field transformers to analyze the properties of self-attention mechanisms has garnered significant attention in recent years due to their ability to comprehensively analyze token interactions. However, analysis of this simple model suggests that mode collapse, where token distributions degenerate to a single point, occurs during long inferences (i.e., many layers), indicating a discrepancy with reality. This study investigates this mean-field transformer model and demonstrates that the introduction of auxiliary variables, such as positional encoding, acts as a counterforce against theoretical mode collapse. Specifically, we show that in the theoretical scheme, the energy-maximizing distribution does not degenerate to a single point; instead, it is characterized by a pushforward of the auxiliary variable distribution, thereby avoiding concentration in the Dirac measure. Our main examples are the positional encoding and the fixed prompt insertion treated as a parallel auxiliary-variable mechanism. Furthermore, we demonstrate that positional encoding and prompt insertion possess universality of representation in the limit, meaning that the limit distribution of inference can exactly represent a wide class of distributions. We also analyze several key properties of positional encoding and metastability, and validate our theoretical results through mathematical experiments. 1. Introduction Transformers are neural networks built around self-attention mechanisms VSP+ 17 . They have played a central role not only in natural language processing but also across machine learning more broadly. Also, recent work on looped or depth-recurrent Transformers, which performs inference by repeatedly iterating through layers that share the same weights, has attracted attention due to its high performance FDRL 24 ; XS 24 ; ACLY 25 ; Che 26 ; GMJ+ 25 . Motivated by their empirical success, a growing theoretical literature seeks to clarify the mathematical principles behind the evolution of token representations YBR+ 20 ; CLJ 20 ; BHK 24 ; CBKZ 24 ; TN 24 . In particular, a shared-block regime is investigated from aspects of length generalization, expressive power, recursive latent reasoning, and compositional generalization FDRL 24 ; ACLY 25 ; Che 26 . A fruitful viewpoint treats repeated self-attention updates as interacting-particle dynamics and studies their mean-field limits DGCC 21 ; GLPR 23 ; GLPR 25 ; Rig 25 . In particular, GLPR 25 ; GLPR 23 formulate simplified self-attention models as interacting particle systems on the sphere and analyze long-time clustering and limiting configurations. Moreover, GKPR 24 shows that these dynamics can exhibit metastability: although they ultimately converge to a single cluster, they may remain near multi-cluster states for very long time scales. This line of work provides a mathematically precise route from iterative self-attention updates to gradient-driven dynamics governed by an energy functional. A recurring conclusion of this framework is degeneracy or mode-collapse of token distributions. In the variational analysis above in regard to the dynamics of distributions of tokens, long-time limits and maximizing configurations often collapse to Dirac measures or single clusters GLPR 25 ; GLPR 23 ; GKPR 24 . In other words, according to these analyses, long inferences performed by (loop-based) Transformers ultimately lead to a collapse to a single point. While there are differing views on this issue based on experience, oversmoothing is not inevitable DBK 24 , and masking together with layer normalization can prevent simple rank-one collapse WAW+ 24 , it has yet to be settled. Based on this situation, we pose the following research questions: What are effective methods for preventing mode collapse in mean-field-based transformer models? Can we provide a theoretical basis for these methods? 1.1. Contributions Figure 1. Distributions \mu_{\tau} of tokens x on \mathbb{S}^{d-1} via self-attention and maximizing energy. In the standard USA model (left), an energy maximizer \mu^{*} is a Dirac measure. In the USA model with auxiliary variables \xi in \mathcal{A} (right), the addition of an auxiliary variable space prevents \mu^{*} from degenerating. The energy is maximized conditionally for each value of \xi internally, and the output distribution can be regarded as its marginalization. This paper endeavors to fill the aforementioned gap by extending the mean-field theoretical framework such as GLPR 23 to include the variables that are used in actual practice but are often omitted from the analysis. In particular, in addition to the dynamics of token variables that has been considered in depth, we include Positional encoding and prefix tokens inserted at inference time LL 21 ; LARC 21 as tangible probabilistic auxiliary variables, and consider the joint law of the token and auxiliary variables with a fixed auxiliary marginal. More generally, we develop the framework of what we call Un normalized S elf- A ttention model with A uxiliary V ariables ( USA-AV ) and show that, with sufficient regularity, the auxiliary variables prevent the token distribution from collapsing. Figure 1 outlines the result shown by our theory. The main contributions of this paper are as follows. • A fixed-auxiliary variational framework for anti-collapse. We formulate self-attention with auxiliary variables (e.g., positional encodings and prompt prefixes) in the USA-AV model, which describes the dynamics of the token variable under a fixed auxiliary marginal through joint laws. • Conditional Dirac structure with marginalized noncollapse. Under a fixed auxiliary marginal, we show that maximizers may still be Dirac in the conditional content variable at each auxiliary value, while the marginalized content distribution remains non-Dirac. This identifies the precise sense in which auxiliary variables prevent collapse: not by destroying conditional Dirac structure, but by allowing the Dirac location to move with the auxiliary variable. • Positional encoding yields orbit-wise exact realization of a target law. For positional variables, we show that RoPE-type kernels already generate spread maximizers, and that arbitrary measurable phase fields go further: once a rotation orbit is fixed, every probability law on that orbit can arise exactly as a maximizing marginalized output law. • Prefix tokens upgrade this to gauge-level exact realization of a target law. For prefix tokens, we prove that any \eta -pushforward law generated by a measurable map into the sphere can be realized exactly as a maximizing marginalized output law through an orthogonal gauge; when the prompt space is standard Borel and the prompt law is nonatomic, this yields exact realization of arbitrary target laws on \mathbb{S}^{d-1} . The results above suggest that positional encodings and prefix tokens function not merely as side information, but as genuine anti mode-collapse mechanisms inside the idealized shared-weight self-attention model analyzed here. Most importantly, these anti mode-collapse mechanisms can be designed with appropriate kernels to produce a target distribution through the marginalization of the auxiliary variables. Throughout, the phrase looped transformer-type self-attention dynamics is meant to emphasize this scope restriction: the theorems concern a shared-weight surrogate, not an arbitrary full Transformer architecture. In Sections 3 and 4 we refer informally to the last two principles as orbit-wise and gauge-level universality , respectively, but the actual mathematical statements are exact realization results for maximizing marginals rather than universal approximation statements for sequence-to-sequence maps. 1.2. Notation We use \|\cdot\| for norms, \langle\cdot,\cdot\rangle for Euclidean inner products, \delta for a Dirac