Attention with Trained Embeddings Provably Selects Important Tokens

Diyuan Wu, Aleksandr Shevchenko, Samet Oymak, Marco Mondelli · 2025

Read Attention with Trained Embeddings Provably Selects Important Tokens by Diyuan Wu, Aleksandr Shevchenko, Samet Oymak, Marco Mondelli (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-17282.txt.

Introduction

Inside a language model, each word is stored as a list of numbers called an embedding, and attention decides which words matter. Diyuan Wu, Aleksandr Shevchenko, Samet Oymak, and Marco Mondelli prove what training does to those embeddings in a 2025 paper. Their test bed is a one-layer attention model built for yes-no classification.

After a single step of gradient training, the embeddings already lean toward the output vector in proportion to how often each token appears in the data. Training the summary token to convergence then makes the softmax pick out the tokens that predict the label, with the widest margin the setup allows. Tests on movie and business review datasets showed behavior close to the theory. The results give a rare proof-backed picture of why attention lands on the words it does.

Catalog record

CREATORDiyuan Wu, Aleksandr Shevchenko, Samet Oymak, Marco Mondelli
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE91 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-17282

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Attention with Trained Embeddings Provably Selects Important Tokens Diyuan Wu, Aleksandr Shevchenko, Samet Oymak, Marco Mondelli Abstract Token embeddings play a crucial role in language modeling but, despite this practical relevance, their theoretical understanding remains limited. Our paper addresses the gap by characterizing the structure of embeddings obtained via gradient descent. Specifically, we consider a one-layer softmax attention model with a linear head for binary classification, i.e., Softmax ​ ( p ⊤ ​ 𝑬 X ⊤ ) ​ 𝑬 X ​ v = ∑ i = 1 T exp ⁡ ( p ⊤ ​ E x i ) ​ E x i ⊤ ​ v ∑ j = 1 T exp ⁡ ( p ⊤ ​ E x j ) \texttt{Softmax}(p^{\top}{\bm{E}}_{X}^{\top}){\bm{E}}_{X}v=\frac{\sum_{i=1}^{T}\exp(p^\top E_{x_i})E_{x_{i}}^{\top}v}{\sum_{j=1}^{T}\exp(p^\top E_{x_{j}})} , where 𝑬 X = [ E x 1 , … , E x T ] ⊤ {\bm{E}}_{X}=[E_{x_{1}},\dots,E_{x_{T}}]^{\top} contains the embeddings of the input sequence, p p is the embedding of the ⟨ cls ⟩ \mathrm{\langle cls\rangle} token and v v the output vector. First, we show that, already after a single step of gradient training with the logistic loss, the embeddings 𝑬 X {\bm{E}}_{X} capture the importance of tokens in the dataset by aligning with the output vector v v proportionally to the frequency with which the corresponding tokens appear in the dataset. Then, after training p p via gradient flow until convergence, the softmax selects the important tokens in the sentence (i.e., those that are predictive of the label), and the resulting ⟨ cls ⟩ \mathrm{\langle cls\rangle} embedding maximizes the margin for such a selection. Experiments on real-world datasets (IMDB, Yelp) exhibit a phenomenology close to that unveiled by our theory. 2 2 footnotetext: ETH Zürich. Email: [email protected] 3 3 footnotetext: University of Michigan. Email: [email protected] 1 Introduction The introduction of the attention mechanism [ 5 , 42 ] marked a paradigm shift in the design of frontier machine learning models, leading to significant advances such as ChatGPT [ 1 ] , Claude [ 4 ] , AlphaFold [ 15 ] , CLIP [ 30 ] and Dall-E [ 31 ] . This success prompted a surge of interest in understanding the structure and function of attention layers, with their optimization dynamics and inductive biases being object of extensive theoretical research [ 2 , 8 , 11 , 24 , 37 , 40 ] (see also Section 2 ). Embeddings are a crucial component of the attention mechanism [ 45 ] , especially for downstream adaptation [ 13 , 18 , 19 ] with some works [ 21 , 45 ] specifically highlighting their importance. However, despite the importance of learning embeddings, the existing analyses of transformer-like architectures either ignore the properties of embeddings by resorting to orthogonal structures [ 44 ] , or omit embeddings completely by considering unprocessed inputs [ 38 ] . Our paper fills this gap by studying directly the embedding training dynamics. Specifically, we aim to provide theoretical insight to the following questions: What is the structure learnt by the embeddings during gradient descent training? How is this structure related to the statistical properties of the data? In Figure 1 , we investigate these questions by analyzing the embeddings of a two-layer transformer trained on a sentiment analysis task on IMDB and Yelp reviews. The plots reveal a remarkable simplicity in the structure of the learned embeddings, which capture the frequency of appearance of tokens in the dataset. Specifically, the predictive mechanism (overlap with the regression coefficient v ) favors the tokens which appear more frequently in the corresponding positive/negative context. A similar pattern emerges at the selection stage of the attention mechanism (overlap with the \mathrm{\langle cls\rangle} embedding p ), i.e., more frequent tokens have a higher attention score. Figure 1: Dot-product of token embeddings with \mathrm{\langle cls\rangle} embedding p (left) and regression coefficients v (right), as a function of the token-wise difference in posterior probabilities, for IMDB (top row) and Yelp (bottom row) datasets. We consider the two-layer attention model in ( 17 ) with all parameters trained until convergence. For the theoretical study of this emergent structure, we focus on a one-layer softmax attention model. Namely, for an input sequence X=[x_{1},\cdots,x_{T}] , the output of the model is given by f(X;p,{\bm{E}})=\texttt{Softmax}(p^{\top}{\bm{E}}_{X}^{\top}){\bm{E}}_{X}v=\frac{\sum_{i=1}^{T}\exp(p^\top E_{x_i})E_{x_{i}}^{\top}v}{\sum_{j=1}^{T}\exp(p^\top E_{x_{j}})}, (1) where {\bm{E}}_{X}=[E_{x_{1}},\dots,E_{x_{T}}]^{\top} contains the embeddings of the input X , p is the embedding of the \mathrm{\langle cls\rangle} token and v is the final regression vector. Our main results are summarized below: • We show that, already after a single step of gradient training with the standard logistic loss, the embeddings {\bm{E}}_{X} capture the importance of tokens in the dataset by aligning with the output vector v proportionally to the corresponding empirical frequencies (Lemma 4.1 ). • In a setting where each sequence contains a single important token, the \mathrm{\langle cls\rangle} embedding obtained from gradient flow must select all important tokens. We further characterize all the possible directions that the \mathrm{\langle cls\rangle} embedding may converge to, which are the max-margin solutions associated to feasible token selections (Theorem 4.3 ). • While in general the \mathrm{\langle cls\rangle} embedding may select irrelevant tokens, we identify sufficient conditions leading to the selection only of important tokens (Lemmas 4.4 and 4.5 ). 2 Related work Implicit bias, margin maximization, attention. The implicit bias literature has been instrumental in understanding the behavior of neural networks or overparameterized models optimized by gradient methods [ 7 , 3 , 27 ] . A key phenomenon is that gradient descent on separable data with logistic loss directionally converges to the max-margin separator [ 33 , 16 ] . More recently, a series of works [ 36 , 25 , 17 , 41 , 20 , 32 , 37 , 35 ] has established an equivalence between the optimization geometry of self-attention and a hard-margin SVM problem selecting a subset of tokens via linear constraints on the outer-products of token pairs. Compared to these works that mostly focus on the training of single-layer attention weights, we point out two differences. First, we study the role of embeddings and their joint training with the \mathrm{\langle cls\rangle} token. Second, under our data model, we establish benign properties of the solution reached at convergence (which may not hold for arbitrary datasets [ 37 ] ). Theory of attention. A line of work [ 24 , 23 , 26 ] has explored whether attention-based architectures can extract causal structure from Markovian inputs. The mechanics of next-token prediction when training a single self-attention layer is characterized in [ 20 ] . Towards understanding how to utilize structural properties of the data, the behavior of transformers on sparse token selection tasks is considered in [ 34 , 43 ] . The study [ 14 ] provides a theoretical justification to the tendency of modern language models to generate repetitive text by showing that the underlying self-attention mechanism collapses into sampling only a limited subset of tokens. This stands in contrast to the slightly different setup of [ 39 ] where the transformer model does not degrade to a “winner-takes-all” strategy. The works [ 11 , 12 , 10 ] take a mean-field view to analyze the clustering behavior in transformer representations that emerges after successive applications of the

Read the complete text