Transformers Efficiently Perform In-Context Logistic Regression via Normalized Gradient Descent
Chenyang Zhang, Yuan Cao · 2026
Read Transformers Efficiently Perform In-Context Logistic Regression via Normalized Gradient Descent by Chenyang Zhang, Yuan Cao (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-06609.txt.
Description
Chenyang Zhang and Yuan Cao show that Transformers can perform logistic regression inside their context window by running normalized gradient descent. In-context learning is often explained by transformers secretly executing algorithms on the prompt. The authors construct multi-layer transformers with softmax attention where each layer performs exactly one step of normalized gradient descent on a logistic loss defined by the context examples.
The constructed model is also reachable by training: one self-attention layer is trained to imitate a single gradient step, then looped to form the full model. Convergence guarantees for the layer training and out-of-distribution guarantees for the looped model are provided. The result sharpens the theory of how attention implements learning algorithms.
Catalog record
| CREATOR | Chenyang Zhang, Yuan Cao |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 330 KB |
| LICENSE | http://creativecommons.org/licenses/by-sa/4.0/ |
| RIGHTS | CC BY SA |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.06609