Transformers Efficiently Perform In-Context Logistic Regression via Normalized Gradient Descent

Chenyang Zhang, Yuan Cao · 2026

Read Transformers Efficiently Perform In-Context Logistic Regression via Normalized Gradient Descent by Chenyang Zhang, Yuan Cao (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-06609.txt.

Description

Chenyang Zhang and Yuan Cao show that Transformers can perform logistic regression inside their context window by running normalized gradient descent. In-context learning is often explained by transformers secretly executing algorithms on the prompt. The authors construct multi-layer transformers with softmax attention where each layer performs exactly one step of normalized gradient descent on a logistic loss defined by the context examples.

The constructed model is also reachable by training: one self-attention layer is trained to imitate a single gradient step, then looped to form the full model. Convergence guarantees for the layer training and out-of-distribution guarantees for the looped model are provided. The result sharpens the theory of how attention implements learning algorithms.

Catalog record

CREATORChenyang Zhang, Yuan Cao
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE330 KB
LICENSEhttp://creativecommons.org/licenses/by-sa/4.0/
RIGHTSCC BY SA
SOURCEarXiv

Source: https://arxiv.org/abs/2605.06609

Browse the full library · About the Tower of Babel