Transformers Trained via Gradient Descent Can Provably Learn a Class of Teacher Models
Chenyang Zhang, Qingyue Zhao, Quanquan Gu, Yuan Cao · 2026
Read Transformers Trained via Gradient Descent Can Provably Learn a Class of Teacher Models by Chenyang Zhang, Qingyue Zhao, Quanquan Gu, Yuan Cao (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-22801.txt.
Description
Zhang and co-authors prove that transformers trained by gradient descent can learn a class of teacher models. The teachers include convolution layers, graph convolution layers, and classic statistical models. A one-layer transformer with simple position-only attention is the student. The authors prove it can recover all teacher parameter blocks and reach optimal population loss. The trained student also generalizes to out-of-distribution data under mild conditions. A shared bilinear structure across tasks makes the unified proof work.
Catalog record
| CREATOR | Chenyang Zhang, Qingyue Zhao, Quanquan Gu, Yuan Cao |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 235963 |
| LICENSE | http://creativecommons.org/licenses/by-sa/4.0/ |
| RIGHTS | CC BY SA |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.22801