Learning Spectral Methods by Transformers
Yihan He, Yuan Cao, Hong-Yu Chen, Dennis Wu, Jianqing Fan, and, Han Liu · 2025
Read Learning Spectral Methods by Transformers by Yihan He, Yuan Cao, Hong-Yu Chen, Dennis Wu, Jianqing Fan, and, Han Liu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-01312.txt.
Description
Yihan He and co-authors ask whether a transformer model can learn a classic set of data methods on its own, in a 2025 paper. Transformers are the building blocks of large language models. Here the focus is on learning without labels, where the model must find structure in new data after training on many past cases.
The authors prove that a trained transformer with enough layers can learn spectral methods, which work from the main axes of spread in data. They show this with a test case that sorts data drawn from two bell-shaped groups. Tests on both made-up and real data back the claim for tasks such as finding main axes and grouping points. The result links the steps inside a transformer to the steps of known iterative methods.
Catalog record
| CREATOR | Yihan He, Yuan Cao, Hong-Yu Chen, Dennis Wu, Jianqing Fan, and, Han Liu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 135 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-01312