Learning Spectral Methods by Transformers

Yihan He, Yuan Cao, Hong-Yu Chen, Dennis Wu, Jianqing Fan, and, Han Liu · 2025

Read Learning Spectral Methods by Transformers by Yihan He, Yuan Cao, Hong-Yu Chen, Dennis Wu, Jianqing Fan, and, Han Liu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-01312.txt.

Description

Yihan He and co-authors ask whether a transformer model can learn a classic set of data methods on its own, in a 2025 paper. Transformers are the building blocks of large language models. Here the focus is on learning without labels, where the model must find structure in new data after training on many past cases.

The authors prove that a trained transformer with enough layers can learn spectral methods, which work from the main axes of spread in data. They show this with a test case that sorts data drawn from two bell-shaped groups. Tests on both made-up and real data back the claim for tasks such as finding main axes and grouping points. The result links the steps inside a transformer to the steps of known iterative methods.

Catalog record

CREATORYihan He, Yuan Cao, Hong-Yu Chen, Dennis Wu, Jianqing Fan, and, Han Liu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE135 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-01312

Browse the full library · About the Tower of Babel