Energy-Based Transformers are Scalable Learners and Thinkers
Alexi Gladstone, Ganesh Nanduru, Md Mofijul Islam, Peixuan Han, Hyeonjeong Ha, Aman Chadha, Yilun Du, Heng Ji, Jundong Li, Tariq Iqbal, [0.6em] UVA · 2025
Read Energy-Based Transformers are Scalable Learners and Thinkers by Alexi Gladstone, Ganesh Nanduru, Md Mofijul Islam, Peixuan Han, Hyeonjeong Ha, Aman Chadha, Yilun Du, Heng Ji, Jundong Li, Tariq Iqbal, [0.6em] UVA (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-02092.txt.
Introduction
Alexi Gladstone and co-authors build transformers that learn by grading their own guesses. Their 2025 paper asks whether slow, checked thinking can grow from plain unsupervised learning alone. Most current tricks for extra thinking time work for one data type or need added training.
Their energy-based transformers score how well a candidate answer fits the input. Answers are then improved by sliding down that score until it settles. The models scaled faster than the standard design across data, size, and compute. Extra thinking at test time helped them more than it helped the baseline. They also beat diffusion models at image cleanup with fewer passes through the network during image tasks at test time.
Catalog record
| CREATOR | Alexi Gladstone, Ganesh Nanduru, Md Mofijul Islam, Peixuan Han, Hyeonjeong Ha, Aman Chadha, Yilun Du, Heng Ji, Jundong Li, Tariq Iqbal, [0.6em] UVA |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 135 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-02092