Energy-Based Transformers are Scalable Learners and Thinkers

Alexi Gladstone, Ganesh Nanduru, Md Mofijul Islam, Peixuan Han, Hyeonjeong Ha, Aman Chadha, Yilun Du, Heng Ji, Jundong Li, Tariq Iqbal, [0.6em] UVA · 2025

Read Energy-Based Transformers are Scalable Learners and Thinkers by Alexi Gladstone, Ganesh Nanduru, Md Mofijul Islam, Peixuan Han, Hyeonjeong Ha, Aman Chadha, Yilun Du, Heng Ji, Jundong Li, Tariq Iqbal, [0.6em] UVA (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-02092.txt.

Introduction

Alexi Gladstone and co-authors build transformers that learn by grading their own guesses. Their 2025 paper asks whether slow, checked thinking can grow from plain unsupervised learning alone. Most current tricks for extra thinking time work for one data type or need added training.

Their energy-based transformers score how well a candidate answer fits the input. Answers are then improved by sliding down that score until it settles. The models scaled faster than the standard design across data, size, and compute. Extra thinking at test time helped them more than it helped the baseline. They also beat diffusion models at image cleanup with fewer passes through the network during image tasks at test time.

Catalog record

CREATORAlexi Gladstone, Ganesh Nanduru, Md Mofijul Islam, Peixuan Han, Hyeonjeong Ha, Aman Chadha, Yilun Du, Heng Ji, Jundong Li, Tariq Iqbal, [0.6em] UVA
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE135 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-02092

Browse the full library · About the Tower of Babel