TQL: Scaling Q-Functions with Transformers by Preventing Attention Collapse

Perry Dong, Kuo-Han Hung, Alexander Swerdlow, Dorsa Sadigh, Chelsea Finn · 2026

Read TQL: Scaling Q-Functions with Transformers by Preventing Attention Collapse by Perry Dong, Kuo-Han Hung, Alexander Swerdlow, Dorsa Sadigh, Chelsea Finn (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-01439.txt.

Description

Perry Dong and co-authors ask why transformers fail to scale for value functions. Scale has driven recent gains, yet reinforcement learning still leans on small value functions, and naively enlarging them with transformers often destabilizes learning. Empirical digging pinpoints the failure: attention scores collapse as capacity grows. Controlling attention-score entropy stops that collapse and steadies training for larger models. The team proposes Transformer Q-Learning, or TQL, to unlock transformer scaling. Scaling from smallest to largest networks, their approach gains up to 43%, while earlier methods degrade.

Catalog record

CREATORPerry Dong, Kuo-Han Hung, Alexander Swerdlow, Dorsa Sadigh, Chelsea Finn
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36977
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602.01439

Browse the full library · About the Tower of Babel