TQL: Scaling Q-Functions with Transformers by Preventing Attention Collapse
Perry Dong, Kuo-Han Hung, Alexander Swerdlow, Dorsa Sadigh, Chelsea Finn · 2026
Read TQL: Scaling Q-Functions with Transformers by Preventing Attention Collapse by Perry Dong, Kuo-Han Hung, Alexander Swerdlow, Dorsa Sadigh, Chelsea Finn (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-01439.txt.
Description
Perry Dong and co-authors ask why transformers fail to scale for value functions. Scale has driven recent gains, yet reinforcement learning still leans on small value functions, and naively enlarging them with transformers often destabilizes learning. Empirical digging pinpoints the failure: attention scores collapse as capacity grows. Controlling attention-score entropy stops that collapse and steadies training for larger models. The team proposes Transformer Q-Learning, or TQL, to unlock transformer scaling. Scaling from smallest to largest networks, their approach gains up to 43%, while earlier methods degrade.
Catalog record
| CREATOR | Perry Dong, Kuo-Han Hung, Alexander Swerdlow, Dorsa Sadigh, Chelsea Finn |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36977 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602.01439