Simply Stabilizing the Loop via Fully Looped Transformer

Rao Fu, Zixuan Yang, Jiankun Zhang, Jing Ma, Hechang Chen, Yu Li, Yi Chang, [0.5em] Hong Kong Baptist University, Jilin University · 2026

Read Simply Stabilizing the Loop via Fully Looped Transformer by Rao Fu, Zixuan Yang, Jiankun Zhang, Jing Ma, Hechang Chen, Yu Li, Yi Chang, [0.5em] Hong Kong Baptist University, Jilin University (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-18797.txt.

Description

Rao Fu and six colleagues stabilize the Looped Transformer, which reuses the same blocks to gain power without adding weights. Reuse lets users trade compute for quality at test time, but training grows unstable as loop counts rise. The authors trace the trouble to gradient swings and residual growth. Their Fully Looped Transformer spreads signals across all layers and reuses attention to damp swings, with no new weights. Training then stays stable at higher loop counts.

Catalog record

CREATORRao Fu, Zixuan Yang, Jiankun Zhang, Jing Ma, Hechang Chen, Yu Li, Yi Chang, [0.5em] Hong Kong Baptist University, Jilin University
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE53407
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.18797

Browse the full library · About the Tower of Babel