AdaPonderLM: Gated Pondering Language Models with Token-Wise Adaptive Depth

Shixiang Song 1,5∗ , He Li 1∗ , Zitong Wang 1,3 , Boyi Zeng 1 , Feichen Song 1 , Yixuan Wang 1,5 , Zhiqin John Xu 4 , Ziwei He 5 , Zhouhan Lin 1,2,5‡ 1 LUMIA Lab, School of Artificial Intelligence, Shanghai Jiao Tong University, 2 Shanghai AI Laboratory, 3 Sun Yat-sen University, 4 Shanghai Jiao Tong University, 5 Shanghai Innovation Institute · 2026

Read AdaPonderLM: Gated Pondering Language Models with Token-Wise Adaptive Depth by Shixiang Song 1,5∗ , He Li 1∗ , Zitong Wang 1,3 , Boyi Zeng 1 , Feichen Song 1 , Yixuan Wang 1,5 , Zhiqin John Xu 4 , Ziwei He 5 , Zhouhan Lin 1,2,5‡ 1 LUMIA Lab, School of Artificial Intelligence, Shanghai Jiao Tong University, 2 Shanghai AI Laboratory, 3 Sun Yat-sen University, 4 Shanghai Jiao Tong University, 5 Shanghai Innovation Institute (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-01914.txt.

Description

Shixiang Song, He Li, Zitong Wang, and colleagues let each token in a language model choose how much compute it needs. Their 2026 work builds on recurrent models that loop a fixed number of times per token. Fixed loops waste effort on easy tokens and starve hard ones of extra passes.

Their model, AdaPonderLM, learns token-wise early exit during pre-training with no hand-set pruning ratios. Gates that grow stricter by iteration decide when each token stops looping, under a monotonic halting mask. Cached key and value states for halted tokens are reused, which keeps training and testing consistent. Across several backbone sizes, inference compute drops by about a tenth at equal quality. Gate analysis shows hard tokens, with high loss, indeed earn more passes than easy ones.

Catalog record

CREATORShixiang Song 1,5∗ , He Li 1∗ , Zitong Wang 1,3 , Boyi Zeng 1 , Feichen Song 1 , Yixuan Wang 1,5 , Zhiqin John Xu 4 , Ziwei He 5 , Zhouhan Lin 1,2,5‡ 1 LUMIA Lab, School of Artificial Intelligence, Shanghai Jiao Tong University, 2 Shanghai AI Laboratory, 3 Sun Yat-sen University, 4 Shanghai Jiao Tong University, 5 Shanghai Innovation Institute
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE34223
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.01914

Browse the full library · About the Tower of Babel