On the Equivalence Between Auto-Regressive Next Token Prediction and Full-Item-Vocabulary Maximum Likelihood Estimation in Generative Recommendation–A Short Note DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Information systems Recommender systems

Yusheng Huang, Shuang Yang, Zhaojie Liu, Han Li · 2026

Read On the Equivalence Between Auto-Regressive Next Token Prediction and Full-Item-Vocabulary Maximum Likelihood Estimation in Generative Recommendation–A Short Note DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Information systems Recommender systems by Yusheng Huang, Shuang Yang, Zhaojie Liu, Han Li (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-15739.txt.

Description

Huang, Yang, Liu, and Li prove that k-token next-token guessing in generative recommendation matches full-item-vocab top-likelihood guessing exactly. This needs a one-to-one map from items to k-token strings. The match holds for both chain and side-by-side token splits, the two top shop schemes. The proof gives the first firm theory base for the top shop recipe and steers next tuning work.

Catalog record

CREATORYusheng Huang, Shuang Yang, Zhaojie Liu, Han Li
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE22930
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.15739

Browse the full library · About the Tower of Babel