MUSE: Resolving Manifold Misalignment in Visual Tokenization via Topological Orthogonality
Panqi Yang, Haodong Jing, Jiahao Chao, Tingyan Xiang, Li Lin, Yao Hu, Yang Luo, Yongqiang Ma · 2026
Read MUSE: Resolving Manifold Misalignment in Visual Tokenization via Topological Orthogonality by Panqi Yang, Haodong Jing, Jiahao Chao, Tingyan Xiang, Li Lin, Yao Hu, Yang Luo, Yongqiang Ma (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-05646.txt.
Description
Turning images into tokens for a unified model forces a trade-off: tokens should rebuild pixels faithfully yet also carry abstract meaning. Panqi Yang and seven co-authors trace that conflict to manifold misalignment. Joint training makes reconstruction and perception gradients fight, a zero-sum game inside the transformer.
Their framework, MUSE, resolves it through topological orthogonality, treating structure as a bridge between the two goals. Structural gradients reshape attention patterns while semantic gradients update feature values. The interference becomes mutual reinforcement. In experiments MUSE reached top generation quality while its tokens stayed useful for understanding tasks.
Catalog record
| CREATOR | Panqi Yang, Haodong Jing, Jiahao Chao, Tingyan Xiang, Li Lin, Yao Hu, Yang Luo, Yongqiang Ma |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 56 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.05646