Why Your Tokenizer Fails in Information Fusion: A Timing-Aware Pre-Quantization Fusion for Video-Enhanced Audio Tokenization

Xiangyu Zhang, Benjamin John Southwell, Siqi Pan, Xinlei Niu, Beena Ahmed, Julien Epps · 2026

Read Why Your Tokenizer Fails in Information Fusion: A Timing-Aware Pre-Quantization Fusion for Video-Enhanced Audio Tokenization by Xiangyu Zhang, Benjamin John Southwell, Siqi Pan, Xinlei Niu, Beena Ahmed, Julien Epps (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-12145.txt.

Description

Zhang and five colleagues improve audio tokenizers by adding video, a step that usually damages sound reconstruction. They show fusion placement matters most, contrastive learning fails for discrete tokens, and fusing along time rather than feature dimensions works best. Their timing-aware pre-quantization fusion preserves high-fidelity reconstruction while beating audio-only and standard multimodal baselines on understanding tasks.

Catalog record

CREATORXiangyu Zhang, Benjamin John Southwell, Siqi Pan, Xinlei Niu, Beena Ahmed, Julien Epps
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE52043
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.12145

Browse the full library · About the Tower of Babel