AVOC: Enhancing Hour-Level Audio-Video Understanding in Omni-Modal LLMs via Retrieval-Inspired Token Compression

Yijing Chen, Wenhui Tan, Xiaoyi Yu, Yuyue Wang, Xin Cheng, Kaisi Guan, Hao Jiang Xiangyang Li Guojie Zhu Ruihua Song · 2026

Read AVOC: Enhancing Hour-Level Audio-Video Understanding in Omni-Modal LLMs via Retrieval-Inspired Token Compression by Yijing Chen, Wenhui Tan, Xiaoyi Yu, Yuyue Wang, Xin Cheng, Kaisi Guan, Hao Jiang Xiangyang Li Guojie Zhu Ruihua Song (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-24286.txt.

Description

Chen and colleagues help omni-modal language models watch hour-long audio-video content without drowning in tokens. Long recordings overflow context windows and repeat themselves endlessly, so most tokens carry nothing new. Their 2026 framework, AVOC, inserts a learnable compression module between the audio-video encoders and the language model, treating compression as a retrieval problem under a fixed budget.

Three classical information-retrieval criteria guide which tokens survive: relevance to the user's query, general importance, and diversity among the chosen set. Each criterion becomes a tailored mechanism inside one unified pipeline, so the retained subset best supports answering the actual question rather than summarising everything equally. On long-form audio-video benchmarks AVOC reaches state-of-the-art performance, beating the second-best model by several points of average accuracy on OmniVideoBench and a companion benchmark. Chen's retrieval framing turns an engineering squeeze into a principled selection problem.

Catalog record

CREATORYijing Chen, Wenhui Tan, Xiaoyi Yu, Yuyue Wang, Xin Cheng, Kaisi Guan, Hao Jiang Xiangyang Li Guojie Zhu Ruihua Song
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE40 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.24286

Browse the full library · About the Tower of Babel