Scaling Audio–Text Retrieval with Multimodal Large Language Models

Jilan Xu Carl Thomé Danijela Horak Weidi Xie Andrew Zisserman, [0.6em] Visual Geometry Group, University of Oxford Epidemic Sound · 2026

Read Scaling Audio–Text Retrieval with Multimodal Large Language Models by Jilan Xu Carl Thomé Danijela Horak Weidi Xie Andrew Zisserman, [0.6em] Visual Geometry Group, University of Oxford Epidemic Sound (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-18010.txt.

Description

Jilan Xu and colleagues re-purpose multimodal large language models for audio-text retrieval, the task of bridging acoustic signals and natural language. Contrastive dual-encoder systems like CLAP show promise, but small-scale encoders cap their power. Their text encoders especially struggle with complex queries that need reasoning or world knowledge.

The authors propose AuroLA, a contrastive language-audio pretrained model that uses a multimodal large language model as a single backbone for retrieval. A scalable data pipeline curates diverse audio from many sources and generates multi-granular captions, from long descriptions to structured tags, through automated annotation. The model is adapted by prompting it to summarize the audio or text input, with the hidden state of a special token serving as the embedding. In experiments, AuroLA consistently beat leading dual-encoder models, including the recent PE-AV, supporting the unified-backbone approach.

Catalog record

CREATORJilan Xu Carl Thomé Danijela Horak Weidi Xie Andrew Zisserman, [0.6em] Visual Geometry Group, University of Oxford Epidemic Sound
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE62 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-18010

Browse the full library · About the Tower of Babel