CCStereo: Audio-Visual Contextual and Contrastive Learning for Binaural Audio Generation

Yuanhong Chen, Kazuki Shimada, Christian Simon, Yukara Ikemiya, Takashi Shibuya, Yuki Mitsufuji · 2025

Read CCStereo: Audio-Visual Contextual and Contrastive Learning for Binaural Audio Generation by Yuanhong Chen, Kazuki Shimada, Christian Simon, Yukara Ikemiya, Takashi Shibuya, Yuki Mitsufuji (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-02786.txt.

Description

Yuanhong Chen and co-authors build CCStereo, a model that turns mono sound into binaural audio guided by video. True stereo hearing cues place sounds in space, which flat recordings lose. The model reads the room and the scene from images while keeping fine spatial grain.

Their design adds a normalization layer that bends sound features to match visual context. A contrastive step sharpens spatial feel by mining shuffled image features as negative cases. Cheap test-time tricks on the video squeeze out more accuracy. Earlier models tended to overfit single rooms and blur placement. CCStereo tops the FAIR-Play and MUSIC-Stereo benchmarks on generation accuracy. Richer stereo from plain video could lift virtual reality, film tools, and assistive listening aids.

Catalog record

CREATORYuanhong Chen, Kazuki Shimada, Christian Simon, Yukara Ikemiya, Takashi Shibuya, Yuki Mitsufuji
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE51 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-02786

Browse the full library · About the Tower of Babel