Cross-Modal Bottleneck Fusion for Noise Robust Audio-Visual Speech Recognition
Seaone Ok, Min Jun Choi, Eungbeom Kim, Seungu Han, Kyogu Lee · 2026
Read Cross-Modal Bottleneck Fusion for Noise Robust Audio-Visual Speech Recognition by Seaone Ok, Min Jun Choi, Eungbeom Kim, Seungu Han, Kyogu Lee (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-08293.txt.
Description
Speech recognition from audio alone degrades in noise, and lip movements can rescue it, but fusion must let audio reach visual cues without drowning clean-speech performance. Seaone Ok and four co-authors propose CoBRA, a cross-modal bottleneck framework that routes exchange through a compact set of learnable tokens.
Regulating information flow through the bottleneck lets the audio stream reliably access essential visual cues even when noise is adverse or out of domain. Despite limited training data, the model beats comparable baselines and rivals large-scale systems, with ablations showing fusion depth matters most. Audio-visual speech researchers gain an efficient, noise-adaptive fusion design that works where data are scarce and conditions are harsh.
Catalog record
| CREATOR | Seaone Ok, Min Jun Choi, Eungbeom Kim, Seungu Han, Kyogu Lee |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 22 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-08293