CAE-AV: Improving Audio-Visual Learning via Cross-modal Interactive Enrichment
Yunzuo Hu, Wen Li, Jing Zhang 1 · 2026
Read CAE-AV: Improving Audio-Visual Learning via Cross-modal Interactive Enrichment by Yunzuo Hu, Wen Li, Jing Zhang 1 (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-08309.txt.
Description
Sound and sight do not always line up in video: a voice can come from off screen, or background noise can drown out the action. Yunzuo Hu and two co-authors build a system called CAE-AV that helps a learning model match what it hears with what it sees, even when the two streams disagree. Their 2026 arXiv paper targets the wobble that such mismatch causes in training.
Two parts do the work. One weighs each frame by how well sound and picture agree, so the model leans on the frames that matter most. The other uses written captions to steer attention toward the right places and moments in the clip. With the main backbones frozen, the system posts the best reported scores on four audio-visual test sets. Readers get a clear look at how caption cues and agreement checks can steady video understanding.
Catalog record
| CREATOR | Yunzuo Hu, Wen Li, Jing Zhang 1 |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 52 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-08309