Rebalanced Multimodal Learning with Data-aware Unimodal Sampling
Qingyuan Jiang, Zhouyang Chi, Xiao Ma, Qirong Mao, Yang Yang, Jinhui Tang · 2025
Read Rebalanced Multimodal Learning with Data-aware Unimodal Sampling by Qingyuan Jiang, Zhouyang Chi, Xiao Ma, Qirong Mao, Yang Yang, Jinhui Tang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-03792.txt.
Description
Qingyuan Jiang and co-authors rebalance multimodal learning at the sampling stage in a 2025 paper. Models that fuse sound, image, and text often let one stream dominate while others lag. Most fixes adjust the training math, but the authors blame a quieter cause: equal sampling feeds each stream the same count, not the same information.
Their method tracks a running measure of the gap between streams as training proceeds. Sampling counts then shift on the fly, by a rule of thumb or by a learned controller, so lagging streams get more data. The sampler plugs into nearly any existing multimodal method without surgery. Tests against a range of strong baselines put the approach on top. The authors frame sampling, long treated as neutral, as a first-class lever for fair multimodal training.
Catalog record
| CREATOR | Qingyuan Jiang, Zhouyang Chi, Xiao Ma, Qirong Mao, Yang Yang, Jinhui Tang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 40 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-03792