Toward Structural Multimodal Representations: Specialization, Selection, and Sparsification via Mixture-of-Experts
Hahyeon Choi, Nojun Kwak · 2026
Read Toward Structural Multimodal Representations: Specialization, Selection, and Sparsification via Mixture-of-Experts by Hahyeon Choi, Nojun Kwak (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-03348.txt.
Description
Hahyeon Choi and Nojun Kwak propose S3, a mixture-of-experts framework for multimodal learning. It builds representations from selectable semantic parts, instead of one fixed embedding. Forcing all signals into a single vector can waste capacity. It can also blur structure that a task needs.
Three steps define the method. Specialization forms concept-level experts in a shared space. Selection routes them per task. Sparsification prunes low-value paths for compact results. Across four MultiBench benchmarks, accuracy improved. Performance followed a reverse U shape in sparsity, peaking in the middle. The authors present selectable parts as a principled option for multimodal models.
Catalog record
| CREATOR | Hahyeon Choi, Nojun Kwak |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 90 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.03348