The Narrow Gate: Localized Image-Text Communication in Native Multimodal Models
Alessandro Pietro Serra, Francesco Ortu, Emanuele Panizon, Lucrezia Valeriani, Lorenzo Basile, Alessio Ansuini, Diego Doimo, Alberto Cazzaniga, Area Science Park, Trieste, Italy SISSA, Trieste, Italy University of Trieste, Trieste, Italy {diego.doimo, alberto.cazzaniga}@areasciencepark.it · 2024
Read The Narrow Gate: Localized Image-Text Communication in Native Multimodal Models by Alessandro Pietro Serra, Francesco Ortu, Emanuele Panizon, Lucrezia Valeriani, Lorenzo Basile, Alessio Ansuini, Diego Doimo, Alberto Cazzaniga, Area Science Park, Trieste, Italy SISSA, Trieste, Italy University of Trieste, Trieste, Italy {diego.doimo, alberto.cazzaniga}@areasciencepark.it (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-06646.txt.
Description
Models that see and speak must pass what the eyes learn over to the words, and that handoff differs sharply by how the model was raised. Alessandro Pietro Serra and seven co-authors map the traffic in 2024, comparing models trained from scratch on mixed media with those adapted from text-only giants.
In native multimodal models, image and text signals stay more separate inside the network. Text-adapted models spread visual news across many image tokens, while native ones funnel it through a single token after the image, a narrow gate. Deleting that gate token wrecked image understanding, and nudging it steered the picture's meaning in the output text with fine control. The result reframes how builders should wire seeing into saying.
Catalog record
| CREATOR | Alessandro Pietro Serra, Francesco Ortu, Emanuele Panizon, Lucrezia Valeriani, Lorenzo Basile, Alessio Ansuini, Diego Doimo, Alberto Cazzaniga, Area Science Park, Trieste, Italy SISSA, Trieste, Italy University of Trieste, Trieste, Italy {diego.doimo, alberto.cazzaniga}@areasciencepark.it |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 55 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-06646