What do vision-language models see in the context? Investigating multimodal in-context learning
Gabriel O. dos Santos, Esther Colombini, Sandra Avila · 2025
Read What do vision-language models see in the context? Investigating multimodal in-context learning by Gabriel O. dos Santos, Esther Colombini, Sandra Avila (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-24331.txt.
Description
Gabriel O. dos Santos and two co-authors systematically study in context learning in vision language models, asking what these models actually take from demonstration examples. In context learning lets a model learn a task from examples in the prompt, with no weight updates. It is well studied for text models but underexplored for models that also see images.
The team evaluates seven models across four architectures on three image captioning benchmarks. They analyze how prompt design, architecture, and training strategy shape performance, and they are the first to track attention patterns as the number of demonstrations grows. Training on interleaved image text data improved in context learning, but did not guarantee true integration of the examples.
Catalog record
| CREATOR | Gabriel O. dos Santos, Esther Colombini, Sandra Avila |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 35 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-24331