MCIF: Multimodal Crosslingual Instruction-Following Benchmark from Scientific Talks
Sara Papi, Maike Züfle, Marco Gaido, Beatrice Savoldi, Danni Liu, Ioannis Douros, Luisa Bentivogli, Jan Niehues · 2025
Read MCIF: Multimodal Crosslingual Instruction-Following Benchmark from Scientific Talks by Sara Papi, Maike Züfle, Marco Gaido, Beatrice Savoldi, Danni Liu, Ioannis Douros, Luisa Bentivogli, Jan Niehues (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-19634.txt.
Description
Eight authors, led by Sara Papi, present MCIF in this 2025 paper, a benchmark for AI models that follow instructions across languages and senses. New multimodal models read text, hear speech, and see images in one system. Testing them is hard: most benchmarks stick to English, one sense at a time, and short inputs.
MCIF is built from scientific talks, which naturally mix slides, speech, and text. Human annotators labeled the tasks, so scores rest on human judgment, not machine guesses. Models must handle long inputs and several languages at once. The benchmark shows where today's systems break: following a spoken instruction about a slide, in a language other than English, over a full talk. That joint test is what real-world use will demand.
Catalog record
| CREATOR | Sara Papi, Maike Züfle, Marco Gaido, Beatrice Savoldi, Danni Liu, Ioannis Douros, Luisa Bentivogli, Jan Niehues |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 49 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-19634