GIMMICK Globally Inclusive Multimodal Multitask Cultural Knowledge Benchmarking
Florian Schneider, Carolin Holtermann, Chris Biemann, Anne Lauscher · 2025
Read GIMMICK Globally Inclusive Multimodal Multitask Cultural Knowledge Benchmarking by Florian Schneider, Carolin Holtermann, Chris Biemann, Anne Lauscher (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-13766.txt.
Description
Florian Schneider, Carolin Holtermann, Chris Biemann, and Anne Lauscher built a test of cultural knowledge for image-aware AI models in twenty twenty-five. Their benchmark, called GIMMICK, spans one hundred forty-four countries in six world regions. It holds six tasks drawn from seven hundred twenty-eight cultural events and topics, from food to rituals.
The team ran twenty image-language models and eleven text-only models through the tasks. Nearly all models favored Western cultures. Bigger models scored higher, and adding images plus hints about place helped. Models knew more about concrete things like food than about customs like rituals. They could often name a culture's broad origin but missed finer detail. The work gives builders a shared yardstick for fairer, more world-aware AI.
Catalog record
| CREATOR | Florian Schneider, Carolin Holtermann, Chris Biemann, Anne Lauscher |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 163 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-13766