MSTS: A Multimodal Safety Test Suite for Vision-Language Models
Paul Röttger, Giuseppe Attanasio , Felix Friedrich , Janis Goldzycher , Alicia Parrish, Rishabh Bhardwaj, Chiara Di Bonaventura, Roman Eng, Gaia El Khoury Geagea, Sujata Goswami, Jieun Han, Dirk Hovy, Seogyeong Jeong, Paloma Jeretič, Flor Miriam Plaza-del-Arco, Donya Rooein, Patrick Schramowski, Ana · 2025
Read MSTS: A Multimodal Safety Test Suite for Vision-Language Models by Paul Röttger, Giuseppe Attanasio , Felix Friedrich , Janis Goldzycher , Alicia Parrish, Rishabh Bhardwaj, Chiara Di Bonaventura, Roman Eng, Gaia El Khoury Geagea, Sujata Goswami, Jieun Han, Dirk Hovy, Seogyeong Jeong, Paloma Jeretič, Flor Miriam Plaza-del-Arco, Donya Rooein, Patrick Schramowski, Ana (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-10057.txt.
Description
Paul Röttger and co-authors build a safety test set for models that take in both pictures and text, in work from 2025. Such models now sit inside chat tools used by the public. Without guards, they can give harmful advice, and the added picture channel brings new risks that text-only tests miss.
The set holds 400 prompts in forty narrow hazard groups. In each prompt, the text and picture are safe alone but risky together. Several open models failed the tests, while some passed only because they missed the point of easy prompts. Prompts in tongues other than English drew more unsafe replies. Text-only runs were safer than mixed runs, and auto graders for safety still fell short.
Catalog record
| CREATOR | Paul Röttger, Giuseppe Attanasio , Felix Friedrich , Janis Goldzycher , Alicia Parrish, Rishabh Bhardwaj, Chiara Di Bonaventura, Roman Eng, Gaia El Khoury Geagea, Sujata Goswami, Jieun Han, Dirk Hovy, Seogyeong Jeong, Paloma Jeretič, Flor Miriam Plaza-del-Arco, Donya Rooein, Patrick Schramowski, Ana |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 43 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-10057