MSTS: A Multimodal Safety Test Suite for Vision-Language Models

Paul Röttger, Giuseppe Attanasio , Felix Friedrich , Janis Goldzycher , Alicia Parrish, Rishabh Bhardwaj, Chiara Di Bonaventura, Roman Eng, Gaia El Khoury Geagea, Sujata Goswami, Jieun Han, Dirk Hovy, Seogyeong Jeong, Paloma Jeretič, Flor Miriam Plaza-del-Arco, Donya Rooein, Patrick Schramowski, Ana · 2025

Read MSTS: A Multimodal Safety Test Suite for Vision-Language Models by Paul Röttger, Giuseppe Attanasio , Felix Friedrich , Janis Goldzycher , Alicia Parrish, Rishabh Bhardwaj, Chiara Di Bonaventura, Roman Eng, Gaia El Khoury Geagea, Sujata Goswami, Jieun Han, Dirk Hovy, Seogyeong Jeong, Paloma Jeretič, Flor Miriam Plaza-del-Arco, Donya Rooein, Patrick Schramowski, Ana (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-10057.txt.

Description

Paul Röttger and co-authors build a safety test set for models that take in both pictures and text, in work from 2025. Such models now sit inside chat tools used by the public. Without guards, they can give harmful advice, and the added picture channel brings new risks that text-only tests miss.

The set holds 400 prompts in forty narrow hazard groups. In each prompt, the text and picture are safe alone but risky together. Several open models failed the tests, while some passed only because they missed the point of easy prompts. Prompts in tongues other than English drew more unsafe replies. Text-only runs were safer than mixed runs, and auto graders for safety still fell short.

Catalog record

CREATORPaul Röttger, Giuseppe Attanasio , Felix Friedrich , Janis Goldzycher , Alicia Parrish, Rishabh Bhardwaj, Chiara Di Bonaventura, Roman Eng, Gaia El Khoury Geagea, Sujata Goswami, Jieun Han, Dirk Hovy, Seogyeong Jeong, Paloma Jeretič, Flor Miriam Plaza-del-Arco, Donya Rooein, Patrick Schramowski, Ana
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE43 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-10057

Browse the full library · About the Tower of Babel