LLMs Lost in Translation: M-ALERT uncovers Cross-Linguistic Safety Inconsistencies

Felix Friedrich, Simone Tedeschi, Patrick Schramowski, Manuel Brack, Roberto Navigli, Huu Nguyen, Bo Li, Kristian Kersting · 2024

Read LLMs Lost in Translation: M-ALERT uncovers Cross-Linguistic Safety Inconsistencies by Felix Friedrich, Simone Tedeschi, Patrick Schramowski, Manuel Brack, Roberto Navigli, Huu Nguyen, Bo Li, Kristian Kersting (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-15035.txt.

Description

Felix Friedrich and co-authors test language model safety across languages in a 2024 paper. A model that refuses a harmful request in English may comply in Italian, and safety work rarely checks. Their benchmark, M-ALERT, holds fifteen thousand prompts in each of five languages, tagged by harm category.

Thirty nine leading models faced the full set. Safety swung widely by language and topic: one flagship model failed often on tax crime prompts in Italian while staying safe elsewhere. A few categories, like cannabis and propaganda crime, tripped nearly every model in every language. The authors argue that multilingual safety testing must become routine, since users everywhere deserve the same guardrails.

Catalog record

CREATORFelix Friedrich, Simone Tedeschi, Patrick Schramowski, Manuel Brack, Roberto Navigli, Huu Nguyen, Bo Li, Kristian Kersting
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE95 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-15035

Browse the full library · About the Tower of Babel