LLMs Lost in Translation: M-ALERT uncovers Cross-Linguistic Safety Inconsistencies
Felix Friedrich, Simone Tedeschi, Patrick Schramowski, Manuel Brack, Roberto Navigli, Huu Nguyen, Bo Li, Kristian Kersting · 2024
Read LLMs Lost in Translation: M-ALERT uncovers Cross-Linguistic Safety Inconsistencies by Felix Friedrich, Simone Tedeschi, Patrick Schramowski, Manuel Brack, Roberto Navigli, Huu Nguyen, Bo Li, Kristian Kersting (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-15035.txt.
Description
Felix Friedrich and co-authors test language model safety across languages in a 2024 paper. A model that refuses a harmful request in English may comply in Italian, and safety work rarely checks. Their benchmark, M-ALERT, holds fifteen thousand prompts in each of five languages, tagged by harm category.
Thirty nine leading models faced the full set. Safety swung widely by language and topic: one flagship model failed often on tax crime prompts in Italian while staying safe elsewhere. A few categories, like cannabis and propaganda crime, tripped nearly every model in every language. The authors argue that multilingual safety testing must become routine, since users everywhere deserve the same guardrails.
Catalog record
| CREATOR | Felix Friedrich, Simone Tedeschi, Patrick Schramowski, Manuel Brack, Roberto Navigli, Huu Nguyen, Bo Li, Kristian Kersting |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 95 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-15035