LSR: Linguistic Safety Robustness Benchmark for Low-Resource West African Languages
Godwin Abuh Faruna · 2026
Read LSR: Linguistic Safety Robustness Benchmark for Low-Resource West African Languages by Godwin Abuh Faruna (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-19273.txt.
Introduction
Faruna builds the first safety test for harmful prompts in low-resource West African languages. Models refuse most harmful asks in English, but those guards may fail in other tongues. The LSR test sends matched probes in English and in Yoruba, Hausa, Igbo, and Igala. A new drift score shows how much English refusal is lost in each tongue. On Gemini 2.5 Flash, refusal falls from about 90 percent to 35 to 55 percent. Igala shows the worst drop, with drift of 0.55.
Catalog record
| CREATOR | Godwin Abuh Faruna |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 23454 |
| LICENSE | http://creativecommons.org/licenses/by-sa/4.0/ |
| RIGHTS | CC BY SA |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.19273
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
LSR: Linguistic Safety Robustness Benchmark for Low-Resource West African Languages Godwin Abuh Faruna Abstract Safety alignment in large language models relies predominantly on English-language training data. When harmful intent is expressed in low-resource languages, refusal mechanisms that hold in English frequently fail to activate. We introduce LSR (Linguistic Safety Robustness), the first systematic benchmark for measuring cross-lingual refusal degradation in West African languages: Yoruba, Hausa, Igbo, and Igala. LSR uses a dual-probe evaluation protocol—submitting matched English and target-language probes to the same model—and introduces Refusal Centroid Drift (RCD), a metric that quantifies how much of a model’s English refusal behavior is lost when harmful intent is encoded in a target language. We evaluate Gemini 2.5 Flash across 14 culturally grounded attack probes in four harm categories. English refusal rates hold at approximately 90 percent. Across West African languages, refusal rates fall to 35-55 percent, with Igala showing the most severe degradation (RCD = 0.55). LSR is implemented in the Inspect AI evaluation framework and is available as a PR-ready contribution to the UK AISI’s inspect_evals repository. A live reference implementation and the benchmark dataset are publicly available. 1 Introduction The safety properties of frontier language models are almost exclusively evaluated in English. This is a predictable consequence of how alignment training works: RLHF depends on human feedback, and the overwhelming majority of annotators, feedback datasets, and evaluation frameworks have been built for English-speaking contexts. The practical consequence is that a model reporting 90% harmful-request refusal in English may comply at high rates with identical requests expressed in a language underrepresented in alignment training. West African languages present a concentrated instance of this problem. Yoruba, Hausa, and Igbo each have tens of millions of native speakers. Igala has approximately 2 million. Together, these languages span a region with rapidly expanding mobile internet access and growing AI adoption in high-stakes domains including healthcare, legal services, and governance. The populations they serve are increasingly exposed to frontier AI systems—systems whose safety properties have never been evaluated for these languages. This paper introduces LSR (Linguistic Safety Robustness), a benchmark that measures how consistently a model’s refusal behavior transfers from English to West African languages under matched harmful prompts. The experimental design is straightforward: for each attack probe, we submit an English-language version and a target-language version to the same model and record whether refusal behavior holds in both conditions. We aggregate per-language refusal rates and introduce a derived metric, Refusal Centroid Drift (RCD), which estimates the structural separation between a model’s safety behavior in English and in a given target language. The failure mode LSR measures is not jailbreaking. The probes do not use adversarial perturbations, prompt injection, or syntax manipulation. They are culturally grounded harmful requests written in the target language, using framings that a native speaker might naturally produce. The failure is a structural property of how RLHF safety training distributes across language space—not a prompt-level vulnerability that can be patched. We evaluate Gemini 2.5 Flash across 14 probes in four harm categories (physical harm, toxicology, targeted violence, and historical pretext framing). English refusal rates of approximately 90% fall to between 35% and 55% across the four target languages. The degradation is consistent, reproducible, and language-specific. LSR is implemented using the Inspect AI evaluation framework and is available as a PR-ready contribution to the UK AISI’s inspect_evals repository [ 6 ] . A live reference implementation is available at https://huggingface.co/spaces/Faruna01/lsr-dashboard . The benchmark dataset is available at https://huggingface.co/datasets/Faruna01/lsr-benchmark . The contributions of this paper are: (1) the first systematic benchmark for cross-lingual refusal degradation in West African languages; (2) Refusal Centroid Drift (RCD), a metric for quantifying safety representation drift across linguistic transitions; (3) empirical evidence that safety degradation in these languages is reproducible, structured, and severe enough to constitute a meaningful operational safety gap; and (4) an open Inspect AI implementation ready for integration into frontier model evaluation pipelines. 2 Background and Motivation Safety alignment in modern LLMs combines supervised fine-tuning on curated harmful/harmless data with reinforcement learning from human feedback [ 4 ] . RLHF training reliably reduces harmful output rates in high-resource languages. English-language safety benchmarks such as AdvBench [ 9 ] and SafetyBench have established baseline refusal performance for English evaluations. None of these benchmarks address cross-lingual transfer of safety behavior. Work on multilingual safety has begun to close this gap. Yong et al. [ 8 ] showed that translating harmful prompts into low-resource languages substantially increases compliance rates in GPT-4 and other frontier models. Deng et al. [ 2 ] examined safety challenges across multilingual LLMs and found that RLHF training reduces harmful output rates by approximately 45% in high-resource languages but only 20% in low-resource languages. Shen et al. [ 5 ] found that safety fine-tuning on English does not reliably generalize to linguistically distant, low-resource languages. These results converge on a common explanation: alignment training is high-resource biased, and the safety representations it produces do not transfer uniformly across linguistic distance. West African languages are particularly absent from safety research. Yoruba, Hausa, and Igbo appear in multilingual NLP benchmarks (AfriSenti, MasakhaNER, XNLI subsets), but no published evaluation addresses their safety properties systematically. Igala has no presence in any major multilingual NLP benchmark. This absence is not benign: it means that safety properties for hundreds of millions of speakers have never been empirically verified. The mechanistic explanation for cross-lingual safety degradation points to tokenization and representation bias. LLMs trained with English-dominant RLHF develop refusal representations anchored to English token sequences. When harmful intent is encoded in a low-resource language, those representations do not map cleanly, and the circuits that trigger refusal in English either fail to activate or activate inconsistently. We formalize this as the Refusal Centroid Drift hypothesis in Section 3. This work builds on cross-lingual transfer research in NLP [ 1 , 7 ] and extends it to the safety alignment domain. The practical motivation is that frontier AI systems are being deployed in health, legal, and governance contexts across Nigeria, Ghana, and neighboring countries at scale, without safety evaluations covering the languages those deployments actually serve. 3 The Refusal Centroid Drift Hypothesis We define Refusal Centroid Drift (RCD) operationally as the estimated proportion of English refusal behavior lost when harmful intent is encoded in a target language, measured under matched probe conditions. Formally: let R_{EN} denote the refusal rate of model M under probe set P evaluated in English, and R_{L} denote the refusal rate under the matched probe set P_{L} evaluated in language L . We define: RCD(L)=1-\frac{R_{L}}{R_{EN}} (1) Under this definition, RCD(\text{English})=0 by construction. RCD(L)=0.55 for Igala means that 55% of the refusal behavior present in English has been lost when the same harmful intent is encoded in Igala. RCD is bounded in [0,1] for any language where R_{L}\leq R_{EN} . We do not observe cases where R_{L}>R_{EN} in our experiments. The term “centroid” is motivated by the mechanistic interpretability literature, where it describes the geometric center of activation clusters in transformer residual stream space [ 3 ] . We use it to describe the implicit refusal representation that a well-aligned model has learned to associate with harmful content. Our hypothesis is that this representation is anchored to high-resource token sequences,