UbuntuGuard: A Culturally-Grounded Policy Benchmark for Equitable AI Safety in African Languages
Tassallah Abdullahi, Macton Mgonzo, Mardiyyah Oduwole, Paul Okewunmi, Abraham Owodunni, Ritambhara Singh, Carsten Eickhoff · 2026
Read UbuntuGuard: A Culturally-Grounded Policy Benchmark for Equitable AI Safety in African Languages by Tassallah Abdullahi, Macton Mgonzo, Mardiyyah Oduwole, Paul Okewunmi, Abraham Owodunni, Ritambhara Singh, Carsten Eickhoff (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-12696.txt.
Introduction
Tassallah Abdullahi and six co authors build a safety benchmark for AI in African languages. Guardian models check AI outputs for harm, but most are Western centric. They are tuned for high resource languages and set safety categories. Low resource African languages are left open to cross lingual safety failures and cultural misalignment.
Rigid, predefined categories also fail to generalize across sociocultural contexts. What counts as harm, and how it is phrased, varies by culture and tongue. Their benchmark, UbuntuGuard, is culturally grounded for this reason. It tests whether safety systems judge African language content fairly and well. Equitable safety is a precondition for fair global AI deployment.
Catalog record
| CREATOR | Tassallah Abdullahi, Macton Mgonzo, Mardiyyah Oduwole, Paul Okewunmi, Abraham Owodunni, Ritambhara Singh, Carsten Eickhoff |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-12696
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
UbuntuGuard: A Culturally-Grounded Policy Benchmark for Equitable AI Safety in African Languages Tassallah Abdullahi, Macton Mgonzo, Mardiyyah Oduwole, Paul Okewunmi, Abraham Owodunni, Ritambhara Singh, Carsten Eickhoff Abstract Current guardian models are predominantly Western-centric and optimized for high-resource languages, leaving low-resource African languages vulnerable to evolving harms, cross-lingual safety failures, and cultural misalignment. Moreover, most guardian models rely on rigid, predefined safety categories that fail to generalize across diverse linguistic and sociocultural contexts. Robust safety, therefore, requires flexible, runtime-enforceable policies and benchmarks that reflect local norms, harm scenarios, and cultural expectations. We introduce UbuntuGuard, the first African policy-based safety benchmark built from adversarial queries authored by 155 domain experts across sensitive fields, including healthcare. From these expert-crafted queries, we derive context-specific safety policies and reference responses that capture culturally grounded risk signals, enabling policy-aligned evaluation of guardian models. We evaluate 13 models, comprising six general-purpose LLMs and seven guardian models across three distinct variants: static, dynamic, and multilingual. Our findings reveal that existing English-centric benchmarks overestimate real-world multilingual safety, cross-lingual transfer provides partial but insufficient coverage, and dynamic models, while better equipped to leverage policies at inference time, still struggle to fully localize African-language contexts. These findings highlight the urgent need for multilingual, culturally grounded safety benchmarks to enable the development of reliable and equitable guardian models for low-resource languages. Our code can be found online. 1 1 1 Code repository available at https://github.com/hemhemoh/UbuntuGuard . 1 Introduction Large Language Models (LLMs) have revolutionized various aspects of natural language processing (NLP), enabling advances in question answering 19 , synthetic data generation 9 ; 5 , scientific discovery 1 , and beyond. These models are typically trained on massive web-based corpora 2 , which may contain social biases and harmful or offensive content. Consequently, LLMs can generate outputs that are not only incorrect but also unsafe or socially harmful. Scaling these models further enhances capabilities 3 , but also amplifies risks, creating a misalignment between model capabilities and societal safety expectations 27 . To mitigate these risks, safety mechanisms have been adopted, including supervised fine-tuning on safety-aligned datasets, reinforcement learning from human feedback (RLHF) 11 ; 22 , and adversarial testing through red-teaming. More recently, guardian models 14 ; 16 or specialized guardrails have emerged as promising defenses, monitoring and filtering unsafe outputs before they reach users. Despite these efforts, significant safety gaps remain. Guardian models are often trained on high-resource language (HRL) data, leaving low-resource languages (LRLs) susceptible to evolving harms, cross-lingual vulnerabilities, and cultural misalignment 29 ; 8 . Most models also rely on rigid, predefined safety categories that do not generalize across diverse linguistic and cultural contexts. Safety is inherently culturally relative, as definitions of harm and acceptable behavior vary across societies, resulting in conflicting legal and ethical requirements. This centralization dilemma, which involves deciding who defines safety rules, combined with divergent requirements across application domains such as healthcare, finance, or education, further complicates the deployment of universally safe LLMs. Figure 1: UbuntuGuard construction pipeline. The pipeline has two stages: (A) data generation and (B) quality control and filtering . Given an input query, we first generate context-aware English policies using GPT-5 (1). These policies, together with the original queries, are used to generate multi-turn user–agent dialogues via NeMo Curator (2). Policies and dialogues are then translated into multiple target languages to form raw multilingual policy–dialogue (MDP) data (3). In the quality control stage, translation quality is evaluated using automatic assessment with GEMBA (4), followed by targeted human review (5). Addressing these challenges requires a new approach. Effective safety mechanisms should rely on flexible, runtime-enforceable policies that can adapt to diverse cultural and geographic contexts. By supporting the plurality of perspectives and granting local regions a measure of sovereignty in defining acceptable behavior, dynamic policies enable guardian models to operate responsibly and contextually, rather than imposing a single universal standard. We introduce UbuntuGuard, the first African policy-based safety benchmark designed to evaluate the robustness of guardian models in culturally and linguistically diverse settings. UbuntuGuard is constructed from adversarial queries authored by 155 domain experts, including physicians, teachers, religious leaders, bankers, lawyers, and human rights advocates, spanning a range of sensitive domains. From these expertly crafted queries, we derive context-specific safety policies and reference responses that enable realistic, policy-aligned evaluation of guardian model behavior. Our major contributions are threefold: • We introduce UbuntuGuard, covering 10 low-resource African languages across multiple themes and domains. • We perform a comprehensive evaluation of seven state-of-the-art guardian models and six general-purpose open-source models. • We highlight insights and future directions for dynamic, culturally aware, and universally adaptable guardian models. 2 Related Work Dynamic Guardian Models: Ensuring the safety and trustworthiness of LLM-generated content has become a central research focus. Early efforts introduced static guardian models 15 , which apply fixed rules or predefined safety criteria to moderate model output. Although effective in constrained settings, these static systems are inherently rigid and cannot easily adapt to evolving user needs or context-specific policies. To address these limitations, recent work has explored dynamic guardian models 14 , which enable user-defined safety policies and more flexible customization at inference time. These dynamic approaches consistently outperform static models in adaptability and policy alignment 14 . However, they remain predominantly evaluated in English-only settings, leaving their robustness and generalizability across multilingual or culturally diverse contexts, particularly in low-resource languages, largely untested. Multilingual Safety Benchmarks. As AI systems are deployed worldwide, the need for multilingual and culturally informed safety evaluations has become increasingly apparent. Previous work demonstrates that safety norms differ widely in cultural contexts 18 ; 16 ; 25 , and that guardian models trained primarily in English struggle with multilingual content, code-switching, or references to local institutions. Although several multilingual safety benchmarks have emerged 7 ; 8 , they overwhelmingly focus on high-resource languages and lack coverage of linguistic phenomena common in low-resource settings. Existing efforts to study policy robustness to multilingual jailbreaks underscore the limitations of current benchmarks 18 ; 16 , which continue to omit African languages. African Languages and Cultural Underrepresentation. African languages remain critically underrepresented in multilingual safety datasets, and methods to curate culturally aware resources often rely on translations or scaffolding from high-resource languages 16 . This is particularly problematic given the linguistic diversity of the African continent, with more than 2,000 languages, and the prevalence of multilingual communication, code-switching, and culturally rooted expressions in daily interactions. Existing guardian models may therefore be ill-equipped to capture culturally specific harms, context-dependent toxicity, and regionally relevant safety norms. These gaps highlight the need for evaluation frameworks that natively reflect African linguistic diversity and sociocultural contexts. UbuntuGuard addresses this need by providing a culturally grounded benchmark designed to support the development and evaluation of safer AI systems for African languages. Dataset Policy Type Dialogues # Langs. African JailbreakBench 4 Static ✓ 1 ✗ WildGuardTest 13 Static ✓ 1 ✗ Aegis-2.0 10 Static ✓ 1 ✗ XSafety 26 Static ✗ 10 ✗ MultiJail 8 Static ✗ 10 ✗ PolyGuard 18 Static ✓ 19 ✗ NemotronSafety-Guard-Dataset-v3 Static ✓ 9 ✗ 16 DynaBench 14 Dynamic ✓ 1 ✗ UbuntuGuard (Ours) Dynamic ✓ 10 ✓ Table 1: Comparison of Safety Benchmarks: Policy Type, Evaluated Dialogues, Language Coverage, and African Language Inclusion. 3