Bridging the Culture Gap : A Framework for LLM-Driven Socio-Cultural Localization of Math Word Problems in Low-Resource Languages

Israel Abebe Azime, Tadesse Destaw Belay, Dietrich Klakow, Philipp Slusallek, Anshuman Chhabra · 2025

Read Bridging the Culture Gap : A Framework for LLM-Driven Socio-Cultural Localization of Math Word Problems in Low-Resource Languages by Israel Abebe Azime, Tadesse Destaw Belay, Dietrich Klakow, Philipp Slusallek, Anshuman Chhabra (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-14913.txt.

Introduction

Israel Abebe Azime and co-authors present a framework to localize math word problems across cultures in this 2025 paper. Models solve math well in English but lag in low resource languages. Test sets made by plain translation keep English names, organizations, and money. That hides the true difficulty of native context problems.

Their framework uses language models to rebuild problems with native entities. Names, institutions, and currencies are swapped for local equivalents at low cost. Experiments show translated tests can misstate multilingual math ability. Proper localization gives a fairer score. The method also reduced English centered bias in the resulting datasets. It offers a cheap path to culturally grounded benchmarks for many languages.

Catalog record

CREATORIsrael Abebe Azime, Tadesse Destaw Belay, Dietrich Klakow, Philipp Slusallek, Anshuman Chhabra
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-14913

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Bridging the Culture Gap : A Framework for LLM-Driven Socio-Cultural Localization of Math Word Problems in Low-Resource Languages Israel Abebe Azime, Tadesse Destaw Belay, Dietrich Klakow, Philipp Slusallek, Anshuman Chhabra Abstract Large language models (LLMs) have demonstrated significant capabilities in solving mathematical problems expressed in natural language. However, multilingual and culturally-grounded mathematical reasoning in low-resource languages lags behind English due to the scarcity of socio-cultural task datasets that reflect accurate native entities such as person names, organization names, and currencies. Existing multilingual benchmarks are predominantly produced via translation and typically retain English-centric entities, owing to the high cost associated with human annotater-based localization. Moreover, automated localization tools are limited, and hence, truly localized datasets remain scarce. To bridge this gap, we introduce a framework for LLM-driven cultural localization of math word problems that automatically constructs datasets with native names, organizations, and currencies from existing sources. We find that translated benchmarks can obscure true multilingual math ability under appropriate socio-cultural contexts. Through extensive experiments, we also show that our framework can help mitigate English-centric entity bias and improves robustness when native entities are introduced across various languages. † † ∗ Equal Contribution. 1 Introduction Figure 1: An example showcasing an English math word problem, its direct translation, and an automatically localized version with culturally adapted entities. While the problem structure remains identical, large language models (LLMs) often fail to answer correctly when entity names or currencies are altered. This highlights a key limitation in current LLM robustness. In this paper, our goal is to audit models and rectify their robustness issues so that remain consistent and accurate across such culturally grounded variations. Mathematical reasoning has been adopted as a core milestone in large language model understanding research Yan et al. 2024 ; Ahn et al. 2024a ; Ahn et al. 2024b . Math word problems (MWPs), a key component of mathematical reasoning tasks, have been widely explored as a challenging benchmark for LLMs Srivatsa and Kochmar 2024 . MWPs are characterized by their integration of mathematical knowledge with scenarios drawn from everyday activities, which can vary across cultures . The ability of low-resource and multilingual LLMs to solve MWPs correctly depends on multiple factors: (1) the language used to prompt the models Adelani et al. 2024 and (2) the linguistic complexity of the questions Srivatsa and Kochmar 2024 . Despite their impressive performance on MWPs, a crucial question remains: do LLMs truly retain performance under culturally diverse MWPs, even when the underlying mathematical structure remains unchanged? Low-resource language (LRL) math word problem evaluation heavily relies on human or machine-translated benchmarks, as demonstrated by the translation of GSM8K Cobbe et al. 2021 (a dataset of 8.79K high-quality, linguistically diverse grade school math word problems) into GSM8K_zh for Chinese Yu et al. 2023 , MGSM for ten typologically diverse languages Shi et al. 2022a , and AfriMGSM for 18 African languages Adelani et al. 2024 . Existing human- or machine-translated benchmarks often fail to account for local cultural contexts that contain day to day activities, common names and currencies. As a result, evaluating LLMs on such non-localized data primarily measures their mathematical understanding ability in English centric scenarios, names and currencies. In this work, we investigate the task of creating more culturally aligned math word problems from their translated variants. Owing to the high costs associated with human-based localization, it is imperative to develop automated frameworks capable of producing large-scale datasets. Therefore, this motivates our first research question in this work: ( RQ1 ) Can LLM-driven pipelines be used to localize translated benchmarks for low-resource languages? Moreover, given that a significant number of benchmark datasets are created using translation Alabi et al. 2025 and do not study appropriate socio-culturally grounded datasets (see Figure 1 ) we propose an automated framework for socio-cultural localization of MWPs in low-resource languages. Using our framework, we can study our second research question, as follows: ( RQ2 ) Does introducing socio-cultural local entities in translated existing benchmarks reveal performance disparities? We undertake several experiments to study RQ2 for both low-resource languages (LRLs) and English to ascertain the performance gaps. Finally, we utilize our framework to generate localized MWPs for 18 African Languages covered by Adelani et al. 2024 and investigate our final research question: ( RQ3 ) Can automated localization improve model robustness by augmenting benchmarks with culturally adapted variants? To address these questions, our work makes the following contributions: • We develop an LLM-driven localization pipeline to generate culturally adapted versions of translated datasets by replacing key entities with apt socio-cultural variants. 1 1 1 https://huggingface.co/collections/israel/cultural-localilzation-687e392f6be2aaa0b7c7831c • Further, through extensive experiments enabled via our framework, we investigate performance disparities between localized and directly translated benchmarks across several LLMs like Gemma and LLaMA models. Our findings reveal how the presence of cultural entities influences LLMs’ ability to solve mathematical word problems. For instance, we observe as much as 9% (numeric match) drop in performance for GPT-4o-mini (with similar trends seen across other LLMs). • Then, using the localized data generated by our automated localization framework, we fine-tune LLMs on these socio-cultural MWP variants and find that model robustness and generalization improves significantly across multiple languages, thereby paving the way for improving the performance of LLMs in a straightforward manner. Stages Example 1 English x_{\text{en}} Mandy owes Benedict $ 100 . They agreed to have monthly interest of 2%. If Mandy was able to pay it after 3 months, how much should she give to Benedict ? 2 Translated x_{\text{trans}}\leftarrow x_{\text{en}} Mandy anadaiwa $ 100 na Benedict . Wamekubali kuwa na riba ya kila mwezi ya 2%. Ikiwa Mandy aliweza kulipa baada ya miezi 3, anafaa kumpa Benedict pesa ngapi? 3 Important entities ent=\texttt{\hbox{\pagecolor{myYellow}LLM}}(x_{\text{en}}) {‘‘personal_names’’:[‘‘Mandy’’,‘‘Benedict’’],‘‘currencies’’:[‘‘$’’]’’, ‘‘organization_names’’:[]} 4 replacement dictionary replacement\_dict=fn(en,db) {‘‘Mandy’’: ‘‘Camari’’, ‘‘Benedict’’: ‘‘Julani’’, ‘‘$’’: ‘‘shilingi’’, ‘‘dollar’’: ‘‘shilingi’’} 5 Entity replaced x_{\text{ent}}=fn(x_{\text{ent}},replacement\_dict) Camari owes Julani shilingi 100 . They agreed to have monthly interest of 2%. If Camari was able to pay it after 3 months, how much should she give to Julani ? 6 Auto Localized \hat{x}_{\text{loc}}=\texttt{\hbox{\pagecolor{myYellow}LLM}}((x_{\text{en}},x_{\text{trans}}),x_{\text{ent}}) Camari anadaiwa shilingi 100 na Julani . Wamekubali kuwa na riba ya kila mwezi ya 2%. Ikiwa Camari aliweza kulipa baada ya miezi 3, anafaa kumpa Julani pesa ngapi? 7 Quality Check length( \hat{x}_{\text{loc}} ) == length( x_{\text{trans}} ), key entities not in \hat{x}_{\text{loc}} , replacement entities in \hat{x}_{\text{loc}} , and similarity( \hat{x}_{\text{loc}} , x_{\text{trans}} ) > 0.8, Full human verification for the test set and sampled human verification for the training set. if failed return ( x_{\text{trans}} ) Table 1: The different stages of our MWP automated localization framework for low-resource languages . We show a step-by-step transformation from English to a culturally and linguistically localized Swahili version: direct translation, name and currency replacements, a semi-localized substitution, and the final fluent localization. Colored highlights indicate aligned entities across languages; occurrences of LLM denote the use of an LLM. 2 Related Work Math World Problems. Math word problems are mathematical questions framed in everyday language, often grounded in real-life scenarios and activities rather than expressed purely through abstract symbols or equations Cobbe et al. 2021 . These problems serve as an engaging way to learn mathematical reasoning, as they require not only computational skills but also the ability to interpret and

Read the complete text