Lost in Cultural Translation: Do LLMs Struggle with Math Across Cultural Contexts?

Aabid Karim *, Abdul Karim *, Bhoomika Lohana *, Matt Keon, Jaswinder Singh, Abdul Sattar · 2025

Read Lost in Cultural Translation: Do LLMs Struggle with Math Across Cultural Contexts? by Aabid Karim *, Abdul Karim *, Bhoomika Lohana *, Matt Keon, Jaswinder Singh, Abdul Sattar (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-18018.txt.

Introduction

A word problem about dollars and pizza tests the same math as one about rupees and flatbread, but language models do not treat them alike. Aabid Karim and co-authors rewrote nearly twelve hundred grade-school math problems into six cultural settings, from Haiti to Somalia, changing names, foods, and places while freezing every number and operation.

Fourteen major models all lost accuracy in unfamiliar settings, by up to about six points for the weakest. Error analysis showed reasoning slips, not just arithmetic ones. Familiarity could also help: one model with regional training data beat bigger rivals on Pakistan-set problems. The authors call for more culturally diverse training data.

Catalog record

CREATORAabid Karim *, Abdul Karim *, Bhoomika Lohana *, Matt Keon, Jaswinder Singh, Abdul Sattar
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE76 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-18018

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Lost in Cultural Translation: Do LLMs Struggle with Math Across Cultural Contexts? Aabid Karim *, Abdul Karim *, Bhoomika Lohana *, Matt Keon, Jaswinder Singh, Abdul Sattar Abstract We demonstrate that large language models’ (LLMs) mathematical reasoning is culturally sensitive: testing 14 models from Anthropic, OpenAI, Google, Meta, DeepSeek, Mistral, and Microsoft across six culturally adapted variants of the GSM8K benchmark, we find accuracy drops ranging from 0.3% (Claude 3.5 Sonnet) to 5.9% (LLaMA 3.1-8B) when math problems are embedded in unfamiliar cultural contexts, even when the underlying mathematical logic remains unchanged. These statistically significant performance reductions (p < < 0.01, confirmed through McNemar tests) reveal that mathematical reasoning in LLMs is not culturally neutral. To create these variants for Haiti, Moldova, Pakistan, Solomon Islands, Somalia, and Suriname, we systematically replaced cultural entities (names, foods, places, etc.) in 1,198 GSM8K questions while preserving all mathematical operations and numerical values. Validation through 130 randomly sampled questions with two independent human annotators (73.8% and 79.2% accuracy) confirmed entity recognition quality, with discrepancies manually corrected. Our quantitative error analysis of 18,887 instances reveals that while explicitly cultural errors (currency: 4.0%, relationships: 4.5%) represent a minority, cultural adaptation affects broader reasoning patterns, with mathematical reasoning errors comprising 54.7% and calculation errors 34.5% of failures. Interestingly, cultural familiarity can enhance performance: Mistral Saba, despite lacking explicit mathematical training, outperforms some larger models on Pakistan-adapted problems due to Middle Eastern and South Asian training data exposure. Our analysis suggests multiple mechanisms contribute to performance degradation, including tokenization differences, unfamiliar cultural references disrupting reasoning pathways, and training data biases toward Western contexts. This study underscores the need for more diverse training data to ensure robust LLM performance across global contexts. † † footnotetext: * These authors contributed equally to this work. 1 Introduction Large Language Models (LLMs) have transformed AI, taking on tasks that once seemed to require human intuition. These models have shown remarkable progress in the field of natural language processing, problem solving, question answering, and computer vision tasks ( Li et al.,, 2025 ) , ( Chu et al.,, 2024 ) , ( Gunter et al.,, 2024 ) , ( Team et al.,, 2023 ) . Their ability to interpret and code complex mathematical reasoning has gained a tremendous amount of attention from the research community ( Mirzadeh et al.,, 2024 ) . However, beneath their impressive capabilities lies a crucial question: Do LLMs truly understand the world’s cultural diversity, or reflect simply the cultural limitation embedded in their training data? More specifically, do they retain their mathematical reasoning when presented with culturally adapted math word problems? How does cultural diversity—or lack thereof—in their training data impact their mathematical reasoning abilities? Studies have consistently highlighted cultural, gender, and sociopolitical biases in LLMs, particularly their Western-centric learning, which affects fairness and adaptability in different linguistic and cultural contexts ( Ramesh et al.,, 2023 ; Ramezani and Xu,, 2023 ) . Hovy and Spruit, (2016) argue that such biases arise because NLP systems reflect the social and cultural characteristics of their training data, often disadvantaging underrepresented populations. Prior work has further shown that surface-level changes in representation, such as names or character roles, can systematically affect model outputs even when the underlying task structure remains constant ( Lucy and Bamman,, 2021 ) , and that models exhibit culturally skewed tendencies in multilingual settings ( Naous et al.,, 2024 ) . Beyond cultural bias, LLMs exhibit fundamental limitations in formal reasoning. Research suggests that LLMs do not engage in formal reasoning ( Valmeekam et al.,, 2023 ) but instead rely on probabilistic pattern-matching to generate outputs based on training data rather than a true conceptual understanding of symbols or ideas ( Boix-Adsera et al.,, 2024 ) . Studies have shown that LLMs struggle with formal logical reasoning, as their outputs are highly sensitive to individual token changes ( Jiang et al.,, 2024 ) . A single transformer layer functions similarly to a one-nearest neighbor algorithm ( Li et al.,, 2024 ) , meaning the model’s reasoning process is heavily influenced by the closest matching examples in its training data. The relationship between training frequency and test performance further supports this idea ( Razeghi et al.,, 2022 ) , demonstrating that LLMs rely more on statistical associations than genuine logical inference. These reasoning limitations intersect with cultural context in a specific way. LLMs are highly sensitive to input tokens, and even slight changes in their tokenization process can alter their reasoning ( Grattafiori et al.,, 2024 ) . If they have not been exposed to diverse and underrepresented cultural norms and contexts during training, they may tokenize culturally specific prompts differently, potentially leading to shifts in reasoning and varied responses. Furthermore, as the number of tokens increases or tasks become more complex, the probability of accurately predicting tokens decreases ( Shi et al.,, 2023 ) . To address these questions, we use the GSM8K dataset ( Cobbe et al.,, 2021 ) , a widely recognized benchmark to evaluate the mathematical reasoning capabilities of LLM ( Mirzadeh et al.,, 2024 ) . However, GSM8K has two key limitations. First, given its extensive use, there is a high probability that LLMs have encountered this dataset during training, making it less reliable for assessing their reasoning abilities. Second, it consists of a single set of math problems, lacking any cultural diversity. To overcome these limitations, we modify the GSM8K test set by introducing culturally adapted versions of each question. Specifically, we synthesize six culturally diverse variants from the original GSM8K test set , one for each continent, while preserving the original mathematical logic and numerical values. The only changes involve replacing culturally specific entities such as names, food items, and contextual references with those relevant to each target culture. We then evaluate 14 LLMs, varying in size and release period, in these culturally adapted versions of GSM8K test set. A detailed comparison of this work with prior related research is provided in Section 2. The remainder of the paper is structured as follows. Section 3 details the dataset creation process, Section 4 discusses our culture (country) selection process, Section 5 shows detailed performance analysis of LLMs, and Sections 6 presents the conclusion. 2 Related Work A growing body of research has documented that NLP systems reflect the social and cultural characteristics of their training data, often disadvantaging underrepresented populations ( Hovy and Spruit,, 2016 ) , with fairness research remaining concentrated predominantly on English and high-resource languages ( Ramesh et al.,, 2023 ; Ramezani and Xu,, 2023 ) . Two lines of work are particularly relevant to our study. Lucy and Bamman, (2021) demonstrate that surface-level changes in representation, specifically character gender in narrative prompts, systematically affect GPT-3 outputs in topics and power dynamics, even when the task structure remains constant. Naous et al., (2024) introduce CAMeL, a dataset contrasting Arab and Western cultural entities, showing that both multilingual and Arabic monolingual LMs exhibit significant bias towards Western entities across tasks including story generation, NER, and sentiment analysis. While these studies establish that cultural context shapes model outputs, both evaluate open-ended tasks without objectively verifiable correct answers, and neither examines mathematical reasoning or spans multiple underrepresented cultures across continents. Our work

Read the complete text