Towards Geo-Culturally Grounded LLM Generations
Piyawat Lertvittayakumjorn, David Kinney, Vinodkumar Prabhakaran, Donald Martin, Jr., Sunipa Dev · 2025
Read Towards Geo-Culturally Grounded LLM Generations by Piyawat Lertvittayakumjorn, David Kinney, Vinodkumar Prabhakaran, Donald Martin, Jr., Sunipa Dev (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-13497.txt.
Introduction
Language models often miss the norms of cultures far from their training data, and a 2025 paper by Piyawat Lertvittayakumjorn and co-authors tests two fixes. The team compared plain models with models fed facts from a custom knowledge base or from live web search.
Search-fed models scored much better on quiz-style tests of cultural facts, such as norms, objects, and institutions. The custom base helped less, held back by gaps in its coverage and weak retrieval. Yet search also raised the risk of flat, stereotyped judgments, and it did not improve how human raters judged open-ended replies. The authors draw a line between knowing cultural facts and showing real fluency in open talk.
Catalog record
| CREATOR | Piyawat Lertvittayakumjorn, David Kinney, Vinodkumar Prabhakaran, Donald Martin, Jr., Sunipa Dev |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 50 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-13497
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Towards Geo-Culturally Grounded LLM Generations Piyawat Lertvittayakumjorn, David Kinney, Vinodkumar Prabhakaran, Donald Martin, Jr., Sunipa Dev Abstract Generative large language models (LLMs) have been demonstrated to have gaps in diverse, cultural knowledge across the globe. We investigate the effect of retrieval augmented generation and search-grounding techniques on the ability of LLMs to display familiarity with a diverse range of national cultures. Specifically, we compare the performance of standard LLMs, LLMs augmented with retrievals from a bespoke knowledge base (i.e., KB grounding), and LLMs augmented with retrievals from a web search (i.e., search grounding) on a series of cultural familiarity benchmarks. We find that search grounding significantly improves the LLM performance on multiple-choice benchmarks that test propositional knowledge (e.g., the norms, artifacts, and institutions of national cultures), while KB grounding’s effectiveness is limited by inadequate knowledge base coverage and a suboptimal retriever. However, search grounding also increases the risk of stereotypical judgments by language models, while failing to improve evaluators’ judgments of cultural familiarity in a human evaluation with adequate statistical power. These results highlight the distinction between propositional knowledge about a culture and open-ended cultural fluency when it comes to evaluating the cultural familiarity of generative LLMs. 1 Introduction † † ⋆ equal contribution Contemporary large language models (LLMs) are pre-trained on huge corpora of natural language text Radford et al. 2019 and then fine-tuned using human feedback to improve their quality Bai et al. 2022 . During both processes, it is possible for text from a particular culture or cultures to be over-represented in the training data Dodge et al. 2021 and for the perspectives, norms, and mores of specific cultures to be over-represented in the feedback from human evaluators Prabhakaran et al. 2022 ; Atari et al. 2023 . Consequently, there is growing recognition of the shortcomings of generative LLMs with respect to their ability to represent and serve people of diverse geo-cultural backgrounds at the global scale Adilazuarda et al. 2024 ; Pawar et al. 2024 ; Agarwal et al. 2025 . Models tend to stereotype different cultures Jha et al. 2023 ; Bhutani et al. 2024 , erase and simplify their representation Qadri et al. 2025 , and provide very limited knowledge and context about artifacts and norms that are salient to them Myung et al. 2024 ; Rao et al. 2024 . Despite these gaps, strategies for intervening and eliciting culturally appropriate content from the models remain under-explored. We study two strategies to improve cultural awareness of LLM generation using external knowledge. In the first strategy, we construct a bespoke cultural knowledge base (KB) and apply a retrieval augmented generation (RAG) technique Lewis et al. 2020 ; Gao et al. 2023 so that the input to the language model includes relevant cultural text from the knowledge base for better generation. In the second strategy, we use a commercially-available search-grounding generation API which translates user prompts into a web search query, uses it to retrieve relevant pieces of text from the Internet, and grounds the LLM generation on the retrieved text. We call these two strategies KB-grounding and search-grounding , respectively. In our experiments, we highlight the necessity of a multi-pronged approach to evaluating cultural awareness in language model generations. Specifically, we leveraged multiple benchmarks Myung et al. 2024 ; Rao et al. 2024 ; Bhutani et al. 2024 to evaluate cultural knowledge and the ability to avoid cultural stereotyping of the models equipped with the two strategies, compared with the vanilla generation baseline. We also conducted a human evaluation of open-ended model responses to various prompts designed to test cultural awareness, wherein evaluators from a specific national culture rated how well the model’s output reflected a culturally informed perspective. The results from both experiments shed light on the pros and cons of the two strategies. Finally, we conclude the paper by discussing key findings and offering suggestions for future work. 2 Improving Cultural Awareness with External Knowledge Retrieval augmented generation (RAG) is a technique for enhancing the quality of large language model generation. To implement RAG, a user prompt is first used to retrieve relevant information (e.g., text or documents) from a database, which is then added to the prompt before being passed to a generative LLM to produce a grounded response Lewis et al. 2020 ; Gao et al. 2023 . RAG has shown to be effective in several applications, particularly those involving tasks the base LLM was not well-trained for such as fact verification Asai et al. 2023 ; Singal et al. 2024 ; Khaliq et al. 2024 and domain-specific question answering (QA) Seo et al. 2024 ; Xiong et al. 2024 ; Kim and Min 2024 . Some existing work also creates bespoke knowledge bases for use with RAG to tailor outputs to their specific applications Sun et al. 2025 ; Li et al. 2024 . Meanwhile, instead of querying a KB, other techniques retrieve external information by searching the internet, which allows access to up-to-date knowledge with high-quality ranking outputs Fan et al. 2024 ; Shuster et al. 2022 ; Lazaridou et al. 2022 ; Yao et al. 2022 ; Nakano et al. 2021 ; Komeili et al. 2022 . In light of these successes, we study both KB-grounding and search-grounding as techniques for improving the cultural awareness of LM generations. 2.1 The Knowledge Base Grounding Strategy We compiled culturally salient data from four large sources—CultureAtlas Fung et al. 2024 , Cube Kannen et al. 2024 , CultureBank Shi et al. 2024 , and SeeGULL Jha et al. 2023 —to serve as our knowledge base. For each entry in each source, we converted it into into text (if it was not already), embedded it into a vector, and added it to a vector store for querying. Fig. 1 (top) shows how the KB-grounding strategy handles an incoming prompt. First, a query rewriter extracts the important parts of the prompt to form a query. We can configure this step to test different KB queries (e.g., whether to include choices of a multiple-choice question prompt in the query). Next, we use the query to retrieve n documents from the KB. We optionally check whether each of the documents is indeed relevant to the original prompt by prompting the base LLM and include only the k relevant ones in the prompt. We call the process with the relevancy check step selective RAG , while non-selective RAG includes all n documents in the prompt. Then, we feed the augmented prompt to the LLM to get a raw answer. For multiple-choice question answering (QA) tasks, we create a manual verbalizer to map the raw answer to one of the choices so that we can compute the model accuracy. In contrast, we use the raw answer as the final output for an open-ended generation task. See the Appendix for details about the KB, the queries, the prompt templates, and the implementation. In Section 3 , we apply KB-grounding to three different LLMs: gemini-flash-1.5 (Gemini) Team et al. 2024 , gpt-4o-mini (GPT) OpenAI et al. 2024 and olmo2-7b (OLMo) OLMo et al. 2025 . Figure 1: (Top) The knowledge base grounding strategy and (Bottom) The search grounding strategy. Dashed boxes indicate optional steps that are executed only under