DIWALI - Diversity and Inclusivity aWare cuLture specific Items for India Dataset and Assessment of LLMs for Cultural Text Adaptation in Indian Context
Pramit Sahoo Maharaj Brahma Maunendra Sankar Desarkar · 2025
Read DIWALI - Diversity and Inclusivity aWare cuLture specific Items for India Dataset and Assessment of LLMs for Cultural Text Adaptation in Indian Context by Pramit Sahoo Maharaj Brahma Maunendra Sankar Desarkar (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-17399.txt.
Introduction
Pramit Sahoo, Maharaj Brahma, and Maunendra Sankar Desarkar test how well language models handle Indian culture. Their 2025 arXiv paper notes that these models often miss cultural alignment and produce biased text, thanks to thin cultural knowledge. Judging that competence is hard, since few data sets capture culture at the level of regions and sub-regions.
Their data set, DIWALI, collects culture-specific items from many sub-regions of India across a range of cultural facets. Thousands of concepts anchor the set. The team uses it to grade models on adapting text to Indian contexts, scoring with the items, with model judges, and with human raters from varied backgrounds. The grading exposes patchy coverage across sub-regions and shallow, surface-level adaptations. The work gives builders a sharper mirror for cultural skill.
Catalog record
| CREATOR | Pramit Sahoo Maharaj Brahma Maunendra Sankar Desarkar |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 69 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-17399
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
DIWALI - Diversity and Inclusivity aWare cuLture specific Items for India Dataset and Assessment of LLMs for Cultural Text Adaptation in Indian Context Pramit Sahoo Maharaj Brahma Maunendra Sankar Desarkar Abstract Large language models (LLMs) are widely used in various tasks and applications. However, despite their wide capabilities, they are shown to lack cultural alignment ( Ryan et al. 2024 ; AlKhamissi et al. 2024 ) and produce biased generations Naous et al. 2024 due to a lack of cultural knowledge and competence. Evaluation of LLMs for cultural awareness and alignment is particularly challenging due to the lack of proper evaluation metrics and unavailability of culturally grounded datasets representing the vast complexity of cultures at the regional and sub-regional levels. Existing datasets for culture specific items (CSIs) focus primarily on concepts at the regional level and may contain false positives. To address this issue, we introduce a novel CSI dataset for Indian culture, belonging to 17 cultural facets. The dataset comprises ∼ \sim 8k cultural concepts from 36 sub-regions. To measure the cultural competence of LLMs on a cultural text adaptation task, we evaluate the adaptations using the CSIs created, LLM as Judge, and human evaluations from diverse socio-demographic region. Furthermore, we perform quantitative analysis demonstrating selective sub-regional coverage and surface-level adaptations across all considered LLMs. Our dataset is available here: https://huggingface.co/datasets/nlip/DIWALI , project webpage 1 1 1 https://nlip-lab.github.io/nlip/publications/diwali/ , and our codebase with model outputs can be found here: https://github.com/pramitsahoo/culture-evaluation . 0 0 footnotetext: Equal contributions 1 Introduction Figure 1: Comparison of the number of cultural concepts between CANDLE Nguyen et al. 2023 and Ours ( DIWALI ), highlighting differences in coverage of cultural concepts for five facets (clothing, drink, cuisine, rituals, and traditions) available in CANDLE (Domain = Country , Subject = India ). Large language models (LLMs), despite having vast knowledge and being trained on extensive large data, are shown to lack cultural knowledge and competence Naous et al. 2024 ; Wang et al. 2024 ; Masoud et al. 2025 . This is primarily attributed to highly biased pre-training data towards Western culture Li et al. 2024 ; Naous and Xu 2025 ; Li et al. 2025 . Cultural text adaptation refers to modifying text from a general or specific culture to align with a particular target cultural group’s linguistic, social, and conceptual norms. Cultural adaptation has a wide range of applications, ranging from education Burstein et al. 2007 , dialogue Singh et al. 2024 , movie subtitling Ferreira 2024 ; El-Farahaty and Alwazna 2024 , hospitality Radojević et al. 2024 , advertisement Barthwal and Gupta 2011 ; Li 2024 , cross-cultural communication Wilczewski et al. 2019 , and storytelling Bhatt and Diaz 2024 ; Hamna et al. 2025 . However, ensuring effective and emotionally resonating cultural adaptation is challenging, as it requires deep understanding and knowledge of social norms, subjective historical references, resonating events, and cultural nuances. With its vast linguistic and cultural diversity, India presents a unique challenge for culture-driven NLP systems. In particular, India houses 28 states and eight union territories 2 2 2 https://knowindia.india.gov.in/states-uts/ , each with multiple languages, dialects, traditions, and socio-cultural norms. Furthermore, cultural diversity exists at the state level and within sub-regions. For example, the state of Assam has communities like Bodo, Rabha, Karbi, and so on -- each with distinct cultural norms, artifacts, and language 3 3 3 https://assam.gov.in/about-us/391 . Despite recent advances, existing LLMs struggle with cultural adaptation. Studies Naous et al. 2024 have shown that LLMs trained on English-centric corpora often fail to generate culturally appropriate content, especially for cultures from different parts of the world. In this work, we systematically evaluate multiple open-weight LLMs for cultural text adaptation on closed-text generation . In particular, we utilize existing text corpora from the education domain belonging to American culture. We rely on cultural concepts to assess the cultural competence of various LLMs for cultural text adaptation tasks. Culture Specific Items (CSIs) represent a particular culture’s concepts, objects, items, and customs. For example, Mekhela Chador (clothing), Pongal (food), and Darjeeling Tea (drink) are cultural concepts from India. Prior work, CANDLE Nguyen et al. 2023 , has made foundational efforts to automatically extract culturally specific concepts from large web text for various cultures. In another work, DOSA Seth et al. 2024 builds community-driven cultural artifacts for a sub-regional culture belonging to India. However, these datasets have a limited representation of cultural concepts from India. Figure 1 shows the comparison of the created dataset with respect to existing concepts from the CANDLE 4 4 4 https://candle.mpi-inf.mpg.de/ framework. To address this gap, we built a novel, large-scale dataset of CSIs for the Indian subregion. In particular, our dataset contains \sim 8k cultural concepts from 17 facets. We evaluate using automatic CSI-based and human evaluation metrics to systematically assess the adaptation quality of various LLMs. Our dataset provides a valuable resource to the community for further research in the comprehensive and standardized assessment of cultural text adaptation. We define culture at the regional or country level, focusing on India. In summary, our contributions are as follows: • We present DIWALI , a high-quality Culture Specific Items (CSIs) dataset of India, covering a total of 36 subregions 5 5 5 Represents states and union territories of India . Our dataset contains a total of 8,817 concepts from 17 cultural facets. • We assess 7 open-weight models across 3 LLM families using both automatic and human evaluation metrics for cultural text adaptation. • We conduct a cultural sub-regional evaluation and analysis of LLMs in the context of cultural text adaptation. 2 Culture & Culture Specific Items Culture is a multifaceted and complex construct. It means different things to different groups of people. It encompasses knowledge, beliefs, morals, values, and customs. Broadly, we can divide culture into material and non-material components. Material components include tangible elements such as food, dress, houses, and ornaments. In contrast, non-material components refer to symbols, ideas, beliefs, norms, values, morality, and attitudes that guide social behavior. For example, the practice of fasting during Navratris 6 6 6 a religious observance common in Northern India. . Culture is a complex concept and hard to define. As a result, the community has turned to using demographic and semantic proxies to approximately define culture Adilazuarda et al. 2024 . Demographic proxies typically include geographical region, language, gender, race, religion, education, and ethnicity. In contrast, people associate semantic proxies with emotional expressions, food habits, social and political relations, actions, and naming conventions. In this work, we consider culture as a regional demographic proxy – with the geographical country India as a region and its states and union territories as sub-regions. Newmark 2003 defined culture as “… the way of life and its manifestations that are peculiar to a community that uses particular languages as a means of expression.” This definition views culture as a collection of objects, customs, and traditions that hold meaning within a specific community. Newmark 2003 categorizes cultural manifestations into five collections or facets: (a) ecology, (b) material culture, (c) social culture, (d) organizations, customs, activities, procedures, concepts, and (e) gestures and habits. Culture Specific Items represent concepts, objects, items, and customs that hold significant cultural importance and relevance to a particular