SANSKRITI : A Comprehensive Benchmark for Evaluating Language Models’ Knowledge of Indian Culture
Arijit Maji 1 , Raghvendra Kumar 1 , Akash Ghosh 1 , Anushka 2 , Sriparna Saha 1 1 Department of Computer Science and Engineering, Indian Institute of Technology Patna, India 2 Department of Computer Science, Banasthali Vidyapeeth University, Rajasthan, India {arijit_2311ai25,raghvendra_2221cs27,aka · 2025
Read SANSKRITI : A Comprehensive Benchmark for Evaluating Language Models’ Knowledge of Indian Culture by Arijit Maji 1 , Raghvendra Kumar 1 , Akash Ghosh 1 , Anushka 2 , Sriparna Saha 1 1 Department of Computer Science and Engineering, Indian Institute of Technology Patna, India 2 Department of Computer Science, Banasthali Vidyapeeth University, Rajasthan, India {arijit_2311ai25,raghvendra_2221cs27,aka (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-15355.txt.
Introduction
Maji and co-authors build a test of how well language models know India. Global models often miss local culture, so the team assembles SANSKRITI, over 21,800 curated question-answer pairs spanning all 28 states and 8 union territories. The questions cover sixteen themes, from rituals, food, and festivals to dress, medicine, transport, and sport.
Running leading large models, India-focused models, and small models on the set reveals wide gaps. Many systems handle famous facts yet stumble on region-specific ones. The authors present the benchmark as the largest of its kind for Indian culture and as a yardstick for making models that serve local users better.
Catalog record
| CREATOR | Arijit Maji 1 , Raghvendra Kumar 1 , Akash Ghosh 1 , Anushka 2 , Sriparna Saha 1 1 Department of Computer Science and Engineering, Indian Institute of Technology Patna, India 2 Department of Computer Science, Banasthali Vidyapeeth University, Rajasthan, India {arijit_2311ai25,raghvendra_2221cs27,aka |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 37 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-15355
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SANSKRITI : A Comprehensive Benchmark for Evaluating Language Models’ Knowledge of Indian Culture Arijit Maji 1 , Raghvendra Kumar 1 , Akash Ghosh 1 , Anushka 2 , Sriparna Saha 1 1 Department of Computer Science and Engineering, Indian Institute of Technology Patna, India 2 Department of Computer Science, Banasthali Vidyapeeth University, Rajasthan, India {arijit_2311ai25,raghvendra_2221cs27,akash_2321cs19,sriparna}@iitp.ac.in, [email protected] Abstract Language models (LMs) are indispensable tools shaping modern workflows, but their global effectiveness depends on understanding local socio-cultural contexts. To address this, we introduce SANSKRITI , a benchmark designed to evaluate language models’ comprehension of India’s rich cultural diversity. Comprising of 21,853 meticulously curated question-answer pairs spanning 28 states and 8 union territories, SANSKRITI is the largest dataset for testing Indian cultural knowledge. It covers sixteen key attributes of Indian culture namely rituals and ceremonies, history, tourism, cuisine, dance and music, costume, language, art, festivals, religion, medicine, transport, sports, nightlife and personalities, providing a comprehensive representation of India’s cultural tapestry. We evaluate SANSKRITI on leading Large Language Models (LLMs), Indic Language Models (ILMs), and Small Language Models (SLMs), revealing significant disparities in their ability to handle culturally nuanced queries, with many models struggling in region-specific contexts. By offering an extensive, culturally rich, and diverse dataset, SANSKRITI sets a new standard for assessing and improving the cultural understanding of LMs. 1 Introduction Language Models (LMs) have revolutionized the way we interact with technology, becoming indispensable tools in applications such as natural language understanding, content generation, knowledge discovery, and decision-making ( Brown et al. 2020 ; Ghosh et al. 2024c ; Devlin et al. 2019 ; Ghosh et al. 2024b ) . Models like Large Language Models (LLMs), Indic Language Models (ILMs), and Small Language Models (SLMs) are increasingly integrated into workflows across industries, enabling seamless communication and efficient problem-solving ( Ouyang et al. 2022 ) 1 1 1 In this study we define Large Language Model(LLM) as any language model of size more than or equal to 7 billion parameters, Small language models(SLM) are any that has less or equal to 5 billion parameters and Indic Language models (ILM) are any model that is developed inside India indigeneously . These advancements, however, come with the critical challenge of ensuring that LMs cater to diverse populations by recognizing and reasoning about their linguistic and cultural contexts ( Bender et al. 2021 ; Ghosh et al. 2025 ) . While LMs excel in syntactic and semantic language understanding, their effectiveness depends significantly on their ability to comprehend cultural diversity ( Lin et al. 2021 ) . A model’s capacity to recognize cultural contexts is crucial for its application in education, governance, healthcare, and entertainment in culturally rich societies ( Liang et al. 2022 ; Ghosh et al. 2024a ) . The inability to incorporate cultural nuances risks perpetuating biases, stereotypes, or inaccuracies, thereby alienating underrepresented communities. Models that can capture such diversity not only perform better but also foster inclusivity and equity in AI-driven solutions ( Blodgett et al. 2020 ) . India’s unparalleled cultural diversity makes it an ideal testbed for evaluating the cultural competence of LMs. The country is home to 28 states and 8 union territories, each characterized by distinct traditions, festivals, cuisines, music, dance forms, architectural styles, and historical narratives ( Sharma 2019 ) . For example, states like Rajasthan and Tamil Nadu exhibit unique artistic and culinary traditions, while festivals such as Diwali, Eid, and Onam highlight diverse religious and regional practices ( Narayanan 2020 ; Tripathi 2019 ) . This cultural richness presents significant challenges for LMs. Existing benchmarks like TyDi QA ( Joshi et al. 2020 ) and XQUAD ( Artetxe et al. 2020 ) primarily focus on multilingualism, overlooking cultural depth. Models trained on global datasets often fail to capture region-specific nuances, such as the significance of Kathakali performances in Kerala or Madhubani art in Bihar ( Patel et al. 2023 ) . This gap in understanding not only limits the utility of these models in culturally complex regions but also hinders their acceptance by local populations ( Liang et al. 2022 ) . Motivation for This Research : The absence of culturally refined benchmarks, especially based on different nuanced attributes, inspired the creation of SANSKRITI , a dataset designed to evaluate the cultural understanding of LMs in the Indian context. Unlike existing datasets like Seth et al. 2024a , which are smaller and do not include all Indian states and union territories, SANSKRITI focuses on cultural aspects across all 28 states and 8 union territories of India, providing a complete picture of the country’s rich cultural diversity. The motivation behind this research lies in bridging the gap between technological advancements in LMs and their real-world applicability in culturally diverse societies. We developed the SANSKRITI dataset to benchmark cultural understanding in India, utilizing diverse data sources to cover key attributes like rituals and ceremonies, history, tourism, cuisine, dance and music, costume, language, art, festivals, religion, medicine, transport, sports, nightlife, and personalities. A team of 40 annotators curated 21,853 questions across four categories: association, country prediction, general awareness, and state prediction. Research Questions: This research aims to address the following questions: 1. How do various language models (LLMs, SLMs, and ILMs) perform on the SANSKRITI dataset, and what are the performance trends across these model categories? 2. How do language models perform across different cultural attributes represented in the SANSKRITI dataset? 3. What are the performance trends of language models across different Indian states and union territories? 4. How do language models perform across various question types in the SANSKRITI dataset? Contributions: Our key contributions are summarized as follows: 1. Introduction of SANSKRITI : We present SANSKRITI , a dataset of 21853 manually curated and validated question-answer pairs, designed to evaluate the cultural competence of LMs in the Indian context. 2. Comprehensive Cultural Coverage : SANSKRITI encompasses 16 key cultural attributes spanning all Indian states and union territories. 3. Benchmarking Across Models : We evaluate leading LLMs, ILMs, and SLMs on SANSKRITI , identifying critical gaps in their ability to reason about culturally nuanced queries. 4. Public Availability : The SANSKRITI dataset and benchmarking results are made publicly available to encourage research in culturally inclusive AI. The resources are available at: https://drive.google.com/drive/folders/1UEkhrcA3-aPQTjTwSIgBC8EbqSS8ntDn?usp=sharing . 2 Related Works 2.1 Culture in LLMs Recent studies have examined the sociocultural reasoning of large language models (LLMs), focusing on their adaptability to diverse cultural values and contexts. Researchers have used tools like the World Values Survey and Hofstede’s dimensions to evaluate LLMs’ understanding of social norms and human values ( Johnson et al. 2022 ; Atari et al. 2023 ; Masoud et al. 2023 ) . Other works have explored cultural artifacts, such as food and art, highlighting LLMs’ knowledge but limited adaptability to user-specific scenarios ( Seth et al. 2024b ; Li et al. 2024 ) . Efforts to improve adaptability include enabling synthetic personas, but studies reveal gaps in understanding non-Western cultures and persistent biases ( AlKhamissi et al. 2024 ; Durmus et al. 2023 ) . Fine-tuning approaches have been applied to instill social norms and enhance performance on culture-specific tasks like hate speech detection, though probing in regional languages often underperforms compared to monolingual English testing ( Dwivedi and Patel