Do You Know About My Nation? Investigating Multilingual Language Models’ Cultural Literacy Through Factual Knowledge
Eshaan Tanwar, Anwoy Chatterjee, Michael Saxon, Alon Albalak, William Yang Wang, Tanmoy Chakraborty, [0.5em] Indian Institute of Technology Delhi · 2025
Read Do You Know About My Nation? Investigating Multilingual Language Models’ Cultural Literacy Through Factual Knowledge by Eshaan Tanwar, Anwoy Chatterjee, Michael Saxon, Alon Albalak, William Yang Wang, Tanmoy Chakraborty, [0.5em] Indian Institute of Technology Delhi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-00657.txt.
Description
Eshaan Tanwar and five co-authors ask a pointed question of multilingual language models: what do you know about my nation? Standard benchmarks span many languages but lean Western in content, leaving regional knowledge untested. Their XNationQA benchmark holds nearly fifty thousand questions on the geography, culture, and history of nine countries, presented in seven languages, built to probe cultural literacy directly.
Eight standard multilingual models were benchmarked, scored with two new transference metrics that track knowledge moving between languages. Access to cultural facts varied sharply by language. Models often knew more about a culture in English than in that culture's own dominant language, and skill in Western languages did not mean literacy about Western countries. Knowledge transfer across languages proved limited, above all in open-source models.
Catalog record
| CREATOR | Eshaan Tanwar, Anwoy Chatterjee, Michael Saxon, Alon Albalak, William Yang Wang, Tanmoy Chakraborty, [0.5em] Indian Institute of Technology Delhi |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 63 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-00657