Evaluating Large Language Model with Knowledge Oriented Language Specific Simple Question Answering
Bowen Jiang, Runchuan Zhu, Jiang Wu, Zinco Jiang, Yifan He, Junyuan Gao, Jia Yu, Rui Min, Yinfan Wang, Haote Yang, Songyang Zhang, Dahua Lin, Lijun Wu, Conghui He · 2025
Read Evaluating Large Language Model with Knowledge Oriented Language Specific Simple Question Answering by Bowen Jiang, Runchuan Zhu, Jiang Wu, Zinco Jiang, Yifan He, Junyuan Gao, Jia Yu, Rui Min, Yinfan Wang, Haote Yang, Songyang Zhang, Dahua Lin, Lijun Wu, Conghui He (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-16591.txt.
Introduction
Bowen Jiang and co-authors launch a benchmark for factual knowledge in many languages in a 2025 paper. Their question set tests simple facts with one clear answer each, built for fast grading by a model judge. It spans nine languages and two domains: world facts and facts tied to each language own culture and history.
Top models were tested, including newer reasoning models. Scores, rankings, and self-awareness of ignorance split sharply between global and local domains. A model strong on world facts could lag badly on local ones. Readers get a sharper map of where language models truly know, and where they only guess.
Catalog record
| CREATOR | Bowen Jiang, Runchuan Zhu, Jiang Wu, Zinco Jiang, Yifan He, Junyuan Gao, Jia Yu, Rui Min, Yinfan Wang, Haote Yang, Songyang Zhang, Dahua Lin, Lijun Wu, Conghui He |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 67 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-16591
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Evaluating Large Language Model with Knowledge Oriented Language Specific Simple Question Answering Bowen Jiang, Runchuan Zhu, Jiang Wu, Zinco Jiang, Yifan He, Junyuan Gao, Jia Yu, Rui Min, Yinfan Wang, Haote Yang, Songyang Zhang, Dahua Lin, Lijun Wu, Conghui He Abstract We introduce KoLasSimpleQA, the first benchmark evaluating the multilingual factual ability of Large Language Models (LLMs). Inspired by existing research, we created the question set with features such as single knowledge point coverage, absolute objectivity, unique answers, and temporal stability. These questions enable efficient evaluation using the LLM-as-judge paradigm, testing both the LLMs’ factual memory and self-awareness (“know what they don’t know”). KoLasSimpleQA expands existing research in two key dimensions: (1) Breadth (Multilingual Coverage) : It includes 9 languages, supporting global applicability evaluation. (2) Depth (Dual Domain Design) : It covers both the general domain (global facts) and the language-specific domain (such as history, culture, and regional traditions) for a comprehensive assessment of multilingual capabilities. We evaluated mainstream LLMs, including traditional LLM and emerging Large Reasoning Models. Results show significant performance differences between the two domains, particularly in performance metrics, ranking, calibration, and robustness. This highlights the need for targeted evaluation and optimization in multilingual contexts. We hope KoLasSimpleQA will help the research community better identify LLM capability boundaries in multilingual contexts and provide guidance for model optimization. We will release KoLasSimpleQA at https://github.com/opendatalab/KoLasSimpleQA . 1 Introduction Large Language Models ( Grattafiori et al. (2024) ; Yang et al. (2024) ; Guo et al. (2025) ) have advanced significantly, yet hallucination—where models produce unverified or misleading information—remains a major challenge, affecting their reliability and broader use. To tackle this, the SimpleQA ( Wei et al. (2024) ) and ChineseSimpleQA ( He et al. (2024b) ) benchmarks were introduced to evaluate LLMs’ factual ability using short, fact-based questions. These questions focus on a single knowledge point, with answers that are objective, stable, and not open to interpretation. Recent studies ( Zhang et al. (2023) ; Shi et al. (2022) ; Huang et al. (2023a) ) indicate that LLM performance varies across languages, particularly in factual ability, with models typically performing better in English. However, the SimpleQA and ChineseSimpleQA benchmarks are limited to English and Chinese. Additionally, most evaluations of LLMs in non-English contexts focus on general knowledge rather than language-specific content like history, culture, and local traditions. While LLMs excel in general knowledge, they often struggle with language-specific facts. Benchmark Lang. Lang. specific Easy to evaluate XTREME ( Hu et al. (2020) ) 40 lang. ✗ ✗ Okapi ( Lai et al. (2023) ) en ✗ ✗ SimpleQA ( Wei et al. (2024) ) en ✗ ✓ ChineseSimpleQA ( He et al. (2024b) ) zh ✗ ✓ MINTQA ( He et al. (2024a) ) en ✗ ✗ BenchMAX ( Huang et al. (2025) 17 lang. ✗ ✗ MMLU-ProX ( Xuan et al. (2025) ) 42 lang. ✗ ✓ KoLasSimpleQA(Ours) 9 lang. ✓ ✓ Table 1 : Comparison between KoLasSimpleQA and other benchmarks. The comparison is conducted along three dimensions: the range of supported languages, whether the benchmark includes language-specific knowledge, and whether it is easy to evaluate. To tackle this problem, we introduce K nowledge- O riented La nguage- S pecific Simple Q uestion A nswering (KoLasSimpleQA), a benchmark comprising simple fact-based QA samples grounded in genuinely language-specific knowledge across nine languages. KoLasSimpleQA has three main features: (1) Foundation: Inspired by Wei et al. (2024) ; He et al. (2024b) , we crafted a question set with attributes such as single knowledge point coverage , absolute objectivity , unique answers , and temporal stability . These questions enable efficient evaluation using the LLM-as-judge paradigm, assessing both the factual memory and self-awareness of LLMs (i.e., their ability to ”know what they don’t know”). (2) Breadth Expansion: Unlike existing work Wei et al. (2024) ; He et al. (2024b) that is limited to a few languages, KoLasSimpleQA includes 9 languages , allowing for performance evaluation of LLMs in multilingual contexts and supporting assessments of global applicability. (3) Depth Exploration: It encompasses both the general domain (global facts) and the language-specific domain (such as history, culture, and regional traditions). We collected data from Wikipedia, categorizing entries into general and language-specific domains based on the number of inter-language links each article has (as shown in Figure 2 ; see details in § 2.2.1 ). Using these classifications and filtered entries, we constructed questions for both domains, enabling a comprehensive and in-depth evaluation of LLMs in multilingual settings. We evaluated mainstream LLMs on KoLasSimpleQA, covering both traditional LLMs and the latest Large Reasoning Models (LRMs). Key insights from our study include: (1) Performance Disparity Across Domains: LLMs perform much worse in the language-specific domain than in the general domain. (2) Translating Non-English Queries: Translating queries into English is a common strategy to enhance multilingual performance. While effective in the general domain, it is less so in the language-specific domain. (3) Calibration Performance: LLMs show significantly poorer calibration in the language-specific domain compared to the general domain. (4) Knowledge Memorization Robustness: LLMs are notably less robust in the language-specific domain than in the general domain. 2 KoLasSimpleQA 2.1 Overview We created a multilingual QA benchmark including both the general and language-specific domains. The general domain covers global knowledge shared across languages, while the language-specific subset targets knowledge unique to individual linguistic and cultural contexts. Our benchmark includes 9 languages: Hungarian (hu), Czech (cs), Serbian (sr), Russian (ru), Chinese (zh), Korean (ko), Thai (th), Arabic (ar), and Vietnamese (vi), along with their English (en) translations. The data construction process is shown in Figure 1 . Figure 1 : Construction pipeline of KoLasSimpleQA. The process includes Wikipedia entry selection based on inter-language links, triple and QA pair generation using GPT-4o, and a two-stage quality control to ensure question quality and diversity. 2.2 Benchmark Construction 2.2.1 Collection of Specific & General Wikipedia Entries We crawled all Wikipedia pages in nine languages and extracted their contents. Unlike conventional methods focusing only on entry content, we specifically extracted inter-language link information for each entry and counted the number of these links, denoted as n_{\text{ill}} . An inter-language link is a hyperlink that connects a Wikipedia page in one language to the related page in another language, aiding navigation across different language versions. These links are usually located in the sidebar at the top right of the webpage, as shown in Figure 2 . It’s important to note that pages in different language versions of Wikipedia are not machine-translated; they are created by users who are typically native speakers of the respective language. If a page exists in a particular language, the topic or knowledge it represents is likely of interest to native speakers of that language. Thus, if an entity lacks versions in other languages ( n_{\text{ill}}=0 ), it is classified as language-specific , indicating the knowledge is unique to that language. Conversely, if an entity appears in many language versions ( n_{\text{ill}} is large), it likely represents global knowledge , making it general . For each of the nine languages, we sample a total of N_{\text{specific}}+N_{\text{general}} entries. Specifically, we randomly select N_{\text{specific}} entries from those where n_{\text{ill}}=0 , ensuring that they represent language-specific knowledge. Formally, this subset is defined as