IndicIFEval: A Benchmark for Verifiable Instruction-Following Evaluation in 14 Indic Languages

Thanmay Jayakumar Mohammed Safi Ur Rahman Khan, Raj Dabre Ratish Puduppully, Anoop Kunchukuttan · 2026

Read IndicIFEval: A Benchmark for Verifiable Instruction-Following Evaluation in 14 Indic Languages by Thanmay Jayakumar Mohammed Safi Ur Rahman Khan, Raj Dabre Ratish Puduppully, Anoop Kunchukuttan (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-22125.txt.

Introduction

Instruction-following tests are mostly English, leaving hundreds of millions of Indic-language speakers without a fair yardstick. Thanmay Jayakumar and four co-authors introduce IndicIFEval, a benchmark for constrained generation across fourteen Indic languages, using instructions that can be checked by rule. Each language gets hundreds of human-verified examples, split between localized translations and newly written native content.

Evaluations of major open and proprietary models show a split: formatting rules are mostly obeyed, while lexical and cross-lingual tasks trip models up badly. Progress in high-resource languages has not closed the gap for the wider Indic family. The benchmark gives builders a concrete target for serving India's languages as well as they serve English.

Catalog record

CREATORThanmay Jayakumar Mohammed Safi Ur Rahman Khan, Raj Dabre Ratish Puduppully, Anoop Kunchukuttan
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE54 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-22125

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

IndicIFEval: A Benchmark for Verifiable Instruction-Following Evaluation in 14 Indic Languages Thanmay Jayakumar Mohammed Safi Ur Rahman Khan, Raj Dabre Ratish Puduppully, Anoop Kunchukuttan Abstract Instruction-following benchmarks remain predominantly English-centric, leaving a critical evaluation gap for the hundreds of millions of Indic language speakers. We introduce IndicIFEval , a benchmark evaluating constrained generation of LLMs across 14 Indic languages using automatically verifiable, rule-based instructions. It comprises around 800 human-verified examples per language spread across two complementary subsets: IndicIFEval-Trans , translated prompts from IFEval 28 carefully localized for Indic contexts, and IndicIFEval-Ground , synthetically generated instructions grounded in native Indic content. We conduct a comprehensive evaluation of major open-weight and proprietary models spanning both reasoning and non-reasoning models. While models maintain strong adherence to formatting constraints, they struggle significantly with lexical and cross-lingual tasks – and despite progress in high-resource languages, instruction-following across the broader Indic family lags significantly behind English. We release IndicIFEval and its evaluation scripts to support progress on multilingual constrained generation 1 1 1 http://github.com/ai4bharat/IndicIFEval . rmTeXGyreTermesX [hindi]rm[ Path=./fonts/ ]NotoSansDevanagari-Regular.ttf Figure 1: Overview of the dataset construction pipeline 1 Introduction Instruction following, the ability of a model to accurately interpret user intent and comply with specified instructions, is a critical aspect of LLM training and evaluation. This capability is commonly assessed using either human annotators 14 ; 3 or LLM-as-a-Judge approaches 27 ; 17 . However, human evaluation is costly and time-consuming, while LLM-based judges can be inconsistent and biased 24 ; 27 . A complementary paradigm using automatically verifiable instructions (where adherence can be determined through simple rule-based checks) has therefore gained traction, as exemplified by the IFEval benchmark 28 . These include instructions such as requesting the presence of certain words, a fixed number of sentences, or specific formats (e.g., JSON), all checkable via automated scripts. At a multilingual level, this evaluation gap becomes more pronounced, as LLMs are often significantly weaker in non-English languages – not just as evaluators 12 ; 8 but also in text generation 1 ; 23 . This raises a natural question: In multilingual and low-resource settings, to what extent can LLMs at least follow simple, rule-based, verifiable constraints? We address this gap with IndicIFEval , a benchmark evaluating verifiable instruction-following across 14 Indic languages, a linguistically diverse family characterized by data scarcity, morphological richness, and distinct script systems, making it a meaningful testbed for both multilingual and low-resource evaluation. Our contributions are: • IndicIFEval-Trans , translated and localized prompts from the English IFEval benchmark, carefully filtered and manually verified by native speakers for cultural suitability and translation quality. • IndicIFEval-Ground , synthetically generated instructions grounded in native Indic topics and content, manually verified by native speakers. Unlike translated prompts, these reflect naturalistic constraints with more real-world contexts. • The first comprehensive multilingual evaluation of verifiable instruction-following spanning diverse model families, parameter sizes from 0.5B to 70B, and both reasoning and non-reasoning configurations, and both open-weight and proprietary models. • A comprehensive framework for extending verifiable instruction-following benchmarks to new languages with the availability of models tailored for translation or general use, and corpora of local contexts. Together, IndicIFEval-Trans and IndicIFEval-Ground form around 800 human-verified examples per language, providing complementary perspectives on multilingual instruction-following: the former enabling direct comparison against English baselines, the latter grounding evaluation in more natural constraints that translation alone cannot capture. 2 Related Work Several recent works have extended the verifiable instruction-following paradigm to multilingual contexts. M-IFEval 7 adapts the format to French, Japanese, and Spanish, highlighting that while models excel at formatting constraints, they struggle with script-specific tasks (e.g., “do not use Katakana”). Similarly, CL-IFEval 20 expands this slightly to include French, Spanish, Hindi, Arabic, and Yoruba. These works mainly employ translations. More comprehensive benchmarks have recently been proposed. MaXIFE 18 , introduces a dataset covering 23 languages with 795 prompts and \sim 1.7k constraint templates, resulting in over 18,000 possible combinations. It features a diverse taxonomy derived from real examples, utilizing both LLM-as-a-judge and normal accuracy evaluation. However, one limitation is its reliance on translation without any localization. 13 craft new Hindi prompts with verifiable constraints through an in-house human annotation process guided by reference samples and Indian cultural themes. Marco-Bench-MIF 26 evaluates 30 languages with a specific focus on the distinction between translation and localization. They identify that machine-translated (MT) data underestimates model accuracy by 7–22% compared to localized data, largely due to implicit English constraints (like capitalization) that become invalid in translation. While this offers deep insights into localization, it does not explore creating new prompts from grounded contexts. Further, it does not compare performance between reasoning and non-reasoning modes, leaving a gap in understanding how reasoning affects instruction adherence. 3 Dataset Construction In this section we detail the process of obtaining the translations based on the English IFEval dataset 28 and LLM-based generations grounded in Indic contexts from sangraha-verified 15 . An overview of the process is illustrated in Figure 1 . 3.1 IndicIFEval-Trans We use Sarvam-Translate 21 for the translations unless explicitly stated otherwise. Preprocessing First, we manually inspect the prompts in the IFEval dataset and assess their usability in non-English contexts. We localize the data with Indic-relevant themes wherever possible by simple modifications of named entities like ‘President of USA’ to ‘Prime Minister of India’ etc. For constraints involving explicit English responses (e.g., capitalization), we prepend the text “Respond in English.” to such prompts. Second, we spot cases that can have problematic translations - abbreviations in the source prompt that may not make sense in non-English languages, improper translations due to translationese, translation collapse, tense-changes, and incomplete translations caused by the MT system. In most cases, we fix this by replacing problematic terms with another that would be easier to translate, however in few cases we chose to drop them. The exact details are outlined in Appendix B . In total, we arrive at 490 from the original 541 examples in this process. Keyword Extraction + Individual Keyword Translation We identify the prompts with keyword constraints and extract all the keywords. The keywords are first translated individually. The keywords that should not be translated (e.g, prompts where the response language is another language), are also first translated, but they are stored separately. Pre-translation Insertion Then, we insert the keyword translation into the respective English prompt in place of the English keywords. We call this the ‘pre-translation’. Full Translation Next, we translate the pre-translated prompts to the target language. Note that for non-keyword constraints, we translate them directly without modifications. We note that we preserve the non-English keywords as highlighted earlier. This is done by replacing the keyword automatically back to its original source after translation. Automatic Verification Finally, using regular expressions, we check that the constraints are described properly in the translations. We manually fix these errors in formatting wherever necessary. We discuss this in detail in Appendix B . 3.2 IndicIFEval-Ground While translation provides a strong baseline for multilingual evaluation, it often lacks cultural grounding and may result in text that may be grammatically correct but constraints that are unnatural. We observe that this unnaturalness is especially true for keyword and length constraints. Required keywords are often arbitrary translations of English terms, which may not naturally occur in

Read the complete text