Eka-Eval : A Comprehensive Framework for Indian Large Language Models

Samridhi Raj Sinha Rajvee Sheth, Abhishek Upperwal, Mayank Singh · 2025

Read Eka-Eval : A Comprehensive Framework for Indian Large Language Models by Samridhi Raj Sinha Rajvee Sheth, Abhishek Upperwal, Mayank Singh (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-01853.txt.

Introduction

Samridhi Raj Sinha, Rajvee Sheth, Abhishek Upperwal and Mayank Singh introduce Eka-Eval, a framework for evaluating language models in Indian and other low-resource languages, in their 2025 arXiv paper. Existing evaluation suites are hard to adapt across tasks, model types and languages. Their framework unifies the process end to end.

A zero-code web interface and an interactive command line tool make it accessible to varied users. It bundles more than fifty multilingual benchmarks across nine categories, supports local and proprietary models, and uses a modular plug-and-play design. Against five rival suites it at least doubled key usability measures, with the happiest users and reproducible benchmarks. The code is open source.

Catalog record

CREATORSamridhi Raj Sinha Rajvee Sheth, Abhishek Upperwal, Mayank Singh
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE29 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-01853

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Eka-Eval : A Comprehensive Framework for Indian Large Language Models Thanks: More information about Eka initiative is present here: https://eka.soket.ai Samridhi Raj Sinha Rajvee Sheth, Abhishek Upperwal, Mayank Singh Abstract The rapid advancement of Large Language Models (LLMs) has intensified the need for evaluation frameworks that address the requirements of linguistically diverse regions, such as India, and go beyond English-centric benchmarks. We introduce Eka-Eval , a unified evaluation framework that integrates over 35+ benchmarks (including 10 Indic benchmarks) across nine major evaluation categories. The Framework provides broader coverage than existing Indian language evaluation tools, offering 11 core capabilities through a modular architecture, seamless integration with Hugging Face and proprietary models, and plug-and-play usability. As the first end-to-end suite for scalable, multilingual LLM benchmarking, the framework combines extensive benchmarks, modular workflows, and dedicated support for low-resource Indian languages to enable inclusive assessment of LLM capabilities across diverse domains. We conducted extensive comparisons against five existing baselines, demonstrating that Eka-Eval achieves the highest participant ratings in four out of five categories. Demo bit.ly/Eka-Eval Code github.com/lingo-iitgn/eka-eval 1 Introduction Large Language Models (LLMs) have rapidly transformed natural language processing (NLP), enabling impressive generalization across diverse tasks including instruction following, reasoning, summarization, translation, and tool use. With the advent of general-purpose foundation models such as GPT-4 ( Achiam et al. 2023 ) , Claude ( Anthropic 2024 ) , Gemini ( Anil et al. 2023 ) and Llama-3 ( Touvron et al. 2023 ) , the focus of research has increasingly shifted from building task-specific models to systematically evaluating these powerful systems. Evaluation plays a critical role not only in measuring progress but in identifying capabilities, exposing limitations, and informing deployment strategies. Figure 1: Architecture of Eka-Eval : A modular framework combining model inference, benchmark registry, evaluation engine, and result processing with support for caching and distributed inference. In response, several evaluation frameworks have emerged, including HELM ( Liang et al. 2022 ) , EleutherAI Harness ( Gao et al. 2021 ) , lm-eval-harness ( Gao et al. 2021 ) , and OpenCompass ( OpenCompass Contributors 2023 ) . However, existing evaluation frameworks are primarily centered on English or other high-resource languages and offer limited support for low-resource or multilingual settings ( Watts et al. 2024 ) . This lack of inclusivity significantly limits the effectiveness of these tools in linguistically diverse regions like India, which has 22 constitutionally recognized languages encompassing over a billion native speakers. While notable benchmarks like IndicGLUE ( Kakwani et al. 2020 ) , IndicGenBench ( Singh et al. 2024 ) , and MILU ( Verma et al. 2024 ) have addressed some gaps in Indic-language evaluation, they remain siloed efforts and lack integration into unified evaluation workflows — requiring users to perform manual configuration to use them in real-world LLM assessment pipelines. Furthermore, popular frameworks such as FreeEval ( Yu et al. 2024 ) and lm-eval-harness ( Gao et al. 2021 ) require extensive configuration and engineering expertise, limiting their adoption by developers and researchers operating in low-resource environments. These challenges create a need for an actively maintained, community driven, multilingual, and task-diverse evaluation suite. Framework Custom Datasets Custom Models Custom Prompting Long Context Tool Use Distributed Inference Visual Analysis Multilingual Indic Production Optimization Interactive CLI Quantization lm-eval-harness ✓ ✓ ✓ ✓ ✗ ✓ ✗ ✗ ✓ ✓ ✓ OpenCompass ✓ ✓ ✓ ✗ ✗ ✓ ✓ ✓ ✓ ✓ ✓ HELM ✓ ✓ ✓ ✗ ✗ ✓ ✓ ✗ ✗ ✓ ✓ OpenAI Evals ✓ ✓ ✓ ✗ ✗ ✗ ✓ ✗ ✓ ✗ ✗ DeepEval ✓ ✓ ✓ ✓ ✓ ✗ ✓ ✓ ✗ ✓ ✗ FreeEval ✓ ✓ ✓ ✗ ✗ ✓ ✓ ✗ ✓ ✓ ✓ indic-eval ✗ ✓ ✗ ✗ ✗ ✓ ✗ ✓ ✓ ✓ ✗ Eka-Eval ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ Table 1: Comparison of LLM Evaluation Frameworks across eleven key capabilities, showing the comprehensive coverage of Eka-Eval . To address these limitations, we introduce Eka-Eval , a unified, extensible, and ready-to-use evaluation framework for LLMs, integrating over 35 benchmarks spanning both global (English) and Indic-language tasks. The overall architecture, presented in Figure 1 , is designed to be modular, compatible with Hugging Face and proprietary models, it offers plug-and-play usability with minimal configuration overhead. To contextualize Eka-Eval’s capabilities, Table 1 compares leading LLM evaluation frameworks across eleven key capabilities, demonstrating its more comprehensive feature coverage relative to existing frameworks. Eka-Eval covers nine major evaluation categories: (i) Code Generation and programming; (ii) Mathematics and logical reasoning; (iii) Reading comprehension; (iv) Commonsense reasoning; (v) World knowledge; (vi) Long-context understanding; (vii) General reasoning and knowledge; (viii) Tool use and API reasoning; and (viiii) Indic-specific NLP benchmarks including ARC-C ( Clark et al. 2018 ) , BoolQ ( Clark et al. 2019 ) , GSM8K ( Cobbe et al. 2021 ) , Math ( Hendrycks et al. 2021 ) , HellaSwag ( Zellers et al. 2019 ) , MMLU ( Hendrycks et al. 2020a ) , SQuAD ( Rajpurkar et al. 2018 ) , APIBench ( Patil et al. 2024 ) , IndicGenBench ( Singh et al. 2024 ) . Our key contributions are: • We propose Eka-Eval , a unified and modular evaluation framework that integrates over 35 benchmarks across nine major evaluation categories, providing comprehensive assessment of both global and Indic-language LLM capabilities. • We provide comprehensive support for most Indian languages and conduct thorough human evaluation across five existing frameworks on four benchmark tasks, demonstrating Eka-Eval ’s effectiveness and reliability. 2 Related Work The evaluation of LLMs has evolved from task-specific benchmarks to comprehensive, modular frameworks that support diverse capabilities and deployment settings. This section categorizes prior work into three major areas: general-purpose evaluation frameworks, specialized capability benchmarks, and multilingual/Indic evaluations. We conclude by highlighting how Eka-Eval integrates and advances these directions. General-Purpose LLM Evaluation Frameworks: Early benchmark suites such as GLUE ( Wang et al. 2018 ) and SuperGLUE ( Wang et al. 2019 ) laid the groundwork for multi-task evaluation, but were primarily English-centric and biased toward high-resource languages. Frameworks like HELM ( Liang et al. 2022 ) expanded evaluation to multidimensional axes—accuracy, calibration, robustness, fairness, and efficiency—applied uniformly across 30+ models. Similarly, BIG-Bench ( Srivastava et al. 2022 ) curated over 200 tasks via crowdsourced contributions, highlighting model generality but with limited support for multilingualism or real-world deployment constraints. In parallel, tooling infrastructure also emerged. lm-eval-harness ( Gao et al. 2021 ) provided reproducible few-shot evaluation pipelines, while OpenAI’s Evals ( OpenAI Contributors 2023 ) offered YAML-based structured evaluation for internal alignment research. Recent frameworks emphasize modularity, extensibility, and accessibility. For instance, OpenCompass OpenCompass Contributors 2023 integrates distributed inference and real-time leaderboards, though it lacks Indic language support. FreeEval Yu et al. 2024 introduces meta-evaluation and contamination detection, but lacks robust multilingual and Indic language support. DeepEval DeepEval Contributors 2024 supports long-context and tool-use tasks but lacks flexible multilingual and low-resource customization, and its default metrics are often too rigid. Specialized Capability Benchmarks: To evaluate emerging LLM capabilities, task-specific frameworks and datasets have been proposed. Tool-use and agentic behavior are assessed in ToolBench ( Qin et al. 2023 ) and API-Bank ( Li et al. 2023 ) respectively, which evaluate model interactions with

Read the complete text