Eye of Judgement: Dissecting the Evaluation of Russian-speaking LLMs with POLLUX

Nikita Martynov, Anastasia Mordasheva, Gorbetskiy Dmitriy, Danil Astafurov, Ulyana Isaeva, Elina Basyrova, Sergey Skachkov, Berestova Victoria, Nikolay Ivanov, Valeriia Zanina, Alena Fenogenova · 2025

Read Eye of Judgement: Dissecting the Evaluation of Russian-speaking LLMs with POLLUX by Nikita Martynov, Anastasia Mordasheva, Gorbetskiy Dmitriy, Danil Astafurov, Ulyana Isaeva, Elina Basyrova, Sergey Skachkov, Berestova Victoria, Nikolay Ivanov, Valeriia Zanina, Alena Fenogenova (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-24616.txt.

Introduction

Nikita Martynov, Anastasia Mordasheva, Dmitriy Gorbetskiy, and a team of co-authors present POLLUX, a 2025 benchmark for judging language models that write in Russian. Most graders lean on slow, costly human side-by-side reads. POLLUX swaps that for clear criteria: for each task type, judge models score answers against a fine rubric and must state their reasons.

The set holds 2,100 prompts written from scratch by experts, spread over thirty-five task types from code to creative writing, each tagged by difficulty. Two judge models, in small and large sizes, are released with it. The authors say criteria-driven judging makes scores easier to read, cheaper to run, and fit to guide model training in place of human panels.

Catalog record

CREATORNikita Martynov, Anastasia Mordasheva, Gorbetskiy Dmitriy, Danil Astafurov, Ulyana Isaeva, Elina Basyrova, Sergey Skachkov, Berestova Victoria, Nikolay Ivanov, Valeriia Zanina, Alena Fenogenova
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE73 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-24616

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Eye of Judgement: Dissecting the Evaluation of Russian-speaking LLMs with POLLUX Nikita Martynov, Anastasia Mordasheva, Gorbetskiy Dmitriy, Danil Astafurov, Ulyana Isaeva, Elina Basyrova, Sergey Skachkov, Berestova Victoria, Nikolay Ivanov, Valeriia Zanina, Alena Fenogenova Abstract Evaluating open-ended generation remains a highly non-trivial challenge, as responses vary in style, quality, and correctness, making reliable assessment difficult. To address this, we introduce POLLUX , an open-source framework for evaluating Russian-speaking large language models (LLMs). Its novelty lies in a criteria-based methodology that improves interpretability by combining a structured benchmark with a family of LLM-as-a-Judge evaluators. For each task type, we define explicit criteria and a scoring protocol in which models not only rate responses but also justify their judgments, offering a transparent alternative to resource-intensive human comparisons. The benchmark spans 35 task types across domains such as code generation, creative writing, and assistant-style interactions, supported by 2,115 expert-authored prompts stratified by difficulty. In addition, we release specialized evaluators (7B and 32B) trained for fine-grained assessment of generative outputs. By uniting a comprehensive taxonomy with automated judges, POLLUX provides scalable and interpretable evaluation tools that move beyond the costs and inconsistencies of human annotation. 1 Introduction Evaluating the open-ended outputs of large language models (LLMs) remains a significant challenge. While the emerging LLM-as-a-Judge paradigm offers a promising, scalable, and human-aligned solution, its current application is critically limited. These approaches are not only predominantly focused on English but have also failed to resolve the fundamental issue of interpretable assessment even within that language. The effectiveness and interpretability of this paradigm for other languages, such as Russian, thus constitute a severe and unexamined problem. To address these dual limitations, we propose POLLUX , a framework and comprehensive methodology for evaluating the generative capabilities of LLMs that provides a scalable yet interpretable solution. Benchmark features a fine-grained hierarchical taxonomy of 35 generative task types inferred from open LLM usage logs spanning diverse domains, including code generation, creative writing, and practical assistant applications, with a total of 2,115 manually crafted prompts. Each task is annotated by an explicitly formulated difficulty level (easy/medium/hard) and constructed entirely from scratch by domain experts to ensure high-quality, unbiased evaluation data. We define a detailed set of criteria for each task type. We also develop a transparent scoring protocol where models assess responses and generate open-ended justifications for their ratings. Moreover, we release a family of LLM-as-a-Judge models (7B and 32B parameters) trained to perform criteria-aligned assessments based on the proposed methodology of generative outputs, both with a score and textual feedback. Our approach aims at a criteria-driven, reproducible evaluation framework, reducing reliance on costly and less consistent human side-by-side comparisons. An overview of POLLUX is presented in Figure 1 . The key contributions of this work include: • A general methodology for LLM evaluation, comprising: – A hierarchical taxonomy of generative tasks , categorized by complexity and domain. – A fine-grained taxonomy of criteria for systematic evaluation. • An open benchmark with prompts and annotations verified by experts. • The release of LLM-as-a-Judge evaluators (7B and 32B) for automated assessment. The benchmark, code, and models are available at the provided links 1 1 1 Benchmark , codebase , models . Figure 1: POLLUX overview and rounded statistics before filtering: benchmark characteristics, including tasks and criteria, information about the experts involved in creating the data, the overflow of LLM-as-a-Judge models, and the synthetic data used for them. 2 Related Work The evaluation of LLMs employs distinct benchmarking paradigms. Static benchmarks (like BIG-bench Srivastava et al. (2023) , HELM Bommasani et al. (2023) , MMLU Hendrycks et al. (2021) , HumanEval Chen et al. (2021) ) target expert knowledge, reasoning, and coding. Generative benchmarks (MT-Bench Zheng et al. (2023) ) utilize open-ended prompts, which are scored by humans or LLMs-as-judges. Recent efforts (WildBench Lin et al. (2025) , Preference Bench Kim et al. (2024) , Auto-J Eval Li et al. (2023) ) stress realism and scalable automation, while Chatbot Arena Chiang et al. (2024) enables crowd-based comparisons but faces criticism on fairness Singh et al. (2025) . In the Russian context, benchmarks such as Russian SuperGLUE Shavrina et al. (2020) and TAPE Taktasheva et al. (2022) remain largely static and classification-oriented. MERA Fenogenova et al. (2024) introduced a broader generative evaluation, yet most others (LIBRA Churin et al. (2024) , RuBLiMP Taktasheva et al. (2024) , RuBia Grigoreva et al. (2024) ) still emphasize closed-answer tasks. REPA Pugachev et al. (2025) adds error-type annotations but is model-specific and error-based. Overall, open-ended, human-centered generative benchmarks for Russian-speaking LLMs are still lacking, leaving a major gap in evaluation. 3 The POLLUX Generative Benchmark Our objective is to emulate the full spectrum of generative, open-ended tasks that can be posed to an AI assistant, and to establish a framework for evaluating the resulting outputs using interpretable criteria, rather than relying exclusively on surface-level assessments, such as those employed in Arena-like A/B testing approaches. Thus, we propose the POLLUX benchmark, which provides a quantitative and qualitative evaluation of LLMs across tasks-criteria taxonomy and expert-annotated data. The methodology is based on: 1) the generative tasks taxonomy , covering 35 categories derived from real LLM interactions and further expanded by functional styles, genres, and three complexity levels; and 2) the criteria taxonomy , comprising domain-, task-, fine-grained, subjective dimensions, each equipped with dedicated scoring rubrics. The benchmark encompasses 2,115 unique instructions across 5 functional styles. 3.1 The Generative Tasks Taxonomy To obtain a hierarchy of generative tasks, a two-stage procedure was applied. The first stage involves bottom-up category mining using instruction clustering, and the second stage marks the point at which the specialized knowledge of domain experts is applied. Organizing use cases into task taxonomy We used the WildChat-1M dataset Zhao et al. (2024) as a source of user-LLM interactions. From its 87K Russian sessions (270K prompts), we applied deduplication using the rapidfuzz WRatio function 2 2 2 rapidfuzz WRatio (threshold 95), removed toxic content per original annotations, excluded prompts over 200 words, and deduplicated via hash of the first and last 200 characters. The final corpus contains 181K user prompts. Then clustering in the form of BERTopic 3 3 3 MaartenGr/BERTopic Grootendorst (2022) pipeline was executed on the instructions embeddings concatenated with embeddings of definitions of corresponding tasks (e.g. debug code or paraphrase text ), which were generated with Llama-3-8B-Instruct 4 4 4 meta-llama/Meta-Llama-3-8B-Instruct Grattafiori et al. (2024) , yielding 4,500 clusters. Three annotators (agreement: 0.81) labeled centroids concisely. Definitions appearing at least 2 times (43.5% of centroids) were removed. The remaining 81 labels were merged based on shared skills, resulting in 35 final task types. Figure 2: The POLLUX generative taxonomy of tasks. The labeled figures highlighted in bright colors are major 35 task groups. Each task group is annotated with corresponding expert panels. The sections within task types schematically illustrate the depth of decomposition within each taxon. Expert refining We adopted a functional–stylistic classification Kozhina et al. (2011) comprising five core textual styles (e.g., Literature, Science) as the superstructure of the taxonomy. Ten expert panels — comprising specialists in relevant task types — identified genre-specific tasks, resulting in a hierarchical taxonomy of 152 fine-grained tasks (grouped into 35 major categories) spanning 15 literary movements, 17 canonical authors,

Read the complete text