REPA : Russian Error Types Annotation for Evaluating Text Generation and Judgment Capabilities

Alexander Pugachev, Alena Fenogenova, Vladislav Mikhailov, Ekaterina Artemova · 2025

Read REPA : Russian Error Types Annotation for Evaluating Text Generation and Judgment Capabilities by Alexander Pugachev, Alena Fenogenova, Vladislav Mikhailov, Ekaterina Artemova (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-13102.txt.

Introduction

Alexander Pugachev and co-authors build a Russian test bed for language model errors in a 2025 paper. Using models as judges of other models is fashionable, but the practice was validated mostly in English. Their REPA data set holds one thousand user queries and two thousand model answers, hand labeled across ten error types plus overall preference.

Six generators were ranked by human taste under three scoring systems. Eight model judges then tried the same job, in plain and example guided modes. Judges in Russian lagged their English form, with position and length biases in play. Still, machine rankings partly tracked human ones, suggesting the approach can improve with language specific care.

Catalog record

CREATORAlexander Pugachev, Alena Fenogenova, Vladislav Mikhailov, Ekaterina Artemova
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE47 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-13102

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

REPA : Russian Error Types Annotation for Evaluating Text Generation and Judgment Capabilities Alexander Pugachev, Alena Fenogenova, Vladislav Mikhailov, Ekaterina Artemova Abstract Recent advances in large language models (LLMs) have introduced the novel paradigm of using LLMs as judges, where an LLM evaluates and scores the outputs of another LLM, which often correlates highly with human preferences. However, the use of LLM-as-a-judge has been primarily studied in English. In this paper, we evaluate this framework in Russian by introducing the Russian Error tyPes Annotation dataset ( REPA 1 1 1 Repa ( ru ) — turnip ( en ). Logo source: flaticon.com ), a dataset of 1k user queries and 2k LLM-generated responses. Human annotators labeled each response pair expressing their preferences across ten specific error types, as well as selecting an overall preference. We rank six generative LLMs across the error types using three rating systems based on human preferences. We also evaluate responses using eight LLM judges in zero-shot and few-shot settings. We describe the results of analyzing the judges and position and length biases. Our findings reveal a notable gap between LLM judge performance in Russian and English. However, rankings based on human and LLM preferences show partial alignment, suggesting that while current LLM judges struggle with fine-grained evaluation in Russian, there is potential for improvement. 1 Introduction Large language models (LLMs) have gained significant attention due to their capabilities to assist expert and non-expert users in a wide range of writing tasks. However, reliable evaluation of such LLMs remains an open question, especially in the context of non-English languages. Recent research has explored methods to automatically evaluate the LLMs using “judge” models that perform pairwise model comparisons and highly correlate with human preferences Zheng et al. (2023) ; Lambert et al. (2024) . While the LLM-as-a-judge approach mitigates the cost of collecting human-based preference data and performing the evaluation at scale, it overlooks the need for a more fine-grained evaluation with respect to quality criteria relevant to the end user. This paper extends the LLM-as-a-judge approach to a fine-grained pairwise comparison that relies on common issues in language generation well-studied in earlier research Mao et al. (2023) ; Hackl et al. (2023) . We introduce the Russian Error tyPes Annotation dataset ( REPA ), which consists of 1k user queries spanning various cases, along with responses from six open-source instruction-finetuned Russian LLMs. REPA comprises fine-grained pairwise human preferences across ten error types, ranging from request following and factuality to the overall impression. We conduct pairwise comparisons on human-annotated data using three rating systems and evaluate five open-source and three proprietary LLMs as judges in several scenarios, including position and length biases. Finally, we analyze how rankings from the best-performing judge align with human annotations across all error types. Our key findings reveal partial alignment between rankings based on human and LLM preferences, suggesting that while LLM judges do not fully replicate human judgment, they can still serve as valuable evaluators. We find that LLaMA-2-based Touvron et al. (2023) models outperform other models in text generation and identify a noticeable performance gap between LLM judges in Russian and English. Our main contributions are: (i) REPA , one of the first human-labeled non-English benchmarks for evaluating text generation based on fine-grained criteria and overall preference; (ii) assessing the performance of eight LLM judges and compare their ranking scores to human judgments; (iii) releasing REPA , annotation and experimental materials. 2 2 2 hf.co/datasets/RussianNLP/repa Category # Queries Avg. # tokens in query Avg # tokens in response Example Generation 213 14.49 \pm 9.61 193.31 \pm 137.35 Rasskazhi istoriyu o tom, kak vazhno byt’ dobrym i otzyvchivym. Tell a story about the importance of being kind and compassionate. Open QA 205 9.79 \pm 6.36 197.30 \pm 150.83 Opredelite i ob’yasnite znachenie muzykal’nogo termina allegro. Define and explain the meaning of the musical term allegro. Brainstorm 163 13.09 \pm 9.45 209.12 \pm 137.91 Sostav’ spisok preimushchestv ispol’zovaniya solnechnykh paneley. Make a list of the advantages of using solar panels. Classify 120 30.63 \pm 18.98 158.46 \pm 136.50 Opredelite: ironiya, sarkazm ili yumor – Ty vsegda takoy umnyy, kogda spish’! Determine: irony, sarcasm, or humor – You’re so smart when you sleep!. Rewrite 104 28.75 \pm 29.80 161.45 \pm 143.17 Perefraziruy s sinonimami: Vladelets magazina khochet bol’she pribyli i rosta! Rephrase using synonyms: The store owner wants more profit and growth. Extract 59 41.20 \pm 30.20 162.31 \pm 131.82 Razberite ukazannuyu datu na sootvetstvuyushchiye komponenty. <…> Break down the given date into its corresponding components. <…> Closed QA 49 91.16 \pm 67.80 145.48 \pm 152.08 O kakom vazhnom sobytii idet rech’ v tekste? <…> What important event is being discussed in the text? <…> Chat 46 41.36 \pm 20.84 199.53 \pm 203.39 Predstav’ chto ty otvechayesh’ pyatiletnemu rebenku. Rasskazhi pro muzyku Shopena. Imagine you’re answering a five-year-old. Tell them about Chopin’s music. Summarize 44 58.54 \pm 32.79 177.72 \pm 124.77 Naydite glavnuyu ideyu sleduyushchego teksta. Find the main idea of the following text. Overall Queries 1003 25.19 \pm 30.12 184.66 \pm 145.72 Data Sources ru_instruct_gpt4: 517 (51.5%), Veles–2.5 337 (33.6%), Tagengo: 121 (12.1%), Aya: 24 (2.4%), Chatbot Arena Conversations: 4 (0.4%) Table 1: REPA dataset statistics and examples. Data sources distribution is provided for the entire dataset. 2 REPA Figure 1: REPA design: (a) collecting user queries from public datasets, (b) generating LLM responses, (c) human annotation of error types. Figure 1 outlines our process for creating REPA : collecting Russian user queries from public sources (§ 2.1 ); generating responses using six LLMs (§ 2.2 ); human annotation based on ten error types (§ 2.3 ). REPA ’s general statistics are in § 2.4 . 2.1 Query Collection Error Type Example Request Following User query: Ob"yasni, pochemu lesa vazhny dlya zhizni na Zemle. LLM response: Original zapisi i kommentarii na LiveInternet. User query: Explain why forests are important for life on Earth. LLM response: The original entry and comments on LiveInternet Factuality Lazan’ya - eto tradicionnoe blyudo russkoj kuhni , kotoroe gotovitsya iz grechnevoj muki , nachinennoj myasnym farshem, ovoshchami i fruktami. Lasagna is a traditional dish of Russian cuisine , which is made from buckwheat flour , stuffed with minced meat, vegetables and fruits. Repetition Vy otkroete dveri i daete klyuch Dzheku. Dzheku daete klyuch. Dzheku daete klyuch. Dzheku daete klyuch. You will open the doors and give the key to Jack. You give Jack the key. You give Jack the key. You give Jack the key. Code-Switching Tokio - eto stolica YAponii i one of the most populous cities in the world . Tokyo is the capital of Japan and one of the most populous cities in the world. Relevance Iz Moskvy v Habarovsk mozhno dobrat’sya na samolete za 8 chasov. Chto obshchego u karandasha i akvariuma? Nichego. You can get from Moscow to Khabarovsk by plane in 8 hours. What do a pencil and an aquarium have in common? Nothing. Harmfulness Ty beznadezhnyj chelovek. You’re a hopeless person. Fluency My kupili novyj televizor, no on ne rabotaet, potomu chto on slomannyj . We bought a new TV, but it doesn’t work because it’s breaking . Contradiction Da, v glavnoj roli byl Morgan Frimen. Net, v glavnoj roli byl

Read the complete text