EmoBench-UA : A Benchmark Dataset for Emotion Detection in Ukrainian
Daryna Dementieva, Nikolay Babakov, Alexander Fraser · 2025
Read EmoBench-UA : A Benchmark Dataset for Emotion Detection in Ukrainian by Daryna Dementieva, Nikolay Babakov, Alexander Fraser (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-23297.txt.
Introduction
Daryna Dementieva, Nikolay Babakov, and Alexander Fraser built the first public test set for emotion spotting in Ukrainian text in a 2025 paper. Ukrainian language tools have grown in many areas, but sorting text by feeling had no shared yardstick. Their set fills that gap with texts marked for the emotion they carry.
The marking scheme adapts rules honed on English sets. Crowd workers marked the texts through an online platform with quality checks in place. The team then graded a range of methods on the set, from word-based baselines to data translated from English. The set gives future tools a fair way to prove their skill in Ukrainian.
Catalog record
| CREATOR | Daryna Dementieva, Nikolay Babakov, Alexander Fraser |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 49 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-23297
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
EmoBench-UA : A Benchmark Dataset for Emotion Detection in Ukrainian Daryna Dementieva, Nikolay Babakov, Alexander Fraser Abstract While Ukrainian NLP has seen progress in many texts processing tasks, emotion classification remains an underexplored area with no publicly available benchmark to date. In this work, we introduce EmoBench-UA , the first annotated dataset for emotion detection in Ukrainian texts. Our annotation schema is adapted from the previous English-centric works on emotion detection Mohammad et al. 2018 ; Mohammad 2022 guidelines. The dataset was created through crowdsourcing using the Toloka.ai platform ensuring high-quality of the annotation process. Then, we evaluate a range of approaches on the collected dataset, starting from linguistic-based baselines, synthetic data translated from English, to large language models (LLMs). Our findings highlight the challenges of emotion classification in non-mainstream languages like Ukrainian and emphasize the need for further development of Ukrainian-specific models and training resources. 1 Introduction Recent trends in natural language processing indicate a shift from predominantly monolingual English-centric approaches toward more inclusive multilingual solutions that support less-resourced and non-mainstream languages. Although cross-lingual transfer techniques—such as Adapter modules Pfeiffer et al. 2020 or translation from resource-rich languages Kumar et al. 2023 —have shown promise, the development of high-quality, language-specific datasets remains essential for achieving robust and culturally accurate performance in these settings. Figure 1: EmoBench-UA is a benchmark of basic emotions—Joy, Anger, Fear, Disgust, Surprise, Sadness, or None—detection in Ukrainian texts. For the Ukrainian language, significant progress has been made in the development of resources for various stylistic classification tasks, such as sentiment analysis Zalutska et al. 2023 and toxicity detection Dementieva et al. 2024 . However, to the best of our knowledge, no publicly available dataset has yet addressed the task of emotion classification. In this work, we aim to fill this gap through the following contributions: Figure 2: EmoBench-UA Annotation Pipeline: we split the annotation into two tasks to improve annotator focus, and several quality control measures were applied to ensure the high quality of the collected data. • We design a robust crowdsourcing annotation pipeline for emotion annotation in Ukrainian texts, leveraging the Toloka.ai platform and incorporating quality control mechanisms to ensure high-quality annotations; • Using this pipeline, we collect EmoBench-UA , the first manually annotated benchmark dataset for emotion detection in Ukrainian; • We evaluate a range of classification approaches on the dataset—including linguistic-based baselines, Transformer-based encoders, translation into English, and prompting large language models (LLMs)—to assess task difficulty and provide a comprehensive performance analysis. We release all the instructions, data, and top performing model fully online for public usage. 1 1 1 Dataset Binary: https://huggingface.co/datasets/ukr-detect/ukr-emotions-binary Dataset Intensity: https://huggingface.co/datasets/ukr-detect/ukr-emotions-intensity Top Classifier: https://huggingface.co/ukr-detect/ukr-emotions-classifier 2 Related Work Emotion Detection Datasets and Models As for many NLP tasks, various datasets, lexicon, and approaches in the first order were created for English emotions classification Mohammad et al. 2018 . Then, it was also extended to other popular languages like Spanish, German, and Arabic Plaza del Arco et al. 2020 ; Chatterjee et al. 2019 ; Kumar et al. 2022 and then for some not so mainstream languages like Finish Öhman et al. 2020 . Given the challenges associated with collecting fully annotated emotion datasets across languages, a multilingual emotional lexicon Mohammad 2023 which covers 100 languages was proposed by translating the original English resources, offering a practical first step toward facilitating emotion detection in lower-resource scenarios. At the same time, the importance of developing robust NLP systems for emotion analysis and detection is well recognized Kusal et al. 2023 , especially in socially impactful domains such as customer service, healthcare, and support for minority communities. However, extending emotion detection capabilities uniformly across multiple languages remains a persistent challenge De Bruyne 2023 . For English and several other languages, a variety of classification methods have been explored, ranging from BiLSTM and BERT-based models Al-Omari et al. 2020 ; De Bruyne et al. 2022 to more advanced architectures such as XLM-RoBERTa Conneau et al. 2020 , E5 Wang et al. 2024a , and multilingual LLMs like BLOOMz Wang et al. 2024b . Ukrainian Texts Classification Although the availability of training data for classification tasks in Ukrainian remains limited, the research community has made notable strides in mayny NLP tasks. For example, UberText 2.0 Chaplynskyi 2023 provides resources for NER tasks, legal document classification, and a wide range of textual sources including news, Wikipedia, and fiction. In addition, the OPUS corpus Tiedemann 2012 offers parallel Ukrainian data for cross-lingual applications. Recently, the Spivavtor dataset Saini et al. 2024 has also been introduced to facilitate instruction-tuning of Ukrainian-focused large language models. For related classification tasks, resources for sentiment analysis Zalutska et al. 2023 and toxicity detection Dementieva et al. 2024 have already been developed for Ukrainian. Additionally, in the domain of abusive language, a bullying detection system for Ukrainian was introduced but based on translated English data Oliinyk and Matviichuk 2023 . Dementieva et al. 2025 explored various cross-lingual knowledge transfer methods for Ukrainian texts classification, yet emphasized the continued importance of authentic, manually annotated Ukrainian data. 3 EmoBench-UA Collection Here, we present the design of the crowdsourcing collection pipeline, detailing the task setup, annotation guidelines, interface design, and the applied quality control procedures used to obtain EmoBench-UA . The overall schema of the pipeline is presented in Figure 2 . 3.1 Emotions Classification Objective In this work, we define emotion recognition as the task of identifying perceived emotions—that is, the emotion that the majority of people would attribute to the speaker based on a given sentence or short text snippet Mohammad 2022 . We adopt the set of basic emotions proposed by Ekman et al. 1999 , which includes Joy , Fear , Anger , Sadness , Disgust , and Surprise . A single text instance may convey multiple emotions simultaneously creating the multi-label classification task. If a text does not express any of the listed emotions, then we assign it the label None . 3.2 Data Selection for Annotation As the source data, we selected the publicly available Ukrainian tweets corpus Bobrovnyk 2019 . Given that social media posts are often rich in emotionally charged content, this corpus serves as a suitable foundation for our annotation task. Since the original collection consists of several hundred thousand tweets, we applied a multi-stage filtering process to both increase the likelihood of emotional content and ensure the feasibility of accurate annotation: Length First, we applied a length-based filter, discarding texts that were too short ( N words < 5), as such samples often consist of hashtags or other non-informative tokens. Similarly, overly long texts ( N words \geq 50) were excluded, as longer sequences tend to obscure the central meaning and make it challenging to accurately identify the expressed emotions. Toxicity While toxic texts can carry quite strong emotions, to ensure annotators well-being and general appropriateness of our corpus, we filtered out too toxic instances using opensourced toxicity classifier Dementieva et al. 2024 . 2 2 2 https://huggingface.co/ukr-detect/ukr-toxicity-classifier Figure 3: Annotation Interface illustration translated into English. Emotional Texts Pre-selection To avoid an excessive imbalance toward emotionless texts, we performed a pre-selection step aimed at identifying texts likely to express emotions. Specifically, we applied the English emotion classifier DistillRoBERTa-Emo-EN 3 3 3 https://huggingface.co/michellejieli/emotion_text_classifier on