Can Large Language Models Generate Effective Datasets for Emotion Recognition in Conversations?

Burak Can Kaplan, Hugo Cesar De Castro Carneiro, Stefan Wermter, Burak Can Kaplan 1 , Hugo Cesar De Castro Carneiro 1 , and Stefan Wermter 1 · 2025

Read Can Large Language Models Generate Effective Datasets for Emotion Recognition in Conversations? by Burak Can Kaplan, Hugo Cesar De Castro Carneiro, Stefan Wermter, Burak Can Kaplan 1 , Hugo Cesar De Castro Carneiro 1 , and Stefan Wermter 1 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-05474.txt.

Introduction

Burak Can Kaplan and co-authors test whether a small language model can create training data for emotion recognition in conversations, in work from 2025. Systems that track how feelings shift during a chat need large labelled datasets. Such data is scarce, biased by its sources, and hard to label because feelings are subjective.

The team used a compact, low-cost model to build six new datasets, two tailored to each of three standard benchmarks. Classifiers trained with the added data grew more robust and scored clearly better on the benchmarks. The team also studied how skewed label mixes affect results. The work suggests cheap synthetic data can stretch scarce emotion datasets, a useful route for labs with small budgets.

Catalog record

CREATORBurak Can Kaplan, Hugo Cesar De Castro Carneiro, Stefan Wermter, Burak Can Kaplan 1 , Hugo Cesar De Castro Carneiro 1 , and Stefan Wermter 1
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE32 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-05474

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Can Large Language Models Generate Effective Datasets for Emotion Recognition in Conversations? Burak Can Kaplan, Hugo Cesar De Castro Carneiro, Stefan Wermter, Burak Can Kaplan 1 , Hugo Cesar De Castro Carneiro 1 , and Stefan Wermter 1 Abstract Emotion recognition in conversations (ERC) focuses on identifying emotion shifts within interactions, representing a significant step toward advancing machine intelligence. However, ERC data remains scarce, and existing datasets face numerous challenges due to their highly biased sources and the inherent subjectivity of soft labels. Even though Large Language Models (LLMs) have demonstrated their quality in many affective tasks, they are typically expensive to train, and their application to ERC tasks—particularly in data generation—remains limited. To address these challenges, we employ a small, resource-efficient, and general-purpose LLM to synthesize ERC datasets with diverse properties, supplementing the three most widely used ERC benchmarks. We generate six novel datasets, with two tailored to enhance each benchmark. We evaluate the utility of these datasets to (1) supplement existing datasets for ERC classification, and (2) analyze the effects of label imbalance in ERC. Our experimental results indicate that ERC classifier models trained on the generated datasets exhibit strong robustness and consistently achieve statistically significant performance improvements on existing ERC benchmarks. I Introduction Emotion recognition in conversations (ERC) is a relatively new field of study that focuses on identifying and understanding human emotions expressed during interactions [ 1 ] . Its primary goal is to detect emotion shifts within dialogues, a capability that has become increasingly important with the rise of social robotics and applications requiring emotionally intelligent systems [ 2 ] . Large Language Models (LLMs) have demonstrated substantial improvements in various natural language processing (NLP) tasks, including affective computing [ 3 ] , and hold potential as effective tools for generating ERC data. Despite their success, most evaluations of LLMs in affective tasks have been conducted with API-based models, which are expensive, or top-performing local models requiring significant computational resources [ 4 , 5 ] . Exploring the capabilities of small and general-purpose LLMs in ERC tasks thus emerges as a promising and cost-efficient alternative. A critical challenge in ERC lies in the scarcity of high-quality, diverse datasets. Most existing datasets are derived from biased sources such as scripted TV shows or social media, which often feature imbalanced label distributions [ 6 , 7 ] . Moreover, crafting such datasets is costly and time-consuming due to participant recruitment, ethical concerns, unbiased dialogue construction, and the difficulty of accurate and consistent labeling. Annotating emotional data often involves subjective interpretations, with annotators frequently providing inconsistent labels for the same utterance. Typically, a majority vote is used to select a label when there is disagreement, but this process is limited by the small number of annotators (usually 3–5), leading to reliability issues [ 7 , 8 ] . Additionally, as highlighted in Tab. I , existing ERC datasets vary significantly in their emotion label sets, speaker numbers, and languages, making it difficult to combine them effectively for transfer learning. Furthermore, the inconsistencies in emotion categories across datasets limit their interoperability, leading some studies to rely on mappings based on psychological studies to align these different label sets [ 5 ] . However, such mappings are often rough approximations, raising concerns about their accuracy and applicability. Rooted in the concept that only the entity expressing a particular affective state can fully recognize it [ 9 ] , we hypothesize that by having the LLM generate both utterances and their corresponding emotion labels simultaneously, we can address these issues and significantly improve dataset consistency. Tab. I highlights the characteristics of popular ERC datasets and underscores their limitations. TABLE I: ERC datasets Dataset Source Speaker Emotions Language MELD [ 6 ] TV Multiple 7 English IEMOCAP [ 8 ] Scripted 2 6 English EmoryNLP [ 7 ] TV 6 7 English DailyDialog [ 10 ] Scripted 2 7 English CPED [ 11 ] TV Multiple 13 Chinese EC [ 12 ] Twitter 3 4 English KDEmor [ 13 ] TV Multiple 3 Korean To tackle these limitations, we propose leveraging a small, general-purpose LLM to synthesize new ERC datasets. By generating both dialogue lines and their corresponding emotion labels in a single step, we aim to improve the consistency and reliability of ERC data while avoiding the costs and complexities associated with traditional data collection and annotation methods. To ensure comparability with existing benchmarks, we generate six new datasets, with two corresponding datasets for each of the three widely used ERC benchmarks. This alignment allows us to systematically evaluate the potential of LLM-generated datasets to supplement existing resources, mitigate label imbalance, and enhance ERC model performance. Fig. 1 shows an example of a dialogue generated by our LLM. (a) Fig. 1: Example of a generated dialogue, featuring multiple speakers, utterances, and corresponding emotion labels. Our contributions are as follows: Firstly, we demonstrate the capability of a small LLM to generate consistent, multi-party ERC datasets suitable for training ERC models. Secondly, we propose a methodology to evaluate the quality and validity of synthesized ERC data, focusing on their utility in improving model robustness and performance. Thirdly, we assess the effects of different label distributions in ERC datasets and analyze how these imbalances affect existing benchmarks. The prompts and parameters used for data generation are shared in the paper, ensuring reproducibility with local LLMs. Additionally, the code repository will be made available to facilitate further advancements in LLM-based ERC dataset creation. The remainder of the paper is organized as follows: Section II reviews existing ERC datasets and related research on LLMs for affective computing, dataset enhancement, and synthetic data generation. Section III details the LLM setup, the dataset synthesis process, prompt engineering, and all parameters used, providing all necessary information for transparency. Section IV presents experimental evaluations of the generated datasets, and Section V concludes with insights and directions for future research. II Related Work II-A Existing Datasets In this study, we focus on the three most popular ERC datasets in Papers With Code 1 1 1 https://paperswithcode.com/task/emotion-recognition-in-conversation : MELD [ 6 ] , EmoryNLP [ 7 ] , and IEMOCAP [ 8 ] . These datasets are chosen to assess the performance of ERC models on our generated datasets, as well as on these existing datasets. MELD is a dataset constructed by extracting lines from the “Friends” TV series. It encompasses 7 emotions: Neutral, Disgust, Anger, Sadness, Fear, Joy, and Surprise. There is a high imbalance among its labels, with Neutral being the most common. On the other hand, Disgust and Fear are notably rare. In some studies, authors perform classification task without either one or both of these labels. According to Papers With Code, weighted F1 classification results for MELD usually lie within the range from 60% to 70%. The EmoryNLP dataset is also constructed from the lines of “Friends” TV series, but it employs different emotion labels than MELD, which are: Sad, Mad, Scared, Powerful, Peaceful, Joyful, and Neutral. For the annotation process, 4 annotators participated, and only 6.17% of the annotations correspond to labels in which there was an unanimous agreement among the annotators. In 9.39% of the annotations, each annotator labeled the same data with different label. Majority

Read the complete text