A Hybrid Framework for Song Lyric Annotation Based on Human-LLM Alignment

Rashini Liyanarachchi, Frank Tran, Md Mahmudul Hasan, Aditya Joshi, Erik Meijering · 2026

Read A Hybrid Framework for Song Lyric Annotation Based on Human-LLM Alignment by Rashini Liyanarachchi, Frank Tran, Md Mahmudul Hasan, Aditya Joshi, Erik Meijering (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-29273.txt.

Introduction

Liyanarachchi and four colleagues study how people and language models label the emotions in song lyrics. Lyrics need not match a song's overall emotion, so labelling them is subjective and little studied. The team built a new sentence-level dataset of lyrics and compared human labels with model labels to see where the two align.

Their observations highlight how subjective the task is and where annotators diverge. On that basis they present a hybrid framework that predicts where human and model labels are likely to misalign. Cases flagged as risky can then get more human attention, while clear cases lean on the model. The aim is to spend scarce human effort where disagreement is most likely, improving lyric emotion annotation without labelling everything twice. The framework and dataset are released openly for other researchers.

Catalog record

CREATORRashini Liyanarachchi, Frank Tran, Md Mahmudul Hasan, Aditya Joshi, Erik Meijering
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE58 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.29273

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

A Hybrid Framework for Song Lyric Annotation Based on Human-LLM Alignment Rashini Liyanarachchi, Frank Tran, Md Mahmudul Hasan, Aditya Joshi, Erik Meijering Abstract Emotion recognition of song lyrics is a challenging task since lyrics may not necessarily align with the overall emotion of a song. As a result, lyrics annotation remains largely underexplored. Drawing inspiration from research in large language model (LLM) assisted annotation, we examine the alignment between humans and LLMs for annotation of lyrics by creating a new sentence-level dataset of lyrics. Our observations highlight the subjectivity of the task and the inherent challenges. Following this, we present a hybrid annotation framework 1 1 1 https://github.com/Rashy98/LyricAnnotator-emotion that optimizes human and LLM annotation by predicting potential misalignment in annotation. I Introduction Emotion plays a central role in how we experience music [ 21 ] . While much research in music emotion recognition (MER) has focused on audio features such as tempo, mode, and timbre, textual modality such as lyrics is often either treated as secondary or simply assigned the same emotional label derived from listening to the full song [ 15 ] . Past research overlooks the fact that lyrics carry unique, often evolving emotional meaning that may not align with the music alone. Consequently, most contemporary MER approaches assign emotion labels to a song as a whole, combining the perceived emotions from both audio and lyrics without disentangling their individual contributions. This oversimplification ignores how lyrics can independently influence emotion perception, even shifting emotional tone line by line, thereby making emotion annotation of lyrics a temporally dynamic and semantically complex task. However, a growing body of recent research challenges the notion that lyrics are secondary in emotional perception. For instance, [ 5 ] found that lyrics significantly shape listeners’ emotional responses, suggesting they should not be treated as interchangeable with audio-based emotion cues. Similarly, [ 13 ] emphasized the multifaceted psychological mechanisms that drive emotional reactions to music, with lyrics contributing through narrative and semantic content. However, annotating lyric emotions is inherently difficult due to subjectivity, cultural variation, and linguistic nuances, and is further complicated by the fact that the process is often labor-intensive and time-consuming. [ 25 ] also demonstrated how subjective and sociocultural biases such as inter-annotator disagreement, individual interpretation, and cultural background, can affect emotion-related tasks in textual domains, complicating the consistency and comparability of annotations. These challenges, spanning cognitive subjectivity, cultural influences, linguistic ambiguity, and annotation scalability, highlight the need for alternative or complementary approaches to emotion annotation. Therefore , we investigate the research question: How well do large language models (LLMs) align with human annotation of emotion in lyrics and, based on this, how can LLMs and humans complement each other in the annotation process? We focus on annotation alignment between humans and LLMs at the sentence level , rather than at the song or verse level, to better capture the temporal and semantic evolution of emotions within lyrics. Sentence-level annotation provides a more precise unit of analysis, aligning with the dynamic, context-sensitive nature of emotional expression in language, an aspect often obscured when assigning a single emotional label to an entire song. Our approach to address the research question is motivated by key obstacles in the annotation process: subjectivity, cultural and interpretive variability, semantic ambiguity, and the high cost of manual annotation at scale. Building on this, we introduce a hybrid annotation framework that integrates human judgment with LLM outputs, aiming to capitalize on the complementary strengths of both. Specifically, we explore methods such as weighted aggregation of annotations and predictive modeling to determine the optimal source of annotation for each instance dynamically. The overarching objective is to develop a robust and reliable framework for lyric emotion annotation, where robustness refers to consistency across diverse linguistic and cultural contexts, and reliability is operationalized in terms of inter-annotator agreement and semantic coherence. This framework is intended to facilitate more precise and scalable emotion labeling, thereby supporting the development of more accurate multimodal MER systems. By leveraging LLMs in combination with human annotators, we aim to explore whether they can (1) increase objectivity and consistency between annotators, (2) reduce susceptibility to cultural bias, (3) capture nuanced emotional cues at a fine-grained level, and (4) significantly reduce the manual effort required for large-scale annotation. The rest of the paper is organized as follows. Section II outlines related work. Section III compares the ability to analyze emotions of humans and LLMs, covering dataset curation, annotation procedures, analyses, and cost considerations. Section IV introduces a hybrid annotation framework combining both sources, including weighted aggregation and annotation source prediction. Section V discusses the limitations and implications of our study. Finally, our main findings and suggestions for future work are summarized in Section VI . II Related Work We first outline past work in dataset collection and annotation, highlighting differences in emotional frameworks and granularity. We then summarize state-of-the-art modeling approaches, from traditional text-based methods to more advanced and multimodal techniques. II-A Lyrics Dataset Collection and Creation The development of lyrics-based emotion recognition systems critically depends on the availability and construction of appropriate datasets. Various studies have approached this challenge by collecting, annotating, and curating lyrics datasets from different sources, with diverse emotional taxonomies, granularity levels, and languages. [ 3 ] focused on collecting lyrics via web scraping (e.g. from Baidu Music) and annotating them with emotion categories derived from listener tags. Their dataset of 3,552 songs was filtered into three broad categories—contentment, depression, and exuberance—based on Thayer’s two-dimensional model [ 27 ] . To enhance label quality, ambiguous and overlapping tags were excluded. In contrast, [ 6 ] compiled lyrics from Billboard charts and Genius.com, applying the NRC Emotion Lexicon [ 18 ] to map words in lyrics to eight emotion categories. This lexicon-based approach offered scalable emotion quantification without the need for manual annotation. Building on the idea of symbolic annotation, [ 26 ] introduced Emotion4MIDI, a large-scale symbolic music dataset with annotations adapted from the EMOPIA dataset [ 11 ] . While not focused exclusively on lyrics, it exemplifies the integration of affective labels such as anger, happiness, sadness, and tenderness, using GoEmotions [ 8 ] categories to standardize emotion representation. To facilitate sentence-level modeling, [ 29 ] curated the Lyric-FM dataset, where each lyric line is annotated with continuous valence and arousal scores. This enables fine-grained learning of emotional nuance across time. The dataset supports supervised training of transformer-based models on sentence-level input, advancing beyond document-level representations. Multilingual and regional considerations have also shaped recent dataset efforts. [ 1 ] introduced a multilingual lyrics dataset incorporating English, Spanish, and Hindi songs. They utilized translated annotations and XLNet models to accommodate language variability, showing enhanced robustness across linguistic domains. Similarly, [ 24 ] developed a Telugu song lyrics dataset annotated in the valence-arousal (VA) space. They leveraged native speaker judgments and applied the XLM-RoBERTa model for domain adaptation, contributing to low-resource language representation in music emotion tasks. In summary, lyrics emotion datasets vary along several dimensions: annotation type (categorical versus continuous), granularity (song-level versus sentence-level), emotional taxonomy (discrete categories, VA space, or lexicon-based tags), and linguistic diversity (monolingual versus multilingual). These datasets provide the foundation for building increasingly expressive, context-aware models of lyrical emotion, and reflect

Read the complete text