From Annotation to Adaptation: Metrics, Synthetic Data, and Aspect Extraction for Aspect-Based Sentiment Analysis with Large Language Models

Nikita Neveditsin, Pawan Lingras, Vijay Mago · 2025

Read From Annotation to Adaptation: Metrics, Synthetic Data, and Aspect Extraction for Aspect-Based Sentiment Analysis with Large Language Models by Nikita Neveditsin, Pawan Lingras, Vijay Mago (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-20715.txt.

Introduction

Nikita Neveditsin, Pawan Lingras, and Vijay Mago look at how well large language models pick out feelings about parts of a text, in a 2025 study. The task is harder when the part being judged is hinted at rather than named. Most past tests used shops and hotels, so the team built fresh test data from sports feedback.

They tried several open models on the new data and set out a fresh score for judging what the models pull out. Their results show both promise and clear limits. For builders of review tools, the work gives a fair test bed and a sober view of what the models still miss.

Catalog record

CREATORNikita Neveditsin, Pawan Lingras, Vijay Mago
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE55 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-20715

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

From Annotation to Adaptation: Metrics, Synthetic Data, and Aspect Extraction for Aspect-Based Sentiment Analysis with Large Language Models Nikita Neveditsin, Pawan Lingras, Vijay Mago Abstract This study examines the performance of Large Language Models (LLMs) in Aspect-Based Sentiment Analysis (ABSA), with a focus on implicit aspect extraction in a novel domain. Using a synthetic sports feedback dataset, we evaluate open-weight LLMs’ ability to extract aspect-polarity pairs and propose a metric to facilitate the evaluation of aspect extraction with generative models. Our findings highlight both the potential and limitations of LLMs in the ABSA task. Table 1: ABSA Subtasks. Subtask Names Extracted Entity Aspect Extraction Liu (2012) , Opinion Target Expression Extraction Pontiki et al. (2015) , Aspect Term Extraction (ATE) Pontiki et al. (2014) ; Scaria et al. (2024) Aspect (e.g., "restaurant atmosphere", "technical support") Aspect Sentiment Classification Liu (2012) , Sentiment Polarity Classification Pontiki et al. (2015) , Aspect Term Polarity Classification Pontiki et al. (2014) Polarity (e.g., "positive", "negative", "neutral") Aspect Category Detection Pontiki et al. (2014) Category (e.g., "food") Opinion Term Extraction Zhang et al. (2023b) Opinion Phrase (e.g, "could be better") 1 Introduction ABSA is a nuanced form of sentiment analysis that focuses on identifying sentiments related to specific aspects within a text Pontiki et al. (2014) . Researchers have decomposed ABSA into various subtasks, such as aspect extraction, sentiment classification, aspect category detection, and opinion term extraction, each contributing to a comprehensive understanding of the problem. Table 1 summarizes these subtasks as discussed in the literature. Combining these tasks allows the extraction of ABSA-related entities in the form of tuples, triples, or quadruples from sentences or documents, resulting in a wide range of compound ABSA solutions. LLMs with their in-context learning (ICL) capabilities Brown et al. (2020) and parameter-efficient fine-tuning methods, such as Low-Rank Adaptation (LoRA) with quantization Dettmers et al. (2024) ; Hu et al. (2021) , offer straightforward yet effective approaches for complex ABSA tasks. These approaches facilitate the extraction of implicit aspects , which are aspects that are not explicitly stated in the text but can be inferred based on context, sentiment, or background knowledge. This study examines the performance of LLMs in extracting aspect-polarity pairs within the under-explored and unanticipated domain of sports feedback. This domain poses unique challenges for ABSA due to its reliance on implicit references and domain-specific terminology. By evaluating LLMs in this context, we provide critical insights into their capacity to adapt to novel data. Moreover, recognizing the linguistic variability involved in expressing implicit aspects, we propose an evaluation metric that calculates precision and recall while accounting for this variability in settings with a high prevalence of implicit aspects. We also demonstrate the broader applicability of this metric, showing its utility in assessing generative LLMs on classic ABSA datasets. Finally, we explore various strategies for adapting LLMs to domain-specific datasets, highlighting key challenges and offering insights for future research. 2 Related Work Aspect-Based Sentiment Analysis Traditional approaches to ABSA, extensively reviewed in the literature Nazir et al. (2020) ; Brauwers and Frasincar (2022) ; Zhang et al. (2023b) , primarily utilize bidirectional encoders Dos Santos et al. (2021) ; Zhang et al. (2023a) , recurrent networks Xu et al. (2020) , graph networks Zhou et al. (2020) ; Wu et al. (2022) ; Wang et al. (2024b) , sequence-to-sequence models Ma et al. (2019) , and ensembles of models Yang et al. (2023) . Various techniques have recently been proposed to improve accuracy, precision, and recall in ABSA-related tasks, for example, context denoising Tian et al. (2024) , abstract meaning representation Ma et al. (2023) , and global semantic features Zhou et al. (2024) . These methods have achieved robust results in within-domain explicit aspect extraction and polarity classification Meng et al. (2019) ; Meškelė and Frasincar (2020) ; Wang et al. (2020) . Recent studies have investigated the ability of LLMs to perform ABSA tasks on both traditional Šmíd et al. (2024) and more complex datasets Deng et al. (2023) ; Krugmann and Hartmann (2024) , highlighting the potential of generative models in key ABSA subtasks Kheiri and Karimi (2023) ; Scaria et al. (2024) ; Yang et al. (2024) . Nevertheless, challenges persist in effectively capturing implicit aspects, particularly in low-resource domains, where difficulties in data collection and annotation further exacerbate the problem Tubishat et al. (2018) ; Wankhade et al. (2022) ; Cai et al. (2021) ; Zhang et al. (2023b) . Data Creation and Annotation for ABSA Advancing ABSA research can benefit from quality datasets. Recent work by Chebolu et al. (2024) demonstrated that human annotation of ABSA datasets involving implicit aspects is challenging and laborious. Generative LLMs have been successfully utilized to create and annotate synthetic datasets, leveraging their capacity to generate creative and contextually rich text Meyer et al. (2022) ; Bao et al. (2023) ; Eldan and Li (2023) ; Mirowski et al. (2023) . Although LLMs may not always match human annotators in accuracy, studies have shown that their annotations can be valuable, particularly when combined with human expertise Goel et al. (2023) ; Gray et al. (2023) ; Mohta et al. (2023) ; He et al. (2024) ; Liyanage et al. (2024) . Moreover, leveraging synthetic data has been explored to enhance the performance of downstream models in various NLP tasks, including ABSA Kramchaninova and Defauw (2022) ; Yu et al. (2023) ; Deng et al. (2023) ; Wang et al. (2024a) . 3 Datasets 3.1 Novel dataset We introduce a novel dataset of artificially generated feedback from volunteers at sports event, a domain not yet represented in existing ABSA datasets. This domain poses unique challenges due to its specific terminology and the abundance of implicit aspects. The dataset facilitates an out-of-domain evaluation of the ABSA capabilities of open-weight LLMs against baseline solutions. Notably, at least 35% of its content comprises implicit aspects 1 1 1 Aspects that do not exactly match any part of a document. . Additionally, the dataset’s domain specificity provides an opportunity to test the generalization capabilities of ABSA solutions beyond their usual training contexts, contributing to a deeper understanding of their real-world applicability. We chose two state-of-the-art models 2 2 2 As of March 2024, when the dataset was generated and annotated for dataset generation: GPT-4 and Gemini 1.0 Ultra. The novel dataset comprises 480 documents, with an average of 222 characters per document. Most of the dataset (75%) was generated using GPT-4, acknowledging its superior reported results for major benchmarks such as MMLU OpenAI (2023) . Additionally, we employed Gemini 1.0 Ultra to generate 25% of the dataset, introducing some diversity of content. Appendix A provides examples of prompts and generated text, illustrating the models’ ability to produce mixed-emotion and diverse style feedback. The dataset annotation process, illustrated in Figure 1 , involved three steps, integrating both LLMs and human annotators. First, LLMs generated initial annotation drafts to alleviate the cognitive and time burden on the expert. Next, volunteers selected the better draft from two options. Finally, the expert revised and refined the selected draft. Step 1: Get initial annotation sets from LLMs Evaluate how number of documents per prompt impacts quality of annotations by LLMs Step 2: Ask volunteers to select

Read the complete text