Large Language Models as Automatic Annotators and Annotation Adjudicators for Fine-Grained Opinion Analysis

Gaurav Negi, M.A. Waskow, Paul Buitelaar · 2026

Read Large Language Models as Automatic Annotators and Annotation Adjudicators for Fine-Grained Opinion Analysis by Gaurav Negi, M.A. Waskow, Paul Buitelaar (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-16800.txt.

Introduction

Gaurav Negi, M. A. Waskow, and Paul Buitelaar test if chatbots can tag fine views in text. Fine tags mark who a view aims at, not just its tone. Such tags take long hours of human work. Crowds must tag each view by hand. Their 2026 preprint asks if big models can take that load.

The team tries models as the first tag hands. They also try models as judges when hands clash. Sound tags would cut the cost of view polls. Brands and states track views to shape their acts. The tests show where bots help and where they slip.

Catalog record

CREATORGaurav Negi, M.A. Waskow, Paul Buitelaar
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE29 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-16800

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Large Language Models as Automatic Annotators and Annotation Adjudicators for Fine-Grained Opinion Analysis Gaurav Negi, M.A. Waskow, Paul Buitelaar Abstract Fine-grained opinion analysis of text provides a detailed understanding of expressed sentiments, including the addressed entity. Although this level of detail is sound, it requires considerable human effort and substantial cost to annotate opinions in datasets for training models, especially across diverse domains and real-world applications. We explore the feasibility of LLMs as automatic annotators for fine-grained opinion analysis, addressing the shortage of domain-specific labelled datasets. In this work, we use a declarative annotation pipeline. This approach reduces the variability of manual prompt engineering when using LLMs to identify fine-grained opinion spans in text. We also present a novel methodology for an LLM to adjudicate multiple labels and produce final annotations. After trialling the pipeline with models of different sizes for the Aspect Sentiment Triplet Extraction (ASTE) and Aspect-Category-Opinion-Sentiment (ACOS) analysis tasks, we show that LLMs can serve as automatic annotators and adjudicators, achieving high Inter-Annotator Agreement across individual LLM-based annotators. This reduces the cost and human effort needed to create these fine-grained opinion-annotated datasets. 1 Introduction There has been a considerable evolution in the field of fine-grained opinion analysis, which has led to many theoretical and practical formulations of opinions and sentiment. These fine-grained methods provide a clearer picture of not just the sentiment polarity (positive, negative, neutral) but also the contextual elements of the aforementioned sentiments. Consider the following example: “I had hoped for better battery life, as it had only about 2-1/2 hours doing heavy computations (8 threads using 100% of the CPU)” In the text, a negative sentiment is being expressed. However, a fine-grain annotation, unlike sentence or document-level annotation, has the opinion span expressing the polarity, i.e. “hoped for better” , along with an explicit instantiation of the target in the text called an aspect term “battery life” , forming an aspect sentiment triplet as specified according to the Aspect Sentiment Triplet Extraction (ASTE) task [ 16 ] . Moreover, the annotation can also include a more coarse-grained property “Battery#Operational_performance” , which is an aspect category, and expresses the attribute of the entity towards which the opinion is specifically directed. Aspect categories enable the encoding of cases where the aspect term is missing and the target sentiment is expressed implicitly. In addition to triplets elements, the quadruple annotations are assigned categories too, adhering to Aspect-Category-Opinion-Sentiment (ACOS) guidelines [ 1 ] . The existing datasets used for fine-grained opinion mining tasks are derived from SemEval tasks and are relatively small, limited to specific domains, and collected exclusively from review websites. As highlighted in [ 18 ] , these datasets are helpful in comparing different ABSA methods but are not sufficient for addressing real-world use cases. The subjective annotation of opinions by human annotators is a non-trivial task and requires strict adherence to annotation guidelines, as documented in [ 13 ] . LLMs are few-shot learners [ 8 ] and exhibit instruction following and reasoning capabilities [ 2 ] , which warrants the assessment of their use for annotating opinions. In this work, we investigate the use of LLMs to annotate text with fine-grained opinions. We also propose a novel LLM-based Adjudication approach that combines multiple opinion annotations for a given text input to resolve inter-annotator disagreement and obtain final opinion annotations. Our main contributions include: • Evaluation of fine-grained opinion annotation accuracy of LLMs (on ASTE and ACOS) • Declarative LLM-driven opinion annotation pipeline using DSPy. • We propose a method for aggregating annotations from multiple LLM-driven artificial annotators to produce final annotations using LLM as an annotation adjudicator. 2 Related Work 2.1 Opinion Formulations Opinion 1 1 1 Unless stated otherwise, we use the term opinion as a broad concept that covers sentiment and its associated information, such as opinion target and the person who holds the opinion, and use the term sentiment to mean only the underlying positive, negative, or neutral polarity implied by opinion. Mining and sentiment analysis have been well explored in natural language processing. Aspect-Based Sentiment Analysis (ABSA), the foundation of fine-grained opinion mining, evolved from feature-based summarisation [ 4 , 10 ] , which involves extracting and summarising opinions about features (attributes/ keywords). In this paper, we experiment with two fine-grained and extractive formulations of downstream ABSA tasks: Aspect Sentiment Triple Extraction (ASTE) [ 17 , 16 ] and Aspect-Category-Opinion-Sentiment Quadruple (ACOS) extraction [ 1 ] . 2.2 LLMs as Annotators Key steps in dataset creation for building any intelligent system include data selection, guideline development, human annotation, quality estimation, and adjudication [ 7 ] . With the improvement of the general ability of LLMs, they have been investigated for annotating various subjective tasks, including span annotations [ 5 ] , argument quality annotations [ 11 ] and propaganda span annotations [ 3 ] . Their findings argue in favour of using LLMs as independent or additional annotators. 2.3 Prompt Engineering LLMs are known to be sensitive to the prompts [ 19 ] and exhibit a higher level of variability in performing complex tasks [ 14 ] . To minimise the spurious interaction between the prompt and model selection as a confounding variable, we adopt DSPy [ 6 ] and the approach of “programming LLMs” rather than prompting them naively. It shifts the focus from performing string manipulation on prompt strings to programming with structured, declarative modules that automatically prepare prompts using a small sample of annotated examples. In this work, we propose a declarative pipeline based on LLMs for fine-grained opinion annotation. In addition to utilising individual LLMs to annotate text with fine-grained opinions, we also propose a novel LLM-based Adjudication approach to combine multiple annotations to resolve inter-annotator disagreement and obtain final annotations. 3 Data 3.1 Data Models We use two fine-grained formulations to describe data models for opinion representation (Figure 1(a) ); these are described subsequently. ASTE Aspect Sentiment Triple Extraction (ASTE) falls under the umbrella of ABSA tasks. Proposed by [ 12 ] , it is a task for extracting opinion triplets, thus providing one-shot answer to: What target is being discussed, How is the sentiment (e.g. positive) and Why is this sentiment (e.g. “ excellent ”). ACOS This data model is based on named Aspect-Category-Opinion-Sentiment (ACOS) Quadruple Extraction, to extract aspect, aspect category, opinion and sentiment as quadruples in text and provide full support for aspect-based sentiment analysis with implicit aspects and opinions 2 2 2 Opinion in ACOS task refers to the span that expresses sentiment. . The primary focus of this formulation is the fine-grained analysis of the opinion target. This task is more challenging than ASTE because it not only adds a category on top of the existing opinion triplet, but also takes into account implicit cases (i.e. missing explicit aspect terms or opinion spans) [ 1 ] . (a) ACOS & ASTE Specifications Task Dataset Train Dev Test ASTE lap14 906 219 328 res14 1126 310 492 res15 607 148 322 res16 857 210 326 ACOS laptop 2934 326 816 restaurant 1530 171 583 (b) Dataset breakdown 3.2 Datasets To align our experiments with those of the original ASTE and ACOS tasks, we use subsections of the original dataset splits, as

Read the complete text