Are You Sure You’re Positive? Consolidating Chain-of-Thought Agents with Uncertainty Quantification for Aspect-Category Sentiment Analysis

Filippos Ventirozos, Peter Appleby, Matthew Shardlow · 2025

Read Are You Sure You’re Positive? Consolidating Chain-of-Thought Agents with Uncertainty Quantification for Aspect-Category Sentiment Analysis by Filippos Ventirozos, Peter Appleby, Matthew Shardlow (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-17258.txt.

Introduction

Filippos Ventirozos, Peter Appleby, and Matthew Shardlow combine AI agents in a 2025 paper on review sentiment. The task sorts opinions by theme: a review may praise food but pan service. Trained models lead the field, but labeled data are scarce and costly for each new domain.

Large language models used with no training offer a way around that cost. The team runs several agents that reason step by step, then merges their answers with a measure of how sure each agent is. Bias in hand labels can make trained models look strong yet fail in new domains. A merged, doubt aware approach aims for results that travel better.

Catalog record

CREATORFilippos Ventirozos, Peter Appleby, Matthew Shardlow
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE50 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-17258

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Are You Sure You’re Positive? Consolidating Chain-of-Thought Agents with Uncertainty Quantification for Aspect-Category Sentiment Analysis Filippos Ventirozos, Peter Appleby, Matthew Shardlow Abstract Aspect-category sentiment analysis provides granular insights by identifying specific themes within product reviews that are associated with particular opinions. Supervised learning approaches dominate the field. However, data is scarce and expensive to annotate for new domains. We argue that leveraging large language models in a zero-shot setting is beneficial where the time and resources required for dataset annotation are limited. Furthermore, annotation bias may lead to strong results using supervised methods but transfer poorly to new domains in contexts that lack annotations and demand reproducibility. In our work, we propose novel techniques that combine multiple chain-of-thought agents by leveraging large language models’ token-level uncertainty scores. We experiment with the 3B and 70B+ parameter size variants of Llama and Qwen models, demonstrating how these approaches can fulfil practical needs and opening a discussion on how to gauge accuracy in label-scarce conditions. 1 Introduction Tracking customer satisfaction is critical for organisations aiming to improve their products and services. However, traditional supervised approaches require bespoke datasets, necessitating time and human labour ( Pustejovsky and Stubbs 2013 ) . The costs of annotation can be a significant challenge for many organisations, prohibiting access to adapted state-of-the-art solutions, and even to evaluation. Hence, in this paper, we explore and compare two popular large language models (LLMs) across various low-training scenarios, specifically zero-shot, for analysing customer satisfaction. Sentiment analysis approaches are commonly used to classify feedback as positive, neutral, or negative for customer satisfaction, but this method often lacks granularity. A single piece of feedback can express multiple sentiments targeting different product or service aspects, leading to ambiguous classifications and limiting actionable insights. To address this, aspect-based sentiment analysis (ABSA) methodologies ( Zhang et al. 2023 ) have been developed to more accurately capture the nuanced sentiments present in customer feedback. In our study, we investigated the ABSA task of aspect category sentiment analysis (ACSA), a type of ABSA. For a given piece of text, ACSA considers the aspect categories and sentiment polarities, as shown in Figure 1 . The categories are a set of classes which are predefined by domain experts or stakeholders. Characteristically, in the figure, the example demonstrates two tuples from the same text. Firstly, the review found the pepperoni pizza, a type of #Food (Category), to be delicious, representing a positive sentiment (Polarity). Secondly, the review found the service to be terrible, representing negative polarity. ACSA Tuple Example Input: The pepperoni pizza was delicious but the service was terrible though. Output: Category Polarity #Food positive #Service negative Figure 1: An ACSA example where a review results in two category-sentiment polarity pairs. Supervised learning approaches hold the state of the art for ACSA tasks ( Cai et al. 2020 ; Ping et al. 2024 ; Xu et al. 2025 ) . In contrast, we explore the zero-shot setting, which eliminates the need for labelled instances and allows us to experiment with LLM agents on novel datasets. This approach is particularly compelling in real-world applications, as organisations and firms often lack the resources or time required to annotate large datasets. In this paper, we propose a novel approach that integrates multiple LLM agents, each utilising a distinct chain-of-thought (CoT) reasoning process. Prior research ( Fei et al. 2023 ; Wang and Luo 2023 ) has introduced a sequential CoT approach, where aspects are first identified, followed by opinions and then polarities. In our experiments, we adapt this framework to the ACSA task and critically evaluate the assumptions underlying CoT by exploring alternative reasoning sequences. Additionally, we introduce mechanisms to combine the outputs of each CoT agent through aggregation techniques that make use of the LLMs’ token confidence scores, also referred to as token-level uncertainty. The primary contributions of this study are enumerated below: 1. We experimented with how the ordering of CoT prompt elements affects LLMs’ performance across datasets. 2. We compared multiple aggregation methods for effectively combining outputs from multiple CoT agents. 3. We benchmark our approach against a previously published zero-shot method, reproduced and evaluated on new datasets. 2 Related Work ABSA has emerged as a particularly interesting research topic due to its increasing popularity and widespread applicability across various domains Rink et al. 2024 ; Namee et al. 2023 ; Kit and Joseph 2023 ; Arianto and Budi 2020 ; Chu et al. 2022 ; Yu et al. 2023 inter alia). In this paper, we address the ACSA task, a type of ABSA. The ACSA task is especially relevant for industry applications, as it involves associating predefined category terms with corresponding sentiment polarities, which can be readily quantified and analysed (more on Section 3.2 ). Although a few studies have considered the ACSA task from a supervised learning perspective ( Cai et al. 2020 ; Ping et al. 2024 ; Xu et al. 2025 ) , it would be beneficial to address it in an unsupervised manner because annotation requires considerable time and human labour, and may also obfuscate reproducibility, as discussed later in this paper. Zero-shot and few-shot methodologies have been employed for the broader ABSA task. For instance, Hellwig et al. 2025 use LLMs with a few-shot in-context learning (ICL) approach, while Wang and Luo 2023 evaluate different prompting techniques for sentiment analysis using LLMs. Fei et al. 2023 investigated a multi-hop CoT approach, and Bai et al. 2024 introduced ChatABSA, an ICL few-shot prompt framework for ABSA that also extends to the ACSA task we address. In our paper, we aim to push the boundaries of zero-shot learning by examining the use of multiple LLM agents provided with no prior knowledge. More agents can yield better results; indeed, Li et al. 2024 discuss how employing additional agents and subsequently performing majority voting can improve outcomes. To the best of our knowledge, there is no prior work on multi-agent collaboration for ACSA. We therefore experiment with this approach, inspired by previous ABSA research on CoT ( Fei et al. 2023 ; Wang and Luo 2023 ) , and we investigate different aggregation techniques that leverage token-level uncertainty in LLMs ( Shorinwa et al. 2024 ) . 3 Methodology 3.1 Problem Statement Our problem statement aligns with prior work on ACSA. Given a text (whether a single sentence from a review or an entire review) our goal is to extract one or more pairs, each consisting of a category and its corresponding polarity, as depicted in Figure 1 . While the order of the pairs may vary due to the generative nature of the models used, the internal structure of each pair must remain consistent: the category always precedes the sentiment polarity. This ensures clarity and uniformity in the extracted results. Such as: Q=\{(c_{i},p_{i})\}_{i=1}^{n} (1) where Q represents the set of pairs, c_{i} is the category, and p_{i} is the polarity for the i -th pair, and n is the number of pairs extracted from the text. In the following subsections, we describe the different methods utilised in our experiments. 3.2 CoT Agents Previous research on related ABSA tasks ( Fei et al. 2023 ; Wang and Luo 2023 ) have typically followed an intuitive sequence: first extracting

Read the complete text