OpinioRAG: Towards Generating User-Centric Opinion Highlights from Large-scale Online Reviews
Mir Tafseer Nayeem, Davood Rafiei · 2025
Read OpinioRAG: Towards Generating User-Centric Opinion Highlights from Large-scale Online Reviews by Mir Tafseer Nayeem, Davood Rafiei (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-00285.txt.
Introduction
Mir Tafseer Nayeem and Davood Rafiei present OpinioRAG in 2025, a framework that builds user-centric opinion highlights from thousands of online reviews. Popular products can draw more reviews than anyone can read. Past methods either fail to scale or give generic summaries that miss what a given user cares about.
OpinioRAG is scalable and needs no training. It retrieves evidence with a retrieval-augmented method and uses a language model to draft tailored highlights. New reference-free metrics check factual consistency with a fine, context-aware eye, tuned for sentiment-rich text. A new data set pairs entities with over a thousand reviews each with expert summaries and hand-labeled queries. Extensive tests support the approach.
Catalog record
| CREATOR | Mir Tafseer Nayeem, Davood Rafiei |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 94 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-00285
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
OpinioRAG: Towards Generating User-Centric Opinion Highlights from Large-scale Online Reviews Mir Tafseer Nayeem, Davood Rafiei Abstract We study the problem of opinion highlights generation from large volumes of user reviews, often exceeding thousands per entity, where existing methods either fail to scale or produce generic, one-size-fits-all summaries that overlook personalized needs. To tackle this, we introduce OpinioRAG , a scalable, training-free framework that combines RAG-based evidence retrieval with LLMs to efficiently produce tailored summaries. Additionally, we propose novel reference-free verification metrics designed for sentiment-rich domains, where accurately capturing opinions and sentiment alignment is essential. These metrics offer a fine-grained, context-sensitive assessment of factual consistency. To facilitate evaluation, we contribute the first large-scale dataset of long-form user reviews, comprising entities with over a thousand reviews each, paired with unbiased expert summaries and manually annotated queries. Through extensive experiments, we identify key challenges, provide actionable insights into improving systems, pave the way for future research, and position OpinioRAG as a robust framework for generating accurate, relevant, and structured summaries at scale 1 1 1 Project website: tafseer-nayeem.github.io/OpinioRAG . 1 Introduction Online reviews are an essential resource for consumers, offering firsthand insights into product quality, service reliability, and overall satisfaction across domains such as e-commerce, travel, and entertainment. With approximately 98 % of online shoppers consulting reviews before making a purchase ( PowerReviews 2023 ) , these user-generated evaluations shape consumer expectations, build confidence, and support informed decision-making ( Foo et al. 2017 ; Gamzu et al. 2021 ) . However, as online platforms expand, the sheer volume of reviews leads to information overload ( Malhotra 1984 ) , making it challenging for users to extract meaningful insights. To cope with this overload, users typically skim a limited number of reviews—often fewer than ten—resulting in biased or suboptimal decisions ( Kwon et al. 2015 ; Murphy 2016 ) . This highlights the pressing need for effective mechanisms that provide structured, personalized, and comprehensive access to relevant information. A common approach to mitigating information overload is summarizing large collections of reviews into concise, digestible summaries ( Hu & Liu 2004 ; Pang & Lee 2008 ; Suhara et al. 2020 ) , distilling salient viewpoints while filtering out irrelevant content to help users quickly grasp essential insights ( Ganesan et al. 2010 ; Hosking et al. 2024a ) . However, existing summarization methods have major limitations. First, prior work has primarily focused on summarizing short-form reviews, which pose relatively mild challenges given the capabilities of modern large language models (LLMs) ( Zhang et al. 2024 ) . Second, most approaches generate generic, one-size-fits-all summaries that fail to cater to personalized user needs. Consumers often seek context-specific insights aligned with their individual preferences—such as room cleanliness , proximity to public transport , availability of fitness facilities , or pet-friendly policies—but, current approaches lack the flexibility to generate query-specific summaries, limiting their utility for real-world decision-making. This work aims to bridge this gap by introducing a structured summarization framework that generates user-centric opinion highlights from large-scale, long-form reviews. A significant challenge in developing such frameworks is the scarcity of annotated datasets that pair user reviews with summaries. To address this limitation, previous studies have employed self-supervised approaches to generate synthetic review-summary pairs by designating individual reviews as pseudo-summaries ( Amplayo & Lapata 2020 ; Elsahar et al. 2021 ) . However, these datasets are limited in scale, typically containing at most 10 reviews per entity ( Angelidis & Lapata 2018 ; Chu & Liu 2019 ; Bražinskas et al. 2020a ) , with only a few extending to hundreds ( Angelidis et al. 2021 ; Bražinskas et al. 2021 ) . Such scales are inadequate for real-world scenarios where entities often accumulate thousands of reviews. Figure 1: Overview of the OpinioRAG Framework. The framework comprises two stages: (1) Retriever —Extracts relevant sentences as evidence for each query from the user reviews, and (2) Synthesizer —Generates structured opinion highlights using LLMs conditioned on the retrieved sentences. The final output summary is a structured collection of opinion highlights addressing various user queries, providing an user-centric overview of reviews. To advance model development and evaluation in this area, we introduce a novel dataset—the first of its kind at this scale—designed to facilitate user-centric summarization from extensive long-form reviews. Our dataset comprises entities with over a thousand user reviews each, paired with unbiased expert reviews and manually annotated queries (Table 1 ). This resource serves as a benchmark for evaluating LLMs’ ability to handle large-scale, noisy, and diverse inputs exceeding 100 K tokens ( Chang et al. 2024 ) , providing a foundation for developing more sophisticated, user-centric opinion summarization models. We further propose a training-free opinion highlight generation framework based on Retrieval-Augmented Generation (RAG) ( Lewis et al. 2020 ) , designed to effectively handle large-scale review corpora with user-centric queries (Figure 1 ). While long-context LLMs can process extensive input, they face limitations such as finite context windows, high computational costs ( Ahia et al. 2023 ) , privacy concerns from processing sensitive user information, and the needle-in-a-haystack problem ( Liu et al. 2024 ; Karpinska et al. 2024 ) , where critical information (e.g., Wi-Fi quality ) is buried among irrelevant content. To estimate factual alignment, we introduce novel, reference-free verification metrics tailored for sentiment-rich domains like product and service reviews. Unlike existing metrics like RAGAs ( Es et al. 2024 ) and RAGChecker ( Ru et al. 2024 ) , which focus on factual consistency in tasks such as question answering, our metrics capture nuanced opinions and sentiment polarity. Our contributions—including a dataset, an opinion highlight generation framework, and verification metrics—offer a unique advantage over traditional RAG-based benchmarks, which typically retrieve context documents from internet-scale repositories like Wikipedia and Common Crawl ( Gao et al. 2024 ) , sources often included in LLM pretraining corpora ( Grattafiori et al. 2024 ; Groeneveld et al. 2024 ) . This dependence on well-curated sources increases the risk of data contamination ( Shi et al. 2024 ) . In contrast, our method focuses on extracting insights from long-form, noisy user reviews, which are far less likely to overlap with LLM pretraining data due to rigorous filtering in pretraining pipelines ( Penedo et al. 2024 ; Longpre et al. 2024 ) . This setting provides a more realistic and challenging testbed for evaluating LLMs’ robustness and generalization in real-world scenarios. 2 Related Work Opinion Summarization. Opinion summarization has been widely studied through extractive and abstractive methods. Extractive approaches ensure factual consistency but often produce disjointed outputs ( Angelidis et al. 2021 ; Li et al. 2023 ) , while abstractive methods improve fluency but risk hallucination ( Bražinskas et al. 2020b ; Amplayo & Lapata 2020 ) . Recent work explores LLMs for summarization in zero- or few-shot settings ( Bhaskar et al. 2023 ; Siledar et al. 2024a ) , but primarily on short-form inputs. In contrast, OpinioRAG targets long-form review corpora exceeding 100K tokens ( Chang et al. 2024 ) , and generates structured, query-specific summaries in a key-point format segmented into PROS and CONS . Benchmarks. Existing datasets are limited in scale as they rely on pseudo-summaries constructed from short reviews ( Amplayo & Lapata 2020 ;