Talk, Snap, Complain: Validation-Aware Multimodal Expert Framework for Fine-Grained Customer Grievances
Rishu Kumar Singh, Navneet Shreya, Sarmistha Das, Apoorva Singh, Sriparna Saha · 2025
Read Talk, Snap, Complain: Validation-Aware Multimodal Expert Framework for Fine-Grained Customer Grievances by Rishu Kumar Singh, Navneet Shreya, Sarmistha Das, Apoorva Singh, Sriparna Saha (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-14693.txt.
Introduction
Rishu Kumar Singh, Navneet Shreya, and three co-authors build VALOR, a framework for sorting customer complaints by aspect and severity. Most past work reads only short posts or reviews, one channel at a time. Real support chats run for many turns and mix text with proof such as screenshots and product photos, which carry key facts.
VALOR routes each case through several expert models that reason step by step with large generative models. A semantic alignment score checks that the text and image stories agree before the final call is made. Finer sorting of grievances helps support teams fix the right problem first and spot severe cases early.
Catalog record
| CREATOR | Rishu Kumar Singh, Navneet Shreya, Sarmistha Das, Apoorva Singh, Sriparna Saha |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 65 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-14693
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Talk, Snap, Complain: Validation-Aware Multimodal Expert Framework for Fine-Grained Customer Grievances Rishu Kumar Singh, Navneet Shreya, Sarmistha Das, Apoorva Singh, Sriparna Saha Abstract Existing approaches to complaint analysis largely rely on unimodal, short-form content such as tweets or product reviews. This work advances the field by leveraging multimodal, multi-turn customer support dialogues—where users often share both textual complaints and visual evidence (e.g., screenshots, product photos)—to enable fine-grained classification of complaint aspects and severity. We introduce VALOR , a Validation-Aware Learner with Expert Routing, tailored for this multimodal setting. It employs a multi-expert reasoning setup using large-scale generative models with Chain-of-Thought (CoT) prompting for nuanced decision-making. To ensure coherence between modalities, a semantic alignment score is computed and integrated into the final classification through a meta-fusion strategy. In alignment with the United Nations Sustainable Development Goals (UN SDGs), the proposed framework supports SDG 9 (Industry, Innovation and Infrastructure) by advancing AI-driven tools for robust, scalable, and context-aware service infrastructure. Further, by enabling structured analysis of complaint narratives and visual context, it contributes to SDG 12 (Responsible Consumption and Production) by promoting more responsive product design and improved accountability in consumer services. We evaluate VALOR on a curated multimodal complaint dataset annotated with fine-grained aspect and severity labels, showing that it consistently outperforms baseline models, especially in complex complaint scenarios where information is distributed across text and images. This study underscores the value of multimodal interaction and expert validation in practical complaint understanding systems. Resources related to data and codes are available here: https://github.com/sarmistha-D/VALOR Introduction Your most unhappy customers are your greatest source of learning. Bill Gates Understanding customer complaints is crucial for improving user experience, service reliability, and product quality, objectives that directly align with the United Nations Sustainable Development Goals (SDGs), particularly SDG 9 (Industry, Innovation, and Infrastructure) and SDG 12 (Responsible Consumption and Production). While prior work has advanced complaint analysis, most efforts focus on short-form, unimodal inputs such as tweets ( 17 ) . These formats often lack the evolving context and emotional nuance found in real-world customer support scenarios. Moreover, modern users increasingly supplement textual complaints with visual content, such as screenshots of broken interfaces or damaged items shared across platforms ( 21 ; 24 ) . However, current research rarely addresses this multimodal and conversational nature of complaints, which introduces unique challenges in cross-modal alignment, contextual reasoning, and fine-grained classification. Figure 1: A conversation snippet from the CIViL dataset. Labels indicate the aspect-severity pairs identified from the conversation. Traditional approaches like aspect-based sentiment analysis primarily assign sentiment polarities (positive, negative, neutral) to specific aspects in isolated reviews. While helpful in gauging user opinion, these methods fall short when complaints evolve over multi-turn dialogues or involve multimodal evidence. Sentiment alone often lacks the specificity needed for actionable insights. For example, a user might report a late delivery and share a photo of a damaged product, each pointing to different aspects (logistics and packaging) and severity levels. Without disentangling such layered signals, automated systems fail to produce serviceable insights for resolution teams. We redefine complaint analysis as a fine-grained multimodal classification task over multi-turn dialogues, jointly modeling conversational flow and aligned images to identify aspect categories and severity levels with contextual accuracy. These structured outputs enable downstream applications such as intent-based ticket routing, escalation prediction, and service analytics. Figure 1 presents an example of such a use case. Building on this formulation, we propose CIViL : Customer Interactions with Visual and Linguistic signals, a curated dataset of multi-turn dialogues annotated with aspect and severity annotations and paired with topically aligned images sourced from social media websites. Motivated by the effectiveness of Large Language Models integrated with Mixture-of-Experts (MoE) architectures, we adopt a modular learning strategy that balances expert specialization with shared representation learning. To this end, we propose Validation-Aware Learner with Expert Routing ( VALOR ) , a multimodal framework that transforms complaint interactions into structured outputs using a robust MoE architecture enhanced with semantic alignment and Chain-of-Thought (CoT) reasoning. To assess the quality of modality integration and expert behavior, we incorporate a validation module that applies a three-part metric system, capturing alignment across modalities, dominance when one modality is more informative, and complementarity when both modalities contribute distinct yet coherent information. This work lays the foundation for real-world multimodal complaint understanding in dialogue systems by transforming unstructured complaints into structured, fine-grained insights; providing annotated resources, scalable methodologies, and actionable feedback to enhance service responsiveness and consumer engagement. Research Objectives: Following are the research objectives of the current study: (1) To investigate how textual and visual signals enhance fine-grained complaint analysis, specifically aspect and severity detection in multi-turn customer support dialogues. (2) To design and evaluate a two-phase MoE framework, VALOR , that combines cross-modal fusion, semantic alignment scoring, and CoT-based expert reasoning to produce accurate predictions from multimodal complaint data. (3) To develop a domain-specific multimodal customer-support dialogue dataset, enabling systematic benchmarking and advancing research in multimodal dialogue-grounded grievance analysis. Contributions: The primary contributions are as follows: (1) We define and investigate the task of fine-grained multimodal complaint understanding within multi-turn dialogues, with a specific emphasis on identifying aspect categories (ACD) and assessing severity levels (SD). (2) We introduce CIViL , a benchmark multimodal dataset of customer-support dialogues, built by extending a subset of the Kaggle customer support corpus with fine-grained aspect and severity annotations, and enriched with topically aligned images. This resource aims to advance research in multimodal conversational complaint understanding. (3) We propose VALOR , a Validation-Aware Learner with Expert Routing tailored for robust identification of real-world multimodal complaints in conversational settings. (4) The proposed framework sets a strong benchmark for fine-grained multimodal complaint analysis in dialogue-based contexts, consistently outperforming baselines across multiple metrics. Related Works Complaint Detection Complaint detection from text has become a key area in computational linguistics. Early methods relied on rule-based systems and handcrafted features to identify dissatisfaction ( 28 ) . With the rise of deep learning, particularly transformer models like BERT ( 2 ) semantic and contextual modeling improved significantly. Recent multitask approaches further enhance generalization by incorporating related cues such as sentiment, emotion, and sarcasm ( 27 ; 29 ; 26 ; 23 ) . However, these studies are limited to short, single-turn texts like tweets which lack contextual depth. Multimodal methods have begun addressing this gap by integrating visual and textual inputs ( 21 ; 1 ; 25 ) , but mostly rely on static features or simplistic fusion, often missing nuanced cross-modal interactions. Moreover, evaluations are typically conducted on product reviews, which lack the temporal structure and expressiveness of real-time, multi-turn conversations. Without multi-turn context, these models struggle to capture evolving emotions and layered complaint dimensions. In contrast, multi-turn dialogues offer richer signals, users elaborate issues over time, correct themselves, and discuss multiple concerns not evident in isolated posts. S.No. Annotation Guidelines 1 Annotations must be carried out independently, without outside influence. 2 Aspect categories should be assigned based on the customer’s viewpoint. 3 Each identified aspect must be paired with an appropriate severity level. 4 Choose the aspect label that most accurately reflects the specific cause of dissatisfaction. 5 Ambiguous cases should be resolved through discussion among annotators and authors. Table 1: Annotation guidelines for