Exploring Cognitive and Aesthetic Causality for Multimodal Aspect-Based Sentiment Analysis
Luwei Xiao, Rui Mao *, Shuai Zhao · 2025
Read Exploring Cognitive and Aesthetic Causality for Multimodal Aspect-Based Sentiment Analysis by Luwei Xiao, Rui Mao *, Shuai Zhao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-15848.txt.
Introduction
Luwei Xiao, Rui Mao, Shuai Zhao, and their co-authors present a 2025 computer science paper on reading feelings in posts that pair text with images. The task is to judge the feeling aimed at one named target in the post, not the post as a whole. Past systems often miss fine visual detail and the reasons behind the feeling a post expresses.
The team built a framework named Chimera that lines up small image patches with words. It turns both wide and narrow views of the image into text, then uses a large language model to add likely causes and impressions behind the feeling. Tests on standard data sets show the method works well. The authors also note it adapts to new tasks more flexibly than some large chat models.
Catalog record
| CREATOR | Luwei Xiao, Rui Mao *, Shuai Zhao |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 80 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-15848
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Exploring Cognitive and Aesthetic Causality for Multimodal Aspect-Based Sentiment Analysis Thanks: Luwei Xiao, and Liang He are with the School of Computer Science and Technology, East China Normal University, Shanghai 200062, China. E-mail: [email protected], [email protected] Rui Mao, Shuai Zhao, Yanhao Jia and Erik Cambria are with the College of Computing and Data Science, Nanyang Technological University, Singapore 639798. E-mail:{rui.mao, shuai.zhao, cambria}@ntu.edu.sg, [email protected] Qika Lin is with the Saw Swee Hock School of Public Health, National University of Singapore 119077. E-mail: [email protected] Thanks: * Corresponding author: Rui Mao Luwei Xiao, Rui Mao *, Shuai Zhao Abstract Multimodal aspect-based sentiment classification (MASC) is an emerging task due to an increase in user-generated multimodal content on social platforms, aimed at predicting sentiment polarity toward specific aspect targets (i.e., entities or attributes explicitly mentioned in text-image pairs). Despite extensive efforts and significant achievements in existing MASC, substantial gaps remain in understanding fine-grained visual content and the cognitive rationales derived from semantic content and impressions (cognitive interpretations of emotions evoked by image content). In this study, we present Chimera: a c ognitive and aest h et i c senti me nt causality unde r st a nding framework to derive fine-grained holistic features of aspects and infer the fundamental drivers of sentiment expression from both semantic perspectives and affective-cognitive resonance (the synergistic effect between emotional responses and cognitive interpretations). Specifically, this framework first incorporates visual patch features for patch-word alignment. Meanwhile, it extracts coarse-grained visual features (e.g., overall image representation) and fine-grained visual regions (e.g., aspect-related regions) and translates them into corresponding textual descriptions (e.g., facial, aesthetic). Finally, we leverage the sentimental causes and impressions generated by a large language model (LLM) to enhance the model’s awareness of sentimental cues evoked by semantic content and affective-cognitive resonance. Experimental results on standard MASC datasets demonstrate the effectiveness of the proposed model, which also exhibits greater flexibility to MASC compared to LLMs such as GPT-4o. We have publicly released the complete implementation and dataset at https://github.com/Xillv/Chimera I Introduction Multimodal aspect-based sentiment classification (MASC) is a valuable task for analyzing user-generated multimodal content on social platforms, aiming to predict the sentiment polarity of a specific target/aspect term within a sentence, based on an image-text pair. In an era marked by growing global interconnectedness, social platforms have become essential channels for individuals to express opinions and share experiences [ 1 , 2 , 3 ] . These platforms support multimodal content, blending text and visual media, which better reflects how sentiment is conveyed [ 4 ] . Consequently, analyzing fine-grained sentiment expression in multimodal scenarios not only improves the depth of sentiment classification but also aligns with the natural manner in which users express opinions and emotions, ultimately supporting more accurate sentiment analysis for applications in finance [ 5 , 6 ] , social research [ 7 , 8 ] , and human-computer interaction [ 9 , 10 ] . Current methodologies for MASC can be broadly divided into two principal categories: visual-text fusion-based approaches and translation-based approaches. Visual-text fusion-based methods address MASC by directly integrating visual content with textual features through various attention-based mechanisms [ 11 , 12 , 13 , 14 , 15 , 16 ] . Yu et al. [ 11 ] were the first to propose the utilization of ResNet for image feature extraction in conjunction with BERT for language sequence modeling, subsequently feeding these components into a BERT encoder to facilitate the interactive modeling of cross-modal representations. Ling et al. [ 14 ] introduced a vision-language pre-training framework that leverages Faster R-CNN for extracting object-level visual features and BART for generating textual features, with the model pre-trained using three task-specific strategies targeting the language, vision, respectively. Yu et al. [ 13 ] presented a novel multi-task learning framework Image-Target Matching Network (ITM), which concurrently performs coarse-to-fine-grained visual-textual relevance detection and visual object-target alignment through cross-modal Transformers. Translation-based approaches focus on mapping visual content into the language space as auxiliary textual representations, leveraging this supplementary information, or integrating it with visual features to enhance MASC [ 17 , 18 , 19 , 20 , 21 , 22 ] . Khan et al. [ 17 ] translated the image into a corresponding caption, which is then jointly input with the sentence into BERT to predict the sentiment polarity associated with specific targets. Yang et al. [ 23 ] exploit a face-sensitive, translation-based approach that translates facial expressions in images into textual sentiment cues, which are then selectively aligned and fused with the targets for enhanced sentiment analysis. Xiao et al. [ 19 ] proposed the CoolNet framework, which generates visual captions for images and extracts syntactic and semantic features from the textual modality, subsequently fusing these with visual features through a cross-modal Transformer. Despite substantial efforts and promising advancements, current solutions continue to encounter the following challenges. First, excessive duplicative visual patches can overshadow critical visual clues relevant to the specific target, leading to considerable misalignment during patch-token interactions. These small visual patches often lack semantic coherence compared to complete visual regions, particularly when aligning targets with their corresponding objects in an image, potentially leading to ambiguous semantic representations. Second, limited studies have focused on the underlying rationale behind sentiment cues, particularly from the perspectives of semantic content and affective-cognitive resonance. Owing to the multimodal nature of Twitter content, which spans diverse facets of daily life, inferring the sentiment associated with specific targets necessitates not only an understanding of the surface-level information in text and images (e.g., facial expressions) but also an in-depth comprehension of the contextual background of particular events and the impressions evoked by the image’s content and aesthetic attributes. To address the aforementioned challenges, this paper proposes Chimera: a c ognitive and aest h et i c senti me nt causality unde r st a nding framework. This framework aims to incorporate and align fine-grained features of specific targets and reasons about semantic and impression rationales. However, two critical issues must be resolved to achieve these objectives: 1) How can specific targets in a sentence be aligned with their corresponding object-level fine-grained features in an image? 2) How can the model be enabled to reason about the emotional causal reasons within the semantic content of image-text pairs and the affective resonance evoked by image aesthetic attributes? For the first question, we propose to make the cross-modal alignment of the target via the visual patch-level by linguistic-aware patch-token alignment and object-level by accurately translating the object feature into language space. Regarding the second issue, while a recent study [ 24 ] developed a reasoning dataset for MASC, this dataset primarily explains the emotional causes within textual content and lacks reasoning capabilities for visual content and the affective resonance evoked by images, limiting its suitability for the multimodal nature of this task. Consequently, we employ a large language model (LLM), GPT-4o, to generate the semantic rationale and impression rationale to understand the causal foundations of emotions. Specifically, our proposed framework first extracts visual patch-level and textual features, feeding them into a tailored linguistic-aware patch-token alignment (LPA) module to achieve patch-token alignment. Concurrently, a translation module (TM) translates the holistic image or object-level content into aesthetic captions or facial descriptions, leveraging multimodal named entity annotations from