MGHFT: Multi-Granularity Hierarchical Fusion Transformer for Cross-Modal Sticker Emotion Recognition CCS: Computing methodologies Computer vision CCS: Information systems Multimedia information systems

Jian Chen, Yuxuan Hu, Haifeng Lu, Wei Wang, Min Yang, Chengming Li, Xiping Hu · 2025

Read MGHFT: Multi-Granularity Hierarchical Fusion Transformer for Cross-Modal Sticker Emotion Recognition CCS: Computing methodologies Computer vision CCS: Information systems Multimedia information systems by Jian Chen, Yuxuan Hu, Haifeng Lu, Wei Wang, Min Yang, Chengming Li, Xiping Hu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-18929.txt.

Introduction

Jian Chen and co-authors teach a system to read feelings in chat stickers in a 2025 paper. Stickers lean on style cues and background jokes, so plain image tools miss their mood. The team first has a big vision-word model write notes on each sticker from several views.

A stacked fusion net then blends those notes into image traits stage by stage, from broad shapes down to fine marks. A text-led merge step seals the blend. On two public sticker sets, the design beat prior tools by a clear gap, with gains in both accuracy and fine feeling splits over strong image models. The code is shared in the open.

Catalog record

CREATORJian Chen, Yuxuan Hu, Haifeng Lu, Wei Wang, Min Yang, Chengming Li, Xiping Hu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE40 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-18929

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MGHFT: Multi-Granularity Hierarchical Fusion Transformer for Cross-Modal Sticker Emotion Recognition CCS: Computing methodologies Computer vision CCS: Information systems Multimedia information systems Jian Chen, Yuxuan Hu, Haifeng Lu, Wei Wang, Min Yang, Chengming Li, Xiping Hu Abstract . Although pre-trained visual models with text have demonstrated strong capabilities in visual feature extraction, sticker emotion understanding remains challenging due to its reliance on multi-view information, such as background knowledge and stylistic cues. To address this, we propose a novel m ulti- g ranularity h ierarchical f usion t ransformer ( MGHFT ), with a multi-view sticker interpreter based on Multimodal Large Language Models. Specifically, inspired by the human ability to interpret sticker emotions from multiple views, we first use Multimodal Large Language Models to interpret stickers by providing rich textual context via multi-view descriptions. Then, we design a hierarchical fusion strategy to fuse the textual context into visual understanding, which builds upon a pyramid visual transformer to extract both global and local sticker features at multiple stages. Through contrastive learning and attention mechanisms, textual features are injected at different stages of the visual backbone, enhancing the fusion of global- and local-granularity visual semantics with textual guidance. Finally, we introduce a text-guided fusion attention mechanism to effectively integrate the overall multimodal features, enhancing semantic understanding. Extensive experiments on 2 public sticker emotion datasets demonstrate that MGHFT significantly outperforms existing sticker emotion recognition approaches, achieving higher accuracy and more fine-grained emotion recognition. Compared to the best pre-trained visual models, our MGHFT also obtains an obvious improvement, 5.4% on F1 and 4.0% on accuracy. The code is released at https://github.com/cccccj-03/MGHFT_ACMMM2025. 1. Introduction Stickers, as a popular and expressive form of online communication, serve as an important medium for users to convey emotions in online chatting ( Zhao et al. 2021 ) . Compared to plain text, stickers encapsulate a rich combination of visual elements and accompanying textual cues, enabling more vivid and nuanced emotional expression ( Tang and Hew 2019 ; Herring and Dainas 2017 ) . With the increasing integration of stickers into social media and instant messaging platforms, Sticker Emotion Recognition (SER) has emerged as a promising research direction, drawing growing interest from the academic community. SER aims to automatically identify and classify the emotional content conveyed through stickers by leveraging both visual and textual modalities ( Liu et al. 2022 ) . This research enhances natural and empathetic human-computer interaction while advancing affective computing with novel insights and practical implications for emotion-aware AI systems. Figure 1 . Comparison with pre-trained models on SER30K dataset. Multimodal Large Language Models such as LLaVA and GPT-4o can understand the content of stickers but cannot effectively recognize the emotion in them. Pre-trained visual models such as CLIP and BLIP can extract visual features for classification, but the performance is not good enough. Compared to these methods, our proposed MGHFT demonstrates an obvious improvement in both accuracy and F1 score. Compared to conventional image-based emotion recognition tasks ( You et al. 2016 ; Yang et al. 2023 ) , SER introduces a set of unique and more complex challenges. One of the most prominent difficulties lies in the implicit nature of emotional cues commonly found on stickers, such as culture background and style, which are often context-dependent and subtle ( Zheng et al. 2025 ; Wang et al. 2024 ) . Additionally, the vast diversity in sticker styles, themes, and visual representations further complicates the task of accurately identifying emotional content ( Chen et al. 2024b ) . Unlike standard facial expression recognition or sentiment analysis from images, interpreting emotions in stickers often requires a holistic understanding that combines multiple views, including the intent behind the sticker ( Khattak et al. 2022 ; Ko 2018 ) , its stylistic tone, character actions, and even cultural or conversational context. Even for humans, accurate emotion understanding of stickers frequently relies on background knowledge and contextual inference, underscoring the complexity of SER and the need for advanced multimodal modeling approaches. Recent studies show that vision-language pre-trained models can effectively extract meaningful visual representations and have been widely applied to image understanding tasks ( Zhou et al. 2017 ; He et al. 2016 ; Radford et al. 2021 ; Li et al. 2023 ) . On the one hand, multimodal foundation models such as LLaVA ( Liu et al. 2024 ) and GPT-4o ( Hurst et al. 2024 ) excel in image-based generation tasks, leveraging their powerful visual-textual reasoning abilities and extensive background knowledge. On the other hand, vision-language models (VLMs) like CLIP ( Radford et al. 2021 ) and BLIP ( Li et al. 2023 ) , which are pre-trained through image-text alignment, have shown competitive performance in image classification tasks. Robust visual understanding provides a crucial foundation for the SER task, but it alone is not sufficient. More rich information about stickers from multi-views like style and details is also important. The results of the experiment presented in Figure 1 also prove this point. As illustrated in Figure 1 , experimental results on the SER30K dataset reveal a significant performance gap between these state-of-the-art VLMs and our proposed method. This discrepancy highlights an important insight. Accurately perceiving and interpreting subtle, implicit emotional cues like intention remains a significant challenge. Therefore, achieving effective emotion recognition in stickers requires a more comprehensive integration of contextual knowledge, emotional reasoning, and multi-view understanding. In this context, inspired by the way humans understand the emotion of stickers from multiple views, we propose a M ulti- G ranularity H ierarchical F usion T ransformer ( MGHFT ) to effectively integrate contextual text information into visual feature extraction. First, we introduce a multi-view sticker interpreter that leverages the powerful visual understanding capabilities of Multimodal Large Language Models (MLLMs) to generate rich textual descriptions from four views: intent, overall style, main character, and character details. These descriptions serve as rich semantic cues, aligning the model’s emotional reasoning with human perception. Second, we develop a hierarchical fusion strategy that incorporates multi-view textual features into different stages of visual representation learning. Specifically, we adopt a Pyramid Vision Transformer (PVT) as the visual backbone and inject view-specific textual features at each layer. In every stage, a CLS token captures global semantics, while local features are selectively attended based on attention weights. Through attention-based fusion and contrastive learning, the model performs fine-grained visual-textual alignment at both the global and local granularity. Finally, we introduce a Text-Guided Fusion Attention (TGFA) mechanism that aggregates and aligns multi-view textual semantics with visual representations across all stages, yielding robust emotion-aware features for SER. Extensive experiments on two large-scale sticker emotion datasets, SER30K and MET-MEME, show that our proposed MGHFT model consistently performs best, validating the effectiveness of MGHFT. Our main contributions can be summarized as follows: • We design a multi-view sticker interpreter that utilizes MLLMs to decompose stickers into multiple views, providing multi-view descriptions, thereby aligning human perceptual modalities to enrich the understanding of stickers. By introducing the knowledge of MLLMs, our model further improves the performance of emotion recognition. • We design a hierarchical fusion mechanism that injects multi-view textual semantics into different stages of visual feature extraction on the PVT backbone. By leveraging attention and contrastive learning at both global and local

Read the complete text