Emotion and Intention Guided Multi-Modal Learning for Sticker Response Selection
Yuxuan Hu, Jian Chen, Yuhao Wang, Zixuan Li, Jing Xiong, Pengyue Jia, Wei Wang, Chengming Li, Xiangyu Zhao · 2025
Read Emotion and Intention Guided Multi-Modal Learning for Sticker Response Selection by Yuxuan Hu, Jian Chen, Yuhao Wang, Zixuan Li, Jing Xiong, Pengyue Jia, Wei Wang, Chengming Li, Xiangyu Zhao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-17587.txt.
Introduction
Yuxuan Hu and eight co-authors improve how chat systems pick the right sticker in reply to a message. Stickers carry emotion and implicit intention at once, and earlier selection methods modeled those two signals separately, which mispicks whenever a sticker's emotion and intention point in different directions. Their framework is the first to model emotion and intention jointly, aligning both within each modality and across the visual and textual channels with a dual-level contrastive scheme.
A fusion module then integrates the two signals progressively: emotion guides which intention knowledge gets selected, attention fuses the guided streams, and a similarity-adjusted matching step makes the final choice. On two public sticker selection datasets, the method consistently beat the state of the art, with better reading of both emotional and intentional cues. The work treats a playful corner of online communication with appropriate seriousness.
Catalog record
| CREATOR | Yuxuan Hu, Jian Chen, Yuhao Wang, Zixuan Li, Jing Xiong, Pengyue Jia, Wei Wang, Chengming Li, Xiangyu Zhao |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 35 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-17587
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Emotion and Intention Guided Multi-Modal Learning for Sticker Response Selection Yuxuan Hu, Jian Chen, Yuhao Wang, Zixuan Li, Jing Xiong, Pengyue Jia, Wei Wang, Chengming Li, Xiangyu Zhao Abstract Stickers are widely used in online communication to convey emotions and implicit intentions. The Sticker Response Selection (SRS) task aims to select the most contextually appropriate sticker based on the dialogue. However, existing methods typically rely on semantic matching and model emotional and intentional cues separately, which can lead to mismatches when emotions and intentions are misaligned. To address this issue, we propose E motion and I ntention G uided M ulti-Modal L earning ( EIGML ). This framework is the first to jointly model emotion and intention, effectively reducing the bias caused by isolated modeling and significantly improving selection accuracy. Specifically, we introduce Dual-Level Contrastive Framework to perform both intra-modality and inter-modality alignment, ensuring consistent representation of emotional and intentional features within and across modalities. In addition, we design an Intention-Emotion Guided Multi-Modal Fusion module that integrates emotional and intentional information progressively through three components: Emotion-Guided Intention Knowledge Selection, Intention-Emotion Guided Attention Fusion, and Similarity-Adjusted Matching Mechanism. This design injects rich, effective information into the model and enables a deeper understanding of the dialogue, ultimately enhancing sticker selection performance. Experimental results on two public SRS datasets show that EIGML consistently outperforms state-of-the-art baselines, achieving higher accuracy and a better understanding of emotional and intentional features. The code is released at https://github.com/Applied-Machine-Learning-Lab/EIGML. Introduction Figure 1: Examples for potential failures in sticker selection when relying solely on emotional or intentional cues. With the rapid proliferation of social media platforms, stickers have attracted significant attention due to their unique ability to express emotions and convey communicative intent ( 48 ; 46 ) . An appropriately selected sticker can not only effectively highlight the emotions and intentions which users wish to express, but also serve as a substitute for complex textual responses ( 9 ; 33 ) . Consequently, the task of Sticker Response Selection (SRS) has emerged as an active research area and garnered significant attention. Compared to typical image-text matching tasks ( 44 ; 39 ; 43 ) , SRS poses more complex, highly context-sensitive challenges. Stickers serve not only as visual symbols but also as rich multi-modal carriers of emotional and intentional cues ( 35 ; 7 ; 32 ) , whose meanings vary greatly across conversational contexts. Thus, models must move beyond surface semantic alignment to effectively capture emotional signals and infer communicative intent for deeper multi-modal understanding ( 4 ; 16 ; 27 ) . However, many methods still rely on shallow semantic matching ( 13 ; 12 ) , failing to grasp subtle emotional and intentional cues. In real dialogues, stickers convey not only emotions but also tone, social stance, and complex communicative strategies. Ignoring these leads to semantically plausible yet contextually inappropriate choices. Although some studies incorporate emotional or intentional information to improve multi-modal alignment ( 40 ; 34 ) , they often treat the two separately, overlooking their intertwined nature in human communication. This separation may cause over-reliance on one aspect and neglect of the other, leading to mismatches in nuanced sticker selection. As shown in Figure 1 , models focusing only on emotion may miss subtleties like sarcasm or irony, while those centered on intention may ignore emotional undertones, resulting in contextually inappropriate selections. These cases show that isolating emotion or intention is insufficient and even harm overall performance. A unified emotion-intention framework is thus needed to better capture the contextual communicative functions of stickers. To address these challenges, we propose the Emotion and Intention Guided Multi-Modal Learning (EIGML) framework to enhance SRS accuracy by jointly modeling emotional and intentional cues. The framework has two core modules targeting key issues. First, to overcome the limitations of semantic matching in capturing subtle emotional variations and implicit intentions, we introduce the Dual-Level Contrastive Framework (DLCF), which boosts sensitivity to emotional and intentional features via intra-modality and inter-modality alignment. Second, to mitigate the bias caused by modeling emotion and intention in isolation, we design the Intention-Emotion Guided Multi-Modal Fusion (IEGMF) mechanism, comprising Emotion-Guided Intention Knowledge Selection (EIKS), Intention-Emotion Guided Attention Fusion (IEGA), and Similarity-Adjusted Matching Mechanism (SAMM). Together, they inject rich, effective information to deepen contextual understanding and improve selection. Extensive experiments on two large-scale datasets, StickerChat and DSTC10-MOD, demonstrate that our method consistently outperforms state-of-the-art models across multiple metrics, proving its effectiveness and robustness in complex scenarios. Our contributions include: • To the best of our knowledge, this is the first framework to jointly model emotion and intention explicitly for the task of Sticker Response Selection (SRS) task, reducing mismatches caused by isolated modeling and improving sticker selection accuracy in dialogue. We also identify the critical challenges in existing SRS methods rendered by the emotional and intentional cues, which have been less explored in the literature. • We propose a novel EIGML framework, which consists of two core components designed to address the mismatch between emotional and intentional features across visual and textual modalities. Building upon semantic alignment, EIGML introduces cross-modal alignment of emotion and intention representations to enhance overall modality consistency. Moreover, EIGML employs a three-stage progressive fusion strategy, where emotional and intentional cues are gradually injected to guide the integration of visual and textual features, thereby improving the effectiveness of sticker selection. • Extensive experimental results on two public benchmark datasets validate the effectiveness and efficiency of the proposed EIGML framework. Related Work Existing sticker response selection models can be grouped into shallow semantic matching and disjoint emotion–intention modeling. Shallow Semantic Matching Methods. Early approaches, such as 20 , use clustering-based methods to predict the next message, which is then replaced with a sticker. Later, a deep interaction network is proposed in 13 , which leverages cross-attention to extract multi-modal features for matching-based sticker retrieval. In 12 , both text and sticker prediction are framed as sequence generation tasks using a unified framework with pretrained GPT-2 ( 31 ) to jointly encode dialogue context for sticker selection. A multi-task learning framework is introduced by 47 to improve understanding of both sticker and dialogue semantics. While these methods show promising semantic matching results, they often neglect the rich emotional and intentional information in stickers, underscoring the need for explicit modeling of such information. Separate Emotion and Intention Modeling Methods. In recent years, several studies have attempted to go beyond semantic matching by incorporating emotional or intentional understanding of both stickers and dialogue. CKS ( 5 ) integrates commonsense knowledge to better recognize emotional expressions and extract unbiased visual features, thereby improving alignment between stickers and dialogues. PBR ( 40 ) focuses on emotional features within both stickers and dialogues, enabling emotion-aware sticker selection. Some works ( 34 ; 24 ) construct datasets with intention labels to incorporate intention into sticker response selection. While these methods introduce emotional or intentional signals, they typically model them separately, failing to capture their intertwined nature. To address this limitation, we propose a unified framework that integrates emotion and intention across modalities, improving both the accuracy and contextual relevance of sticker response selection. Method Overview We first introduce the problem definition of SRS, followed by an overview of our proposed framework