VISAFF: Speaker-Centered Visual Affective Feature Learning for Emotion Recognition in Conversation
Linan Zhu, Zihao Zhai, Xiao Han Yuqian Fu, Xiangfan Chen, Xiangjie Kong, Guojiang Shen Zhejiang University of Technology ETH Zurich, Switzerland · 2026
Read VISAFF: Speaker-Centered Visual Affective Feature Learning for Emotion Recognition in Conversation by Linan Zhu, Zihao Zhai, Xiao Han Yuqian Fu, Xiangfan Chen, Xiangjie Kong, Guojiang Shen Zhejiang University of Technology ETH Zurich, Switzerland (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-18547.txt.
Introduction
Linan Zhu and six co-authors present VISAFF, a 2026 speaker-centered visual affective feature learning system for emotion recognition in conversation. Vision-language models often attend to backgrounds or passive listeners rather than the active speaker, and fine-tuning them is costly. VISAFF steers frozen models to the active speaker's emotional cues without training, then complements uncertain visuals with text and audio. On two real-world datasets it matches state-of-the-art methods while eliminating expensive fine-tuning.
Catalog record
| CREATOR | Linan Zhu, Zihao Zhai, Xiao Han Yuqian Fu, Xiangfan Chen, Xiangjie Kong, Guojiang Shen Zhejiang University of Technology ETH Zurich, Switzerland |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36055 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.18547
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
VISAFF: Speaker-Centered Visual Affective Feature Learning for Emotion Recognition in Conversation Linan Zhu, Zihao Zhai, Xiao Han Yuqian Fu, Xiangfan Chen, Xiangjie Kong, Guojiang Shen Zhejiang University of Technology ETH Zurich, Switzerland Abstract Emotion Recognition in Conversation (ERC) is essential for effective human-machine interaction, aiming to identify speakers’ emotional states in multi-turn dialogues. Early text-based methods struggle with complex scenarios like sarcasm because they inherently neglect vital non-verbal information. While recent Vision-Language Models (VLMs) address this by analyzing video directly, they are not inherently tailored for ERC and often focus on emotionally irrelevant background regions or passive listeners rather than the active speaker. Furthermore, fine-tuning these large models incurs prohibitive computational costs. Additionally, isolated visual signals are frequently ambiguous or technically compromised without the context of linguistic content and vocal prosody. To address these challenges, we propose VISAFF , a speaker-centered VIS ual AFF ective feature learning framework for ERC. VISAFF consists of two stages: Speaker-Centered Affective Grounding and Reliability-Guided Affective Complementation . VISAFF utilizes a tuning-free approach to unlock the reasoning capabilities of frozen VLMs, efficiently steering them to focus on the active speaker’s emotional visual cues without heavy training overheads. In the second stage, we introduce a reliability-guided affective complementation mechanism that dynamically leverages textual and acoustic modalities to compensate for visual uncertainty. Experiments on two real-world datasets demonstrate that VISAFF achieves highly competitive performance compared to state-of-the-art methods in a tuning-free setting, significantly enhancing computational efficiency by eliminating the need for expensive fine-tuning of large VLMs. The source code is available at https://anonymous.4open.science/r/speaker-2365/ . 1 Introduction With the development of visual sensing platforms such as smart glasses, in-vehicle cameras, and embodied agents, machines can continuously observe facial expressions, body movements, and interactive behaviors in real-world interactions [ 35 , 7 ] . However, effective human-machine interaction requires not only understanding observable behaviors, but also inferring human affective states to support contextually appropriate responses and decisions [ 3 , 31 ] . Consequently, Emotion Recognition in Conversation (ERC) has emerged to address this need, aiming to identify the emotional state of each speaker at each utterance in multi-turn dialogues [ 27 , 4 ] . Early ERC methods rely on textual signals to capture contextual semantics and emotional expressions [ 10 , 16 ] . While these approaches are straightforward, often inferring emotional states directly from key lexical cues [ 26 , 22 ] , they inherently neglect vital non-verbal information. Crucial emotional indicators such as facial expressions, vocal prosody (tone of voice), and body gestures are completely overlooked. This limitation becomes particularly evident in complex scenarios where a speaker deliberately masks their true intentions or uses sarcasm [ 5 , 28 ] , causing text-only methods to fail in accurately extracting the underlying affective state. Figure 1 : Motivation of speaker-centered visual affective feature learning for ERC. (a) Need for task-guided visual grounding : without explicit ERC-oriented guidance, visually salient regions may not match the speaker’s affective cues. (b) Context-dependent visual cues : visual expressions can be ambiguous without dialogue and acoustic context. Existing research has begun to directly analyze videos for Emotion Recognition in Conversation (ERC), especially with the emergence of Vision-Language Models (VLMs) [ 35 , 7 , 3 ] . These models break the limitations of traditional approaches that either suffer from information loss by narrowly focusing solely on localized facial expressions [ 6 , 34 ] or struggle with background noise and non-target speaker interference when using global video representations [ 12 , 8 , 31 ] . However, VLMs are not inherently tailored for ERC. They do not automatically fixate on the current speaker’s emotion-specific regions [ 3 , 35 , 7 ] . As illustrated in Figure 1 (a), their attention is often distracted by visually salient but emotionally irrelevant areas, such as dynamic backgrounds or passive listeners. While fine-tuning or parameter-efficient adaptations (e.g., LoRA) can steer VLMs toward relevant visual cues [ 13 , 11 ] , these training paradigms incur prohibitive computational costs and demand extensive annotated resources. This brings us to our first core challenge: How can we unlock the latent ERC reasoning capabilities of VLMs using a tuning-free approach, enabling them to efficiently and precisely attend to the active speaker’s emotional visual cues? Beyond isolated visual extraction, affective cues must be contextualized within a unified multimodal framework to ensure robust interpretation. As demonstrated in Figure 1 (b), a facial expression such as a smile is not strictly indicative of positive affect. Contextual nuances may reveal it to be a superficial facade, mapping instead to a neutral or negative underlying state. This ambiguity underscores that visual signals are highly sensitive to accompanying linguistic content, vocal prosody, and conversational history. Furthermore, real-world video streams are frequently compromised by technical artifacts like motion blur, rendering purely visual inference unreliable. Addressing these multimodal dependencies introduces our second challenge: How can we design an adaptive complementation mechanism that dynamically leverages textual and acoustic modalities to compensate for the uncertainty and unreliability of visual cues? To address these challenges, we propose VISAFF , a two-stage speaker-centered VIS ual AFF ective feature learning framework for ERC. Specifically, we leverage a tuning-free approach in the first stage to activate the inherent ability of the VLM to associate the current speaker with emotion-related visual cues, seamlessly circumventing huge training overheads. Moreover, to enhance the visual stream, we design a reliability-guided affective complementation mechanism in the second stage that adaptively integrates auxiliary text and audio information. By gating these modalities based on visual reliability, VISAFF significantly improves the model’s overall robustness. This innovative approach provides a highly efficient framework for multimodal reasoning, making it an effective solution for dynamic dialogue scenarios. The main contributions of this paper are summarized as follows: \bullet We propose VISAFF , a speaker-centered visual affective feature learning framework for ERC that activates frozen VLMs for visual affective reasoning without ERC-specific fine-tuning, and further enhances visual representations with textual and acoustic context. \bullet We introduce SCAG and RGAC : SCAG performs zero-shot speaker-centered affective grounding with frozen VLMs, while RGAC adaptively complements visual affective features with text and audio according to visual reliability. \bullet Experiments on two real-world ERC datasets demonstrate the effectiveness of VISAFF, supported by ablation studies and confidence-binned analyses. 2 Related Work ERC has been widely studied through contextual modeling and multimodal learning. Early methods mainly focus on textual context modeling, including sequential utterance modeling, speaker-state tracking, graph-based speaker relation modeling, and pretrained language representations [ 26 , 22 , 10 , 14 , 19 , 29 , 17 , 16 ] . Later multimodal methods introduce acoustic and visual signals through dynamic fusion, self-distillation, hypergraph modeling, adaptive graph learning, and evidence-cause reasoning [ 12 , 8 , 20 , 37 , 33 , 1 , 38 ] . Recent large-model-based methods further explore LLMs for ERC. DialogueLLM adapts LLMs for conversational emotion reasoning [ 39 ] , LaERC-S introduces speaker-related cues for zero-shot ERC [ 9 ] , and SpeechCueLLM converts acoustic cues into textual prompts for speech-aware reasoning [ 36 ] . These methods demonstrate the potential of large models for ERC, but they mainly rely on language-centered or speech-aware reasoning. For visual modeling, existing methods usually rely