Nuanced Emotion Recognition Based on a Segment-based MLLM Framework Leveraging Qwen3-Omni for A/H Detection
1 st Liang Tang, 2 nd Hongda Li, 3 rd Jiayu Zhang, 4 th Long Chen, 5 th Shuxian Li, 6 th Siqi Pei, 7 th Tiaonan Duan, 8 th Yuhao Cheng · 2026
Read Nuanced Emotion Recognition Based on a Segment-based MLLM Framework Leveraging Qwen3-Omni for A/H Detection by 1 st Liang Tang, 2 nd Hongda Li, 3 rd Jiayu Zhang, 4 th Long Chen, 5 th Shuxian Li, 6 th Siqi Pei, 7 th Tiaonan Duan, 8 th Yuhao Cheng (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-13406.txt.
Introduction
Liang Tang and co-authors build a segment-based framework to detect ambivalence and hesitancy in videos. These subtle states show through cross-modal clashes among face, voice, and text cues. Long videos strain compute and token limits, so the team splits them into clips of at most 5 seconds. They fine-tune Qwen3-Omni-30B-A3B on the BAH dataset with LoRA and full-parameter strategies. The model reads visual and auditory signals jointly for each segment. Test accuracy reached 85.1 percent, well above prior benchmarks, the team reports.
Catalog record
| CREATOR | 1 st Liang Tang, 2 nd Hongda Li, 3 rd Jiayu Zhang, 4 th Long Chen, 5 th Shuxian Li, 6 th Siqi Pei, 7 th Tiaonan Duan, 8 th Yuhao Cheng |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 17694 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.13406
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Nuanced Emotion Recognition Based on a Segment-based MLLM Framework Leveraging Qwen3-Omni for A/H Detection 1 st Liang Tang, 2 nd Hongda Li, 3 rd Jiayu Zhang, 4 th Long Chen, 5 th Shuxian Li, 6 th Siqi Pei, 7 th Tiaonan Duan, 8 th Yuhao Cheng Abstract Emotion recognition in videos is a pivotal task in affective computing, where identifying subtle psychological states such as Ambivalence and Hesitancy holds significant value for behavioral intervention and digital health. Ambivalence and Hesitancy states often manifest through cross-modal inconsistencies such as discrepancies between facial expressions, vocal tones, and textual semantics, posing a substantial challenge for automated recognition. This paper proposes a recognition framework that integrates temporal segment modeling with Multimodal Large Language Models. To address computational efficiency and token constraints in long video processing, we employ a segment-based strategy, partitioning videos into short clips with a maximum duration of 5 seconds. We leverage the Qwen3-Omni-30B-A3B model, fine-tuned on the BAH dataset using LoRA and full-parameter strategies via the MS-Swift framework, enabling the model to synergistically analyze visual and auditory signals. Experimental results demonstrate that the proposed method achieves an accuracy of 85.1% on the test set, significantly outperforming existing benchmarks and validating the superior capability of Multimodal Large Language Models in capturing complex and nuanced emotional conflicts. The code is released at https://github.com/dlnn123/A-H-Detection-with-Qwen-Omni.git . I Introduction Emotion recognition [ 5 ] in videos has become an important research topic in affective computing, with applications in digital health [ 7 ] , human–computer interaction, and behavioral analysis. In particular, recognizing subtle psychological states such as Ambivalence and Hesitancy (A/H) [ 3 ] is critical in behavioral change interventions. A/H reflects a conflicting emotional state in which individuals simultaneously experience positive and negative [ 1 ] attitudes toward a behavior or decision, often leading to delayed or abandoned behavioral change. Detecting such states automatically can support digital health systems, enabling adaptive interventions and scalable behavior change programs. Despite its importance, automatic recognition of A/H remains challenging [ 6 ] . Unlike basic emotions such as happiness or sadness, A/H manifests as a subtle and conflicting affective state, often expressed through inconsistencies across modalities [ 2 , 4 , 9 ] , including facial expressions, speech patterns, language cues, and body movements. These multimodal conflicts can occur both across modalities (e.g., positive speech with hesitant tone) and within a modality (e.g., fluctuating facial expressions), making the task particularly difficult for conventional emotion recognition systems. The recently introduced BAH dataset [ 3 ] provides a benchmark for studying A/H recognition in videos. It contains 1,427 videos from 300 participants, totaling approximately 10.6 hours of recordings, with expert annotations at both video and frame levels. Participants respond to predefined behavioral questions designed to elicit ambivalent or hesitant responses, producing naturalistic multimodal signals such as facial expressions, audio cues, and spoken language transcripts. However, existing baseline approaches demonstrate limited performance, highlighting the difficulty of the task and the need for more effective modeling strategies. A key challenge in applying modern multimodal large models to this task lies in the temporal length of video inputs. Directly feeding long videos into large multimodal models often exceeds token limits and increases computational cost, while also introducing irrelevant contextual noise. Meanwhile, previous studies indicate that A/H signals are typically short temporal segments, with an average duration of approximately 4.29 seconds within videos. This observation suggests that modeling shorter temporal windows may be sufficient for detecting A/H while improving computational efficiency. Fig. 1: A/H Recognition Algorithm Overall Pipeline To model multimodal interactions effectively, we adopt Qwen3-Omni-30B-A3B [ 8 ] , a powerful multimodal large language model capable of integrating visual and audio information. The model is fine-tuned on the BAH dataset using the proposed clip-based strategy, enabling it to capture subtle cross-modal cues and temporal inconsistencies associated with ambivalence and hesitancy. Experimental results demonstrate that our approach significantly improves recognition performance, achieving 85.1% accuracy on the test set, substantially outperforming previously reported baselines. In summary, our work makes the following contributions: • Clip-based temporal modeling: We propose a simple yet effective strategy that segments videos into 5-second clips, allowing multimodal models to capture local A/H signals while respecting token constraints. • Multimodal large model adaptation: We adapt Qwen3-Omni-30B-A3B for the A/H recognition task, leveraging its multimodal reasoning capability to detect subtle cross-modal emotional conflicts. • Strong empirical performance: Our method achieves 85.1% accuracy on the test set, demonstrating the effectiveness of combining temporal segmentation with multimodal large models for A/H recognition. These results suggest that large multimodal models, when combined with appropriate temporal segmentation strategies, can significantly advance the detection of complex and subtle emotional states such as ambivalence and hesitancy. II Method In this section, we describe the proposed framework for Ambivalence/Hesitancy (A/H) recognition. As shown in Fig.1, the overall pipeline consists of three stages: (1) A/H-aware video preprocessing based on annotation timestamps, (2) clip-based temporal segmentation, and (3) multimodal large model fine-tuning and inference using Qwen3-Omni-30B-A3B. II-A A/H-Aware Video Preprocessing The BAH dataset provides detailed annotations indicating whether a video contains A/H and, if present, the temporal segments where A/H occurs. We leverage this information to construct more informative training samples. Specifically, for each video we first read the annotation file that includes a global A/H label and detailed time intervals describing where A/H appears. If the video does not contain A/H (i.e., the global label is negative), the entire video is retained as a single training sample. In contrast, if the video contains A/H, we extract only the annotated temporal segments corresponding to A/H occurrences. Formally, for a video V with duration T , the annotation file provides a set of temporal segments: S=\{(t_{1}^{\text{start}},t_{1}^{\text{end}}),(t_{2}^{\text{start}},t_{2}^{\text{end}}),\ldots,(t_{k}^{\text{start}},t_{k}^{\text{end}})\}. (1) Each segment is cropped using ffmpeg to produce a new video clip that focuses on the time interval where A/H signals appear. This preprocessing step significantly reduces irrelevant visual and acoustic information while increasing the density of meaningful A/H cues in the training data. II-B Clip-Based Temporal Segmentation Although A/H segments are shorter than full videos, their duration can still vary significantly. Directly feeding long videos into a multimodal large model is inefficient and may exceed the model’s token limits. To address this issue, we adopt a clip-based temporal segmentation strategy. Given a video segment with duration T , we divide it into multiple shorter clips with a fixed maximum length of 5 seconds. The segmentation process follows a uniform sliding strategy: • If T<5s , the clip is kept unchanged. • If T\geq 5s , the video is divided into consecutive clips of 5 seconds. • Very short trailing segments are discarded to avoid extremely short clips. This strategy ensures that each input sample contains a compact temporal window that is sufficient to capture subtle emotional cues while remaining within the token capacity of the multimodal model. II-C Multimodal Large Model Fine-Tuning To model cross-modal emotional signals, we adopt Qwen3-Omni-30B-A3B , a large multimodal language model capable of jointly processing visual and audio inputs. The model integrates video frames, audio features, and textual prompts within a unified transformer architecture. Algorithm 1 Overall Pipeline for A/H Recognition 0: Raw video set \mathcal{V} , annotation file \mathcal{A} , clip length \Delta=5 s, multimodal model M 0: Video-level