Conflict-Aware Multimodal Fusion for Ambivalence and Hesitancy Recognition

Salah Eddine Bekhouche, Hichem Telli, Azeddine Benlamoudi, Salah Eddine Herrouz, Abdelmalik Taleb-Ahmed, Abdenour Hadid · 2026

Read Conflict-Aware Multimodal Fusion for Ambivalence and Hesitancy Recognition by Salah Eddine Bekhouche, Hichem Telli, Azeddine Benlamoudi, Salah Eddine Herrouz, Abdelmalik Taleb-Ahmed, Abdenour Hadid (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-15818.txt.

Introduction

Bekhouche and co-authors build a detector for ambivalence and hesitancy in people. Such states show as clashes between words, voice, and face. Text-only tools over-flag the state and miss its absence. Their ConflictAwareAH model reads video, audio, and text feeds. Pairwise gaps between modality embeddings act as two-way cues. Big gaps flag conflict, small gaps confirm calm consistency. A text-guided late fusion step adds further gains. The model hit 0.694 Macro F1 on the BAH test split. Training took under 25 minutes on one GPU.

Catalog record

CREATORSalah Eddine Bekhouche, Hichem Telli, Azeddine Benlamoudi, Salah Eddine Herrouz, Abdelmalik Taleb-Ahmed, Abdenour Hadid
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE31711
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.15818

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Conflict-Aware Multimodal Fusion for Ambivalence and Hesitancy Recognition Salah Eddine Bekhouche, Hichem Telli, Azeddine Benlamoudi, Salah Eddine Herrouz, Abdelmalik Taleb-Ahmed, Abdenour Hadid Abstract Ambivalence and hesitancy (A/H) are subtle affective states where a person shows conflicting signals through different channels—saying one thing while their face or voice tells another story. Recognising these states automatically is valuable in clinical settings, but it is hard for machines because the key evidence lives in the disagreements between what is said, how it sounds, and what the face shows. We present ConflictAwareAH , a multimodal framework built for this problem. Three pre-trained encoders extract video, audio, and text representations. Pairwise conflict features—element-wise absolute differences between modality embeddings—serve as bidirectional cues: large cross-modal differences flag A/H, while small differences confirm behavioural consistency and anchor the negative class. This conflict-aware design addresses a key limitation of text-dominant approaches, which tend to over-detect A/H (high F1-AH) while struggling to confirm its absence: our multimodal model improves F1-NoAH by +4.6 points over text alone and halves the class-performance gap. A complementary text-guided late fusion strategy blends a text-only auxiliary head with the full model at inference, adding +4.1 Macro F1. On the BAH dataset from the ABAW10 Ambivalence/Hesitancy Challenge, our method reaches 0.694 Macro F1 on the labelled test split and 0.715 on the private leaderboard, outperforming published multimodal baselines by over 10 points—all on a single GPU in under 25 minutes of training. 1 Introduction Ambivalence is when someone holds conflicting feelings about the same thing at the same time. Hesitancy is closely related: it is the reluctance or the delay before committing to a decision. Both states come up frequently in clinical psychology, behavioural economics, and human–computer interaction, yet they remain difficult to detect automatically. Why is this clinically important? In medical interviews, a patient may agree to a treatment plan verbally but show signs of doubt or stress on their face. Catching this mismatch early can help clinicians adjust their approach—for instance during motivational interviewing 17 —or flag attitudes linked to vaccine hesitancy 6 and treatment refusal. From a computational point of view, A/H is different from the “Big 6” basic emotions. Happiness or anger tend to show up clearly through a single channel: facial action units, prosody, or word choice. A/H, on the other hand, often produces contradictory cross-modal signals. A person says “I feel fine” while their voice wavers. Their words express agreement, but micro-expressions betray uncertainty. Standard multimodal fusion methods are designed to find what the modalities agree on; they can actually suppress the disagreement signals that define A/H. The ABAW10 Ambivalence/Hesitancy Video Recognition Challenge 12 provides a benchmark to study this problem. It uses the BAH dataset 7 : 1,427 clinical interview videos from 300 participants, annotated at the video level with binary A/H labels. With only 778 training videos and expert-level annotations that are expensive to obtain, the dataset is intentionally small—making regularisation and pre-trained representations essential. Our contributions are as follows: 1. Conflict-aware multimodal fusion. We introduce explicit pairwise conflict features that act as bidirectional cues: large cross-modal differences flag A/H, while small differences anchor the negative class. This corrects the systematic class bias of text-dominant approaches: our conflict-aware model improves F1-NoAH by +4.6 points over text alone and halves the class-performance gap—a crucial property for clinical deployment where false-positive A/H flags waste clinician time. 2. Text-guided late fusion. A text-only auxiliary head is blended with the full multimodal output at inference, adding +4.1 Macro F1 by leveraging the strong linguistic signal while keeping multimodal evidence for ambiguous cases. 3. Efficient, reproducible baseline. Our full pipeline trains in under 25 minutes on a single NVIDIA RTX 4090, yet outperforms published multimodal baselines by more than 10 points and achieves 0.715 Macro F1 on the official ABAW10 leaderboard. 2 Related Work Affective Behaviour Analysis and the ABAW Challenges. The Affective Behaviour Analysis in-the-Wild (ABAW) workshop series has been a principal driver of progress in multimodal affective computing, defining benchmarks for facial action unit (AU) detection, valence-arousal estimation, and discrete expression recognition 11 . The eighth edition introduced the Ambivalence/Hesitancy (A/H) Recognition Challenge, marking the first large-scale benchmark for this clinically motivated task 11 ; 12 . The tenth edition (ABAW10) 12 continues this track with the full BAH dataset 7 , providing participant-wise train/val/test splits and an additional unlabelled hold-out for challenge evaluation. Several teams competed in ABAW-8. Savchenko 24 achieved the highest reported Macro F1 of 0.772 using a text-dominant approach: frame-level visual emotion features were aggregated statistically and combined with transcript embeddings, with the linguistic modality contributing the decisive gain. This result established text as the primary carrier of A/H information. Hallmen et al. 8 fused Wav2Vec 2.0 acoustic features with BERT transcript embeddings and ViT visual features via a BiLSTM temporal module, confirming that semantic content of speech is more discriminative than acoustic prosody alone. Collectively, these submissions converge on a modality hierarchy—text > audio > visual—that motivates our text-guided late fusion strategy. BAH Dataset and A/H Recognition. González et al. 7 introduced the BAH dataset and established video-level multimodal baselines. Their best multimodal fusion result—LFAN co-attention 29 over all three modalities—achieves 0.59 Macro F1 at video level; zero-shot inference with a multimodal LLM 16 ; 30 and the full transcript reaches 0.63. Annotator cue analysis reveals that facial–language inconsistency dominates (42.6% of annotated conflict segments), followed by language–body (26.6%), providing direct empirical motivation for our explicit conflict features. Video Representation Learning. VideoMAE 26 extends masked autoencoders to video by masking a high proportion of spatiotemporal tube patches and reconstructing pixel values, producing representations that transfer well to action recognition. We adopt VideoMAE-Base (86M parameters) pre-trained on Kinetics-400, sampling 16 frames per clip. VideoFocalNet 27 is an alternative video backbone with strong results on the BAH video-level benchmark (Macro F1 0.566) using facial crops; however, it provides visual features only and does not integrate speech or transcript. Temporal modelling is particularly relevant for A/H recognition: hesitation cues accumulate over time 7 . Prior work employs Temporal Convolutional Networks (TCN) 3 and BiLSTM-based sequential models 8 for temporal affective modelling. In our architecture, multi-window inference—averaging predictions over five randomly sampled 16-frame windows—provides implicit temporal coverage without an explicit temporal decoder, a deliberate efficiency trade-off given the 778-sample training set. Self-Supervised Speech Representations. HuBERT 10 learns speech representations via offline k-means cluster assignment as pseudo-labels, achieving strong performance on speech tasks without supervision. We use HuBERT-Base fine-tuned on LibriSpeech 960h (94M parameters) to encode raw 16 kHz mono audio, capturing prosodic cues—pauses, pitch variation, stuttering—that annotators identify as primary A/H indicators in the BAH codebook 7 . Wav2Vec 2.0 1 is an alternative initially explored but replaced due to HuBERT’s superior sensitivity to expressive clinical speech. Emotion-Aware Language Models. GoEmotions 5 is a large-scale human-annotated emotion dataset covering 27 fine-grained emotion categories including confusion, doubt, and uncertainty. 23 23 fine-tuned RoBERTa-Base 15 on GoEmotions, producing a model with heightened sensitivity to hedging and hesitation language. We adopt this model as our text encoder, encoding the full video transcript rather than only the text aligned with the sampled visual window, reflecting the video-level nature of A/H: verbal hesitation indicators may occur at any point in the interview 7 . Multimodal

Read the complete text