Multimodal Ambivalence/Hesitancy Recognition in Videos for Personalized Digital Health Interventions
Manuela González-González, Soufiane Belharbi, Muhammad Osama Zeeshan, Masoumeh Sharafi, Muhammad Haseeb Aslam, Lorenzo Sia, Nicolas Richet, Marco Pedersoli, Alessandro Lameiras Koerich, Simon L Bacon & Eric Granger · 2026
Read Multimodal Ambivalence/Hesitancy Recognition in Videos for Personalized Digital Health Interventions by Manuela González-González, Soufiane Belharbi, Muhammad Osama Zeeshan, Masoumeh Sharafi, Muhammad Haseeb Aslam, Lorenzo Sia, Nicolas Richet, Marco Pedersoli, Alessandro Lameiras Koerich, Simon L Bacon & Eric Granger (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-11730.txt.
Introduction
González-González and ten colleagues explore deep learning for ambivalence and hesitancy recognition in videos for digital health. Ambivalence and hesitancy are subtle mixed feelings between positive and negative views or acceptance and refusal, making patients delay, avoid, or abandon care. They show up as mismatched feelings across or within channels like language, face, voice, and body. Experts can spot them, but adding them to digital care is costly. Automatic recognition is therefore key for tailored care, covering supervised learning, unsupervised domain adaptation, and zero-shot inference.
Catalog record
| CREATOR | Manuela González-González, Soufiane Belharbi, Muhammad Osama Zeeshan, Masoumeh Sharafi, Muhammad Haseeb Aslam, Lorenzo Sia, Nicolas Richet, Marco Pedersoli, Alessandro Lameiras Koerich, Simon L Bacon & Eric Granger |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 43217 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.11730
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Multimodal Ambivalence/Hesitancy Recognition in Videos for Personalized Digital Health Interventions Manuela González-González, Soufiane Belharbi, Muhammad Osama Zeeshan, Masoumeh Sharafi, Muhammad Haseeb Aslam, Lorenzo Sia, Nicolas Richet, Marco Pedersoli, Alessandro Lameiras Koerich, Simon L Bacon & Eric Granger Abstract Using behavioural science, health interventions focus on behaviour change by providing a framework to help patients acquire and maintain healthy habits that improve medical outcomes. In-person interventions are costly and difficult to scale, especially in resource-limited regions. Digital health interventions offer a cost-effective approach, potentially supporting independent living and self-management. Automating such interventions, especially through machine learning, has gained considerable attention recently. Ambivalence and hesitancy (A/H) play a primary role for individuals to delay, avoid, or abandon health interventions. A/H are subtle and conflicting emotions that place a person in a state between positive and negative evaluations of a behaviour, or between acceptance and refusal to engage in it. They manifest as affective inconsistency across modalities or within a modality, such as language, facial, vocal expressions, and body language. While experts can be trained to recognize A/H, as done for in-person interactions, integrating them into digital health interventions is costly and less effective. Automatic A/H recognition is therefore critical for the personalization and cost-effectiveness of digital health interventions. In this paper, we explore the application of deep learning models for A/H recognition in videos, a multi-modal task by nature. In particular, this paper covers three learning setups: supervised learning, unsupervised domain adaptation for personalization, and zero-shot inference via large language models (LLMs). Our experiments are conducted on the unique and recently published BAH video dataset for A/H recognition. Our results show limited performance, suggesting that more adapted multi-modal models are required for accurate A/H recognition. In addition, better methods for modeling spatio-temporal and multimodal fusion are necessary to leverage conflicts within and across modalities. Our code is publicly available: github.com/sbelharbi/ah-digital-health-interventions . Keywords: Digital Behavioural Change Intervention, Behavioural Ambivalence/Hesitancy Dataset, Ambivalence/Hesitancy Recognition in Videos, Affective Computing. Figure 1: Conceptual illustration of the theoretical pathway by which ambivalence/hesitation (A/H) recognition can influence a digital behaviour change intervention: a user interacts with a smartphone app targeting physical activity, while the app detects A/H and adaptively adjusts cues and strategies to deliver more personalized support. 1 Introduction The majority of medical health interventions aim to modify high-risk health behaviours that contribute to the growing burden of chronic diseases 33 . Such diseases account for substantial amounts of morbidity 7 and mortality 24 worldwide 102 . Major efforts have been put in place to positively modify these high-risk behaviours that impact human health and well-being through the development of effective behaviour change interventions 64 , 79 , 97 . Despite these efforts, healthcare providers still struggle to support patients to acquire and maintain healthy behaviours 72 . For instance, cardiovascular disease (CVD), which remains the main contributor of deaths before age 75 75 , is largely caused by modifiable high-risk behaviours such as smoking, low-quality diet, physical inactivity, and medication non-adherence 76 . In addition, roughly 50% of premature deaths from heart disease, stroke, and cancer are found to be related to high-risk behaviours 63 . Though behaviour change-focused health interventions targeting high-risk behaviours have been shown to consistently improve health outcomes 16 , they do not seem to work for everyone, suggesting that greater personalization is needed 36 , 37 , 113 . While in-person health interventions are the most common form of intervention in healthcare systems 65 , 86 , they are costly, have limited accessibility, and are difficult to scale. Digital health interventions are potentially cheaper, more scalable options which could leverage common and affordable modalities such as web- and phone-based applications 103 , intending to maintain the effectiveness of in-person interventions whilst also creating better access and a greater chance for personalization. Achieving and maintaining long-term behaviour change often includes overcoming ambivalence and hesitancy (A/H) 66 , 67 , 96 , which are the primary reasons for individuals to delay, avoid, or abandon health behaviour changes 9 , 10 , 61 , 70 , 95 , 104 . A/H is a subtle and conflicting emotion manifested by an affective inconsistency across multiple modalities or within a modality, such as facial and vocal expressions, and body language. The presence of this conflicting affect sets an individual in a state between positive and negative evaluations of a behaviour, or between the acceptance and refusal to act, which often constitutes a barrier to initiating behaviour change and a trigger for discontinuing health interventions or behaviour change efforts. Healthcare providers (e.g., clinicians, therapists, interventionists) often identify A/H through a combination of speech and non-verbal cues, e.g., facial expressions and tone 31 , 44 , 70 during in-person interactions. However, human integration into digital health interventions and e-health tools can be costly and less effective. Therefore, designing robust automated methods for A/H recognition can provide a cost-effective alternative to adapt for individual users and operate seamlessly in real-time and resource-limited environments. Leveraging machine learning (ML) for automating digital health intervention has recently gained much attention 6 , 52 , 69 . Recent research on ML in emotion recognition focuses mainly on seven basic discrete emotions in images or videos, e.g, ‘happy’, ‘sad’, and ‘surprised’ 4 , 13 , 54 , 106 . Other models in the literature predict ordinal levels, including levels linked to health states, e.g., pain, depression, and stress estimation 2 , 8 , 88 , 111 , 74 . Other affects are continuous predictions such as valence-arousal 17 , 80 , 82 , 81 , 93 . However, real-world scenarios present more complex cases of emotions. Recently, there has been an increased interest in designing robust affect models for compound emotions, a case where a mixture of basic emotions is manifested 42 , 11 , 84 . In particular, compound emotions commonly occur in daily interactions. Recognizing affective states linked to compound emotions and health states is, however, more difficult to discern as they are subtle, ambiguous, and resemble basic emotions. A/H recognition is related to such tasks where intention and attitude are conflicting or in an intermediate (in-between) state, between willingness and resistance 59 , or positive and negative affect 1 . This can manifest in how individuals express themselves and can be recognized 28 in their facial expression, tone, verbal expressions, and body language. As a result, A/H is inherently multimodal, arising from subtle interactions among multiple cues both across modalities and within each modality. Unfortunately, such discord is extremely difficult to detect – a task that requires human training. This is a tedious and expensive procedure, leading to ineffective and less scalable digital interventions especially under limited resources. Integrating automated and reliable tools for A/H recognition can have a major impact on improving digital health interventions. Although A/H is a common topic in behavioural science 9 , 34 , 61 , it remains unexplored in the ML community, and as such, in the design of eHealth components. In this paper, we explore the application of deep learning models for the newly introduced task, A/H recognition. Our experiments cover different learning scenarios, in particular supervised learning of A/H recognition at the frame and video level, where different modalities are explored, such as visual,