Visual and Textual Prompts in VLLMs for Enhancing Emotion Recognition
Zhifeng Wang, Qixuan Zhang, Peter Zhang, Wenjia Niu, Kaihao Zhang · 2025
Read Visual and Textual Prompts in VLLMs for Enhancing Emotion Recognition by Zhifeng Wang, Qixuan Zhang, Peter Zhang, Wenjia Niu, Kaihao Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-17224.txt.
Introduction
Zhifeng Wang, Qixuan Zhang, and Peter Zhang lead a team on this paper from two thousand twenty-five, whose text adds Niu, Zhang, Sankaranarayana, Caldwell, and Gedeon. It boosts how vision language models read feelings in video, by feeding them richer prompts.
Past methods zoom on the face alone and miss body language, scene, and the feelings of others in the shot. The frame, SoVTP, packs spatial marks such as boxes and face points, face muscle signals, and scene cues into one prompt plan. The whole scene stays in view while fine face moves can still be read. Broad tests show large gains over past visual prompting tricks, with no training on the target task.
Catalog record
| CREATOR | Zhifeng Wang, Qixuan Zhang, Peter Zhang, Wenjia Niu, Kaihao Zhang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 58 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-17224
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Visual and Textual Prompts in VLLMs for Enhancing Emotion Recognition Thanks: Qixuan Zhang, Zhifeng Wang, Wenjia Niu, Kaihao Zhang, Ramesh Sankaranarayana and Sabrina Caldwell were with the School of Computing, Australian National University (ANU) Canberra, ACT 2601, Australia. (e-mail: (qixuan.zhang, zhifeng.wang, wenjia.niu, Kaihao.Zhang, Ramesh.Sankaranarayana, Sabrina.Caldwell)@anu.edu.au) Peter Zhang was the senior AI scientist at the Quriosity Pty Ltd. (e-mail: [email protected] Tom Gedeon was with Human-Centric Advancements Chair in AI, Curtin University and Australian National University. (e-mail: [email protected]) Zhifeng Wang, Qixuan Zhang, Peter Zhang, Wenjia Niu, Kaihao Zhang Abstract Vision Large Language Models (VLLMs) exhibit promising potential for multi-modal understanding, yet their application to video-based emotion recognition remains limited by insufficient spatial and contextual awareness. Traditional approaches, which prioritize isolated facial features, often neglect critical non-verbal cues such as body language, environmental context, and social interactions, leading to reduced robustness in real-world scenarios. To address this gap, we propose Set-of-Vision-Text Prompting (SoVTP), a novel framework that enhances zero-shot emotion recognition by integrating spatial annotations (e.g., bounding boxes, facial landmarks), physiological signals (facial action units), and contextual cues (body posture, scene dynamics, others’ emotions) into a unified prompting strategy. SoVTP preserves holistic scene information while enabling fine-grained analysis of facial muscle movements and interpersonal dynamics. Extensive experiments show that SoVTP achieves substantial improvements over existing visual prompting methods, demonstrating its effectiveness in enhancing VLLMs’ video emotion recognition capabilities. I Introduction Emotion recognition is a key area in computer vision [ 1 ] , enabling systems to understand and respond to human emotions in applications like virtual reality, HCI [ 2 ] , surveillance [ 3 , 4 ] , and mental health [ 5 ] . Robust emotion understanding requires not only analyzing facial expressions but also interpreting body language and contextual interactions—often under challenging visual conditions [ 6 , 7 ] —and thus benefits from advanced restoration techniques [ 8 ] . Traditional methods for emotion recognition [ 9 , 10 , 11 , 12 ] primarily focus on facial features, using handcrafted features or shallow machine learning models [ 13 ] . While these approaches have shown success in controlled environments, they often fail in real-world settings where emotions are influenced by intricate interpersonal dynamics, environmental factors, and cultural variations. Fig. 1: Illustration of the SoVTP prompting for enhancing facial expression recognition in VLLMs. The approach incrementally adds (1) body masks for person identification and tracking, (2) face boxes with numbering to ground and differentiate individuals, (3) facial landmarks to analyze and locate the spatial relationships of facial action units, and (4) facial action units for detailed muscle movement analysis. This visual prompting strategy enables fine-grained emotion recognition by preserving contextual understanding and improving model interpretability in complex scenes. Fig. 2: Comparative analysis of emotion recognition in a complex setting , assessing the accuracy of facial emotion classification using plain text prompts versus SoVTP prompts, which integrates context, others’ emotions, body language, and facial muscle movement cues. Top : plain text prompts. Bottom : enhanced results using SoVTP. It demonstrates improved precision that incorporates environmental and social context, as well as facial muscle analysis. Recent studies [ 14 , 15 , 16 ] have explored zero-shot prompting techniques to enhance image-language tasks by guiding Large Language Models (LLMs) with specific visual markers. For example, RedCircle [ 17 ] employs red circles around objects to focus a Vision-Language Model’s attention on specific regions, improving tasks like zero-shot referring expression comprehension and keypoint localization. Similarly, ReCLIP [ 18 ] uses a region-scoring approach that combines colorful boxes for cropping and blurring irrelevant areas to enhance performance. Yang et al. [ 14 ] introduced fine-grained visual prompting with segmentation masks and a ‘Blur Reverse Mask’ technique that blurs regions outside the target area to preserve spatial context and reduce distractions. While these advances have demonstrated success in tasks like object grounding and segmentation, they face limitations when applied to emotion recognition. Methods like RedCircle and ReCLIP rely heavily on cropping techniques, which can result in the loss of critical contextual information essential for understanding emotions. Similarly, fine-grained visual prompts using Blur Reverse Mask can obscure facial features, introduce ambiguities, and hinder the accurate interpretation of nuanced emotional expressions. Although these methods hold considerable promise, their effectiveness in the domain of emotion recognition has yet to be systematically explored. To address these limitations, this paper systematically examines various forms of visual and textual prompting. Specifically, we propose a novel framework, Set-of-Vision-Text Prompting (SoVTP), illustrated in Fig. 1 . SoVTP utilizes spatial information, including numerical labels, bounding boxes, facial landmarks, and action units, to precisely identify targets while preserving the broader background context. This approach significantly enhances the zero-shot performance of facial expression recognition, enabling more robust and nuanced emotion analysis. The top part of Fig. 2 presents results derived from plain text prompts, indicating basic contextual information and the overall emotional assessment of individuals, but it yields incorrect results. In contrast, the bottom part of the Fig. 2 shows enhanced emotion recognition outcomes using SoVTP, which incorporates detailed analysis such as facial action units to identify specific muscle movements indicative of happiness. This integrated framework demonstrates improved precision in capturing nuanced emotional states by utilizing environmental, social, body language and anatomical cues, leading to more accurate differentiation between emotions such as ‘neutral’ and ‘happy’. To summarize, our main contributions are: • The paper proposes a novel Set-of-Vision-Text Prompting (SoVTP) approach that preserves holistic scene context rather than relying on isolated facial-cropping methods by overlaying spatial annotations (e.g., face IDs, landmarks, action units) on full-scene inputs, and integrates body posture, environmental cues, and social dynamics to enrich multimodal signals. • The SoVTP framework introduces a multi-stage prompting that treats zero-shot emotion recognition as structured inference over chained reasoning steps, enabling VLLMs to marginalize intermediate evidence without fine-tuning. • We present comprehensive experimental results that demonstrate SoVTP significantly improves VLLMs’ emotion recognition accuracy, achieving substantial gains over existing approaches. Fig. 3: Overview of the proposed SoVTP framework for emotion recognition , illustrating a multi-step approach combining vision and text prompts. The framework includes four key steps: (1) Muscle Movement, using Action Units to analyze facial muscle activity; (2) Others’ Emotions, identifying and labeling expressions of nearby individuals; (3) Context, extracting situational information to enhance emotional understanding; and (4) Body Language, assessing posture and gestures for additional emotional cues. The SoVTP prompting approach integrates these visual and contextual elements to improve accuracy in a complex scene II Related Work II-A Vision Large Language models Recent advances in VLLMs have been inspired by the success of traditional LLMs such as LLaMA [ 19 ] , PaLM [ 20 ] and GPT-4 [ 21 ] , which have shown remarkable abilities in zero-shot generalization in a range of NLP tasks. These LLMs have demonstrated the ability to understand complex instructions and perform diverse tasks without explicit fine-tuning. The success of these models has motivated the development of VLLMs, which extend these capabilities to the visual domain by using large-scale video-text paired datasets. VideoLLaMA2 [ 22 ] employs an image-level CLIP encoder as the vision backbone with a novel Spatial-Temporal Convolution Connector (STC Connector) for enhanced spatial-temporal representation learning, allowing effective aggregation of frame features while preserving local details without generating