Empathy Level Prediction in Multi-Modal Scenario with Supervisory Documentation Assistance

Yufei Xiao, Shangfei Wang · 2025

Read Empathy Level Prediction in Multi-Modal Scenario with Supervisory Documentation Assistance by Yufei Xiao, Shangfei Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-02558.txt.

Introduction

Yufei Xiao and Shangfei Wang work on predicting empathy, the felt sense of sharing another person's state. Most past methods read text alone, which misses tone of voice and facial cues. They also ignore extra documents that describe a scene and could enrich the reading.

Their 2025 method joins video, audio, and text in one multi modal system. Supervisory documents, holding privileged extra information, assist training as well. Richer inputs should help machines judge empathy in settings like counseling, education, and care, where missing a cue can change the right response. A judge who can hear, see, and read at once is far less prone to miss a key cue.

Catalog record

CREATORYufei Xiao, Shangfei Wang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE53 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-02558

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Empathy Level Prediction in Multi-Modal Scenario with Supervisory Documentation Assistance Yufei Xiao, Shangfei Wang Abstract Prevalent empathy prediction techniques primarily concentrate on a singular modality, typically textual, thus neglecting multi-modal processing capabilities. They also overlook the utilization of certain privileged information, which may encompass additional empathetic content. In response, we introduce an advanced multi-modal empathy prediction method integrating video, audio, and text information. The method comprises the Multi-Modal Empathy Prediction and Supervisory Documentation Assisted Training. We use pre-trained networks in the empathy prediction network to extract features from various modalities, followed by a cross-modal fusion. This process yields a multi-modal feature representation, which is employed to predict empathy labels. To enhance the extraction of text features, we incorporate supervisory documents as privileged information during the assisted training phase. Specifically, we apply the Latent Dirichlet Allocation model to identify potential topic distributions to constrain text features. These supervisory documents, created by supervisors, focus on the counseling topics and the counselor’s display of empathy. Notably, this privileged information is only available during training and is not accessible during the prediction phase. Experimental results on the multi-modal and dialogue empathy datasets demonstrate that our approach is superior to the existing methods. Fig. 1: An example of the supervisory documents. The left is the conversation in the counseling session, and the right is the corresponding supervisory document I Introduction Empathy is characterized by an emotional response that arises from the interplay between inherent traits and situational factors. This empathetic response is spontaneously triggered, yet it is also sculpted by deliberate cognitive control. The emergent emotion closely mirrors the individual’s direct or imagined experience and cognitive understanding of the emotional stimulus, accompanied by an awareness that the origin of this emotion lies outside oneself [ 2 ] . It is an essential element of human communication and interaction that enables our profound understanding and connection with others. It requires the capacity to discern and respond to the sentiments and perspectives of others. Therefore, predicting empathy plays a significant role in a wide range of applications, including, but not limited to mental health support, social media, and human-robot interaction. For instance, for mental health care, precise empathy prediction can assist therapists in adapting their approaches, aligning more closely with the emotional needs of clients, resulting in more effective and personalized treatments. Previous studies on predicting empathy aimed at designing efficient network architectures for predicting empathy levels [ 26 , 14 , 9 , 21 , 16 ] . They modeled the interaction between the client and the counselor during counseling sessions into feature representations to predict empathy ratings [ 23 , 24 , 10 ] . Although previous studies have made significant progress in empathy prediction, they mainly focused on using unimodal input, particularly text. Most real-world scenarios requiring empathy understanding, such as mental counseling, usually involve multi-modal input, including text, audio, and video. The overreliance on singular modalities leads to the incomprehensive understanding of empathy. Furthermore, few works in empathy prediction involve the utilization of privileged information, which may contain additional empathetic content. For example, the supervisory documents are objective descriptions of the entire counseling process conducted by a supervisor, which is independent of mental counseling. As illustrated in Figure 1, the descriptions mainly include topics and backgrounds of the counseling session, such as family, work, marriage, and others, as well as the client’s demands and emotional expressions, and the counselor’s empathetic behavior. Apart from these objective contexts and settings, this description typically includes the supervisor’s subjective evaluation of the effectiveness of the counselor’s empathetic behavior. Given their extensive expertise, psychological supervisors can offer clearer and more precise analyses of empathy in the counseling process. Therefore, the information contained in the supervisory documents, both subjective and objective, facilitates a more comprehensive analysis of empathetic expressions in counseling conversations, but they are exclusively accessible during the training phase, lacking presence in the testing phase and predictive downstream tasks. To address these limitations, we propose an advanced multi-modal empathy prediction method integrating video, audio, and text information. The method comprises two parts: Multi-Modal Empathy Prediction Network and Supervisory Documentation Assisted Training. The proposed empathy prediction network takes inputs from text, audio, and visual sources and generates a multi-modal representation to predict the empathy level. In addition, we utilize the supervisory documents to assist in the extraction of text features of the counseling process. We propose the supervisory documentation assisted training framework including a Latent Dirichlet Allocation (LDA) module to extract the topic distribution of the supervisory documents. We use this distribution as supervision to learn the corresponding conversation’s topic distribution from the original text feature. In this way, the supervisory documents constrain the text features to focus on critical empathy information, thereby improving the accuracy of empathy prediction. We conduct a series of experiments on the Multi-Modal Empathy Dataset in Counseling [ 27 ] (MEDIC) and the Mental Health Subreddits Dataset [ 15 ] . To the best of our knowledge, the MEDIC stands as the only publicly available multi-modal empathy dataset. Specifically, we perform ablation experiments and then compare our method with related works on this multi-modal dataset. The experimental results show that our method improves the accuracy and F1 score with multi-modal input and assistance of supervised documents, compared to the single-modal and unassisted cases. Additionally, we conduct cross-dataset experiments on the Mental Health Subreddits Dataset [ 15 ] to validate the generalization capabilities of our approach. Compared to related work, our method demonstrates clear advantages. Contributions of this work are summarized as follows: (1) We introduce a multi-modal empathy prediction network, which enables us to gather more comprehensive information across multiple modalities. (2) We novelly leverage supervisory documents through a dedicated training framework to assist the text feature extraction. This integrates the rich empathy information from supervisory documents into the text features. II Related Work In this section we will give a brief review on previous works of empathy prediction and assisted training with privileged information. II-A Empathy prediction Empathy prediction is to estimate the level of empathy displayed in given communication scenarios, such as counseling sessions or online conversations. Related studies on predicting empathy mainly focus on developing network architectures to predict empathy. Gibson et al. [ 6 ] introduced a recurrent neural network (RNN) that maps the text content of each speaker turn to a series of behavioral acts, which was then employed to initialize lower layers of the network to predict counselors’ empathy ratings. They conducted the experiments on the dataset including motivational interviews collected from six independent clinical studies on addiction counseling. Sharma et al. [ 15 ] proposed a computational approach to comprehend the expression of empathy in online mental health platforms. They created a new unified, theoretically grounded framework to characterize empathy communication in text-based conversations. Xiao et al. [ 25 ] employed an N-gram language model based on a maximum likelihood strategy to differentiate empathic from non-empathic utterances to measure empathy. They carried out the experiments on the dataset of the transcripts from clinical studies on substance use. Kumano et al. [ 8 ] developed a probabilistic model that employed a hierarchical structure involving facial expressions, and other behaviors, including utterances and gaze direction to estimate emotional

Read the complete text