Paralinguistic Emotion-Aware Validation Timing Detection in Japanese Empathetic Spoken Dialogue
Zi Haur Pang, Yahui Fu, Yuan Gao, Tatsuya Kawahara · 2026
Read Paralinguistic Emotion-Aware Validation Timing Detection in Japanese Empathetic Spoken Dialogue by Zi Haur Pang, Yahui Fu, Yuan Gao, Tatsuya Kawahara (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-09307.txt.
Introduction
Zi Haur Pang and colleagues detect the right moment for emotional validation in Japanese empathetic spoken dialogue, working from speech alone without text context. Validation, a therapy technique that acknowledges another person's feelings, helps most when its timing and frequency are right. They fuse two HuBERT-based encoders, one tuned to paralinguistic cues and one to speech emotion, then fine-tune the pair on the timing task. Tests on an emotional storytelling corpus show clear gains.
Catalog record
| CREATOR | Zi Haur Pang, Yahui Fu, Yuan Gao, Tatsuya Kawahara |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 22399 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.09307
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Paralinguistic Emotion-Aware Validation Timing Detection in Japanese Empathetic Spoken Dialogue Zi Haur Pang, Yahui Fu, Yuan Gao, Tatsuya Kawahara Abstract Emotional Validation is a psychotherapy communication technique that involves recognizing, understanding, and explicitly acknowledging another person’s feelings and actions, which strengthens alliance and reduces negative affect. To maximize the emotional support provided by validation, it is crucial to deliver it with appropriate timing and frequency. This study investigates validation timing detection from the speech perspective. Leveraging both paralinguistic and emotional information, we propose a paralinguistic- and emotion-aware model for validation timing detection without relying on textual context. Specifically, we first conduct continued self-supervised training and fine-tuning on different HuBERT backbones to obtain (i) a paralinguistics-aware Self-Supervised Learning (SSL) encoder and (ii) a multi-task speech emotion classification encoder. We then fuse these encoders and further fine-tune the combined model on the downstream validation timing detection task. Experimental evaluations on the TUT Emotional Storytelling Corpus (TESC) compare multiple models, fusion mechanisms, and training strategies, and demonstrate that the proposed approach achieves significant improvements over conventional speech baselines. Our results indicate that non-linguistic speech cues, when integrated with affect-related representations, carry sufficient signal to decide when validation should be expressed, offering a speech-first pathway toward more empathetic human–robot interaction. † † address: Graduate School of Informatics, Kyoto University, Kyoto, Japan {pang, fu, gao, kawahara}@sap.ist.i.kyoto-u.ac.jp 1 Introduction Empathy is central to human-robot and human-agent interaction: when systems show that they “get” the user, people disclose more, follow guidance, and build trust. In counseling and mental-health contexts, empathic virtual agents and robots have been linked to higher perceived empathy, trust, and satisfaction, while cautioning that poorly timed or generic displays can feel inauthentic [ 21 , 12 ] . Yet common empathic phrases (e.g., I’m sorry to hear that ) often sound formulaic, especially for users who suppress or struggle to name emotions. In psychotherapy, a more specific technique, emotional validation (validation) , helps. Validation means recognizing, understanding, and explicitly acknowledging another person’s feelings and actions; classic work formalized multiple levels of validation and tied the skill to better alliance and reduced negative affect [ 13 ] . Subsequent studies report that validation predicts reductions in distress and better engagement across clinical and youth settings [ 23 , 3 ] . Crucially, everyday validation arrives through both words and paralinguistic signals: tone, pitch, loudness, timing, and short listener responses (backchannels). Speech science shows that vocal cues carry emotion reliably and shape judgments such as warmth and credibility, even when lexical content is held constant [ 1 , 8 ] . A practical challenge follows: when should a system validate? Over-validating can feel insincere; under-validating misses support. Conversation research indicates that the timing of listener responses is guided by acoustic–prosodic patterns and local interactional signals. In Japanese conversation, short listener responses (aizuchi) occur frequently and at specific positions, with prosodic triggers such as low-pitch regions, pauses, and pre-boundary lengthening; these cues can invite backchannels or signal imminent turn-yielding [ 10 , 11 ] . These findings suggest that appropriate validation timing may be detectable from speech alone, without relying on lexical context. In dialogue systems, empathy has been pursued via commonsense reasoning [ 20 ] , emotion-cause extraction [ 6 ] , and persona conditioning [ 5 , 2 ] . However, only a few works target validation directly, and most are text-based. Recent work frames validation as a pipeline, from validation timing detection, emotion identification, to validating response generation, with initial results in Japanese empathetic dialogue, but it remains open how far non-linguistic speech cues alone can carry the signal [ 16 , 17 ] . At the same time, there is a rich line of speech research on turn-taking and backchannel prediction that leverages prosody and timing for listener behavior, including Japanese conversations and multi-modal settings, suggesting a promising foundation for speech-first validation timing [ 22 , 7 ] . This paper tests the hypothesis that correct validation timing can be detected from speech alone by jointly modeling paralinguistic and emotional information in Japanese emotional dialogue. We isolate the timing decision (“validate now or not”) without using lexical context and propose a two-branch speech approach: (i) continued pre-training of a HuBERT encoder on emotional dialogue to emphasize paralinguistic patterns, and (ii) multi-task fine-tuning of a second HuBERT encoder to learn emotion cues. We then fuse both encoders and fine-tune for the final validation timing detector. Using the TUT Emotional Storytelling Corpus (TESC) for evaluation, we comprehensively compare against strong speech and language baselines and analyze fusion and training strategies. Figure 1: Architecture of the proposed method. The method consists of (1) a multitask speech emotion recognition model, (2) a paralinguistic-aware self-supervised learning model, and (3) feature fusion for the validation timing detection task. 2 Proposed Methods 2.1 Multi-Task Speech Emotion Recognition Current self-supervised speech encoders such as HuBERT and wav2vec 2.0 are pre-trained on general speech and may therefore mismatch our downstream task on emotional spoken dialogue. To reduce this domain gap, we fine-tune a pre-trained model and enrich feature learning with auxiliary tasks under a multi-task learning (MTL) framework. As shown in top part of Fig. 1 , given an input waveform U we take the last-layer hidden sequence \mathbf{f}\!\in\!\mathbb{R}^{t\times d} from the backbone as the latent representation, where t is the utterance length and d the hidden dimension. We apply mean pooling over time (excluding padded frames) to obtain an utterance-level vector \bar{\mathbf{h}}=\frac{1}{t}\sum_{i=1}^{t}\mathbf{f}_{i}\in\mathbb{R}^{d}, (1) followed by dropout ( p{=}0.1 ). Two linear heads map \bar{\mathbf{h}} to emotion and sentiment logits, respectively: \displaystyle\mathbf{z}^{e} \displaystyle=\mathbf{W}^{e}\bar{\mathbf{h}}+\mathbf{b}^{e},\quad\mathbf{p}^{e}=\mathrm{softmax}(\mathbf{z}^{e}), (2) \displaystyle\mathbf{z}^{s} \displaystyle=\mathbf{W}^{s}\bar{\mathbf{h}}+\mathbf{b}^{s},\quad\mathbf{p}^{s}=\mathrm{softmax}(\mathbf{z}^{s}). (3) In this MTL branch, emotion is modeled as a fine-grained 7-way categorical prediction ( anger, disgust, fear, joy, neutral, sadness, surprise ), while sentiment is modeled as a coarse 3-way polarity prediction ( negative, neutral, positive ). Learning both encourages complementary affect representations: the emotion head captures discrete affect categories, whereas the sentiment head provides an auxiliary polarity signal that regularizes the shared utterance representation. To mitigate label skew in emotion classification, we use a class-weighted cross-entropy; sentiment uses standard cross-entropy: \displaystyle\mathcal{L}_{\mathrm{emotion}} \displaystyle=-\!\!\sum_{c=1}^{C_{e}}w^{(e)}_{c}\,y^{e}_{c}\log p^{e}_{c}, (4) \displaystyle\mathcal{L}_{\mathrm{sentiment}} \displaystyle=-\!\!\sum_{c=1}^{C_{s}}y^{s}_{c}\log p^{s}_{c}, (5) where C_{e}{=}7 and C_{s}{=}3 , y^{e},y^{s} are one-hot targets, and w^{(e)}_{c} are inverse-frequency weights estimated on the training split. Instead of fixing the task weights, we learn them end-to-end. Specifically, a scalar logit parameter \tilde{\alpha} is optimized jointly with model parameters and mapped through a sigmoid, \alpha=\sigma(\tilde{\alpha})\in(0,1) . The total loss is \mathcal{L}=\alpha\,\mathcal{L}_{\mathrm{emotion}}+(1-\alpha)\,\mathcal{L}_{\mathrm{sentiment}}, (6) which adapts the balance between tasks during training and removes manual tuning. We initialize \tilde{\alpha}{=}0 so that \alpha{=}0.5 at the start of training. 2.2 Paralinguistic-Aware Self-Supervised Learning Motivated by psychological evidence linking paralinguistics and validation [ 8 ] , we conduct continue pre-training with a HuBERT-style masked unit prediction objective [ 9 ] , as in Fig. 1 bottom. We first derive discrete pseudo-target units by running k -means on MFCC frames and align the resulting unit IDs to the encoder frame rate. Let K be the number of clusters and \{\mathbf{h}_{t}\}_{t=1}^{T} the frame-level hidden states from the encoder. A linear classifier maps each state to unit logits \mathbf{z}_{t}\!\in\!\mathbb{R}^{K} : \mathbf{z}_{t}=\mathbf{W}\mathbf{h}_{t}+\mathbf{b},\qquad u_{t}\in\{1,\dots,K\}. (7) We apply time-span masking on the feature axis by sampling span starts and masking consecutive steps; only masked, non-padded