Uncertainty-Aware Cross-Modal Knowledge Distillation with Prototype Learning for Multimodal Brain-Computer Interfaces

Hyo-Jeong Jang, Hye-Bin Shin, Seong-Whan Lee · 2025

Read Uncertainty-Aware Cross-Modal Knowledge Distillation with Prototype Learning for Multimodal Brain-Computer Interfaces by Hyo-Jeong Jang, Hye-Bin Shin, Seong-Whan Lee (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-13092.txt.

Introduction

Hyo-Jeong Jang, Hye-Bin Shin, and Seong-Whan Lee teach brain signal models from vision models in a 2025 paper. Brain wave reads are a base tool for brain computer links, yet noise in signals and labels drags them down. Borrowing knowledge from image models can help, if two gaps are bridged.

The gaps are the mismatch between brain and image feature spaces, and labels that clash between teacher and truth. Their frame aligns features through prototype matching and adds a task specific head to settle label clashes. On a public multi modal set, the method beat single mode and mixed rivals on emotion tasks. The authors pitch it for real brain link use.

Catalog record

CREATORHyo-Jeong Jang, Hye-Bin Shin, Seong-Whan Lee
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE28 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-13092

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Uncertainty-Aware Cross-Modal Knowledge Distillation with Prototype Learning for Multimodal Brain-Computer Interfaces Hyo-Jeong Jang, Hye-Bin Shin, Seong-Whan Lee Abstract Electroencephalography (EEG) is a fundamental modality for cognitive state monitoring in brain-computer interfaces (BCIs). However, it is highly susceptible to intrinsic signal errors and human-induced labeling errors, which lead to label noise and ultimately degrade model performance. To enhance EEG learning, multimodal knowledge distillation (KD) has been explored to transfer knowledge from visual models with rich representations to EEG-based models. Nevertheless, KD faces two key challenges: modality gap and soft label misalignment. The former arises from the heterogeneous nature of EEG and visual feature spaces, while the latter stems from label inconsistencies that create discrepancies between ground truth labels and distillation targets. This paper addresses semantic uncertainty caused by ambiguous features and weakly defined labels. We propose a novel cross-modal knowledge distillation framework that mitigates both modality and label inconsistencies. It aligns feature semantics through a prototype-based similarity module and introduces a task-specific distillation head to resolve label-induced inconsistency in supervision. Experimental results demonstrate that our approach improves EEG-based emotion regression and classification performance, outperforming both unimodal and multimodal baselines on a public multimodal dataset. These findings highlight the potential of our framework for BCI applications. I INTRODUCTION Electroencephalography (EEG) is one of the neural monitoring modalities widely used in brain–computer interfaces (BCIs) [ 1 , 2 ] . As EEG reflects the brain’s electrical activity in real time, it serves as one of the most direct physiological indicators of a user’s cognitive and emotional states [ 3 , 4 ] . However, EEG signals are inherently noisy and uncertain, owing to factors such as sensor artifacts, subject variability, experimental limitations, and the absence of a clearly defined ground truth [ 5 , 6 , 7 ] . These factors collectively give rise to semantic ambiguity into EEG interpretation, which is especially problematic in emotion recognition due to the prevalence of label noise [ 8 , 9 ] . To complement unimodal EEG-based systems, multimodal BCIs have emerged as a promising approach to enhance decoding accuracy by integrating diverse signals [ 10 ] . Recent studies have proposed algorithmic strategies to incorporate additional modalities to enhance EEG signal interpretation [ 11 ] . Various approaches have been explored to exploit these complementary signals, including knowledge distillation (KD), multimodal feature fusion, contrastive learning, adversarial learning, and co-training [ 12 , 13 , 14 ] . Such multimodal learning techniques aim to improve robustness and generalization by integrating different modalities that capture the same underlying user state or intention. Among them, KD [ 15 , 16 ] has emerged as a powerful technique for transferring structured knowledge from a high-quality teacher modality to a lower-quality student modality [ 17 ] . This transfer allows the student model to overcome deficiencies in its input modality and improve performance. Despite the promise of KD, it faces two critical challenges in multimodal learning: modality gap and soft label misalignment. The modality gap [ 18 ] arises from the heterogeneous nature of representations across modalities. Specifically, while conventional knowledge distillation methods have shown effectiveness in unimodal settings, they often fail to generalize across modalities. Soft label misalignment [ 19 ] occurs when noisy labels introduce uncertainty, leading to inconsistencies between the ground truth and the distillation target provided by the teacher model. When learning from both, the student may receive conflicting signals and either down-weight or disregard the teacher’s predictions [ 20 ] . In the present study, we propose a cross-modal KD framework that leverages the visual modality to guide EEG feature learning through semantically informed representation alignment. Our framework incorporates two key components. First, we employ a prototype-based similarity module that bridges the modality gap by projecting EEG and visual features into a shared latent space, reducing representational discrepancies and enabling alignment based on semantic confidence [ 21 ] . Second, our cross-modal KD component addresses soft label misalignment by interpreting the student’s intermediate features through the teacher’s head, enabling the transfer of informative signals that would be disregarded due to prediction-label inconsistencies [ 22 , 19 ] . We evaluate our framework on a public multimodal emotion dataset, demonstrating its superiority over state-of-the-art unimodal and multimodal baselines in both discrete and continuous emotion recognition tasks. Through ablation studies, we analyze the contribution of each loss component to the overall performance enhancement of our proposed framework. By capturing information overlooked by conventional KD, our method improves robustness and semantic reliability in EEG-based emotion recognition. Fig. 1: Overview of the proposed cross-modal KD framework. Intermediate features are extracted from both the teacher and student networks via their respective feature extractors. These features are passed through a prototype-based similarity module, which facilitates evidence-based uncertainty estimation and class-level semantic alignment. Additionally, the intermediate feature from the student is injected into the pre-trained teacher’s task-specific distillation head to obtain predictions \hat{y}_{t|s} , which are then compared with the teacher’s own predictions \hat{y}_{t} . II PROPOSED METHOD II-A Problem Definition In this paper, we address the problem of learning robust EEG representations under label noise and data uncertainty by distilling and transferring knowledge gained from the visual modality. The dataset contains paired EEG and visual samples \{x_{s}^{i},x_{t}^{i},y^{i}\}_{i=1}^{N} . Here, x_{s} and x_{t} represent data from the student (EEG) and teacher (visual) modalities, respectively, and y denotes the corresponding ground truth labels. The feature extractors \mathcal{E} and \mathcal{V} map raw EEG and visual signals into their respective feature spaces. Also, EEG label noise lead to discrepancies between the ground truth labels and the distillation target from the teacher model, which negatively affects the student model’s training. Our goal is to learn a function f_{\theta}:\mathcal{X}_{s}\rightarrow\mathcal{Y} that maps EEG representations to labels while leveraging visual knowledge. This is achieved by minimizing the total loss \mathcal{L} , which integrates all loss components in our framework. II-B Proposed Framework This work aims to mitigate modality discrepancy and label noise in EEG-based learning. To begin with, we introduce the prototype-based similarity module, which builds upon prior work [ 21 ] using prototype-based similarity to improve cross-modal feature alignment. In our framework, this module performs prototype-based similarity alignment guided by Dirichlet-based uncertainty estimation, allowing the model to align modality-specific representations while explicitly accounting for semantic ambiguity. Given EEG samples x_{s} and visual samples x_{t} , we extract their features f_{s} , f_{t} and corresponding embeddings \mathbf{e}_{s} , \mathbf{e}_{t} as follows: f_{s},\mathbf{e}_{s}=\mathcal{E}(x_{s}),\ f_{t},\mathbf{e}_{t}=\mathcal{V}(x_{t}), (1) where \mathcal{E} and \mathcal{V} denote the EEG and visual feature extractors, respectively. We construct a similarity matrix Q using cosine similarity to measure the pairwise relationships between EEG and visual embeddings, which serves as the basis for cross-modal feature alignment and is computed as: Q_{ii}(\mathbf{e}_{s},\mathbf{e}_{t})=\beta\cdot\left({\frac{\mathbf{e}_{s}^{i}}{||\mathbf{e}_{s}^{i}||}}\right)\cdot\left({\mathbf{e}_{t}^{i}\over||\mathbf{e}_{t}^{i}||}\right)^{T}. (2) Here, Q\in\mathbb{R}^{N\times N} denotes the pairwise similarities between EEG and visual samples in a batch, and \beta is a temperature parameter. The diagonal elements Q_{ii} correspond to matching EEG-visual pairs, while the off-diagonal elements Q_{ij} indicate non-matching pairs. To align EEG representations with visual features, we define the similarity-based loss function using a softmax-based contrastive (InfoNCE) formulation: \mathcal{L}_{sim}=-{1\over N}\sum_{i=1}^{N}\log{{\exp(Q_{ii}(\mathbf{e}_{s},\mathbf{e}_{t}))}\over{\sum_{k=1}^{N}\exp(Q_{ik}(\mathbf{e}_{s},\mathbf{e}_{t}))}}. (3) By optimizing \mathcal{L}_{sim} , the model is encouraged to align EEG and visual features by maximizing the similarity between matched pairs. However, this loss does not account for the

Read the complete text