EVL-ECG: Efficient ECG Interpretation With Multi-Aspect Heterogeneous Knowledge Distillation

Nguyen Hong Dang, Nhi Ngoc-Yen Nguyen, Huy-Hieu Pham · 2026

Read EVL-ECG: Efficient ECG Interpretation With Multi-Aspect Heterogeneous Knowledge Distillation by Nguyen Hong Dang, Nhi Ngoc-Yen Nguyen, Huy-Hieu Pham (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-29977.txt.

Introduction

Dang Nguyen Hong, Nhi Ngoc-Yen Nguyen, and Huy-Hieu Pham shrink heart-signal diagnosis for bedside care, where huge models cost too much. Their EVL-ECG distills cardiac logic across unlike architectures with three tricks: cross-attention lining up, optimal-transport feature matching across leads, and relation matching carrying the teacher's latent reasoning. Tests lift area scores up to about two and a half percent and clinical accuracy about one percent over rivals. The result is a two-billion-parameter heart model small enough for tight clinical settings.

Catalog record

CREATORNguyen Hong Dang, Nhi Ngoc-Yen Nguyen, Huy-Hieu Pham
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE30917
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.29977

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

EVL-ECG: Efficient ECG Interpretation With Multi-Aspect Heterogeneous Knowledge Distillation Nguyen Hong Dang, Nhi Ngoc-Yen Nguyen, Huy-Hieu Pham Abstract High-fidelity ECG interpretation is increasingly reliant on massive foundation models, yet their deployment in clinical edge-care remains hindered by extreme computational demands. While knowledge distillation (KD) is a promising solution, traditional methods fail to capture the complex spatio-temporal dependencies of ECG signals when transferring knowledge across heterogeneous architectures. In this paper, we propose EVL-ECG, a framework specifically designed for cross-architecture distillation of cardiac diagnostic logic. EVL-ECG introduces three ECG-aware innovations: (1) Multi-Head Cross-Attention Alignment, which harmonizes architectural discrepancies to preserve fine-grained morphological features; (2) Optimal Transport-based Visual Feature Matching, utilizing optimal transport to maintain global structural relationships across ECG leads despite mismatched token representations; and (3) Geometric Intra-Architecture Relation Matching, which distills the latent diagnostic reasoning of the teacher model. Evaluations across ECG benchmarks demonstrate that EVL-ECG yields improvements of up to 2.4% AUC and 1.1% clinical accuracy over existing baselines. Notably, EVL-ECG establishes an efficient 2B-parameter ECG foundation model, suitable for resource-constrained clinical environments. 1 Introduction Vision-Language Models (VLMs) have advanced Electrocardiogram (ECG) interpretation from simple classification to generative clinical reporting ( 13 ; 24 ; 14 ; 19 ; 15 ; 35 ) . However, their massive architectures prohibit real-time clinical deployment. While Knowledge Distillation (KD) offers a compression solution ( 31 ; 5 ; 7 ) , distilling from a VLM teacher to a smaller student faces two critical barriers: severe vocabulary mismatch due to tokenizer heterogeneity ( 10 ; 8 ) , and unbalanced visual tokens caused by differing visual encoders. Existing KD methods treat cross-tokenizer and visual-token mismatches in isolation ( 27 ; 29 ; 3 ; 6 ; 8 ; 7 ; 5 ) , restricting the use of modern, efficient small language models as backbones. To bridge this gap, we propose EVL-ECG, a Multi-Aspect Heterogeneous KD framework that employs Multi-Head Cross-Attention to adaptively aggregate dense teacher representations into a cohesive feature space. This architecture utilizes Optimal Transport regularization to maintain vital global spatial and morphological ECG characteristics through soft visual token alignment, while simultaneously applying Geometric Intra-Architecture Relation Matching to distill complex diagnostic logic via distance- and angle-wise potentials, ensuring the student model masters the intricate clinical patterns inherent in ECG interpretation. We also discuss the related works in Appendix A . Our main contributions are: 1. We propose EVL-ECG, a unified heterogeneous knowledge distillation framework that resolves tokenizer and visual-token mismatches between massive teachers and efficient students through a Multi-Head Cross-Attention alignment and Optimal Transport regularization, ensuring the preservation of critical morphological features in ECG images. 2. We introduce a Geometric Intra-Architecture Relation Matching module that distills the teacher’s internal diagnostic logic via distance- and angle-wise potentials, enabling the student model to inherit sophisticated reasoning patterns for complex cardiac analysis while maintaining a resource-friendly size. 3. Extensive evaluations across multiple ECG interpretation benchmarks demonstrate that EVL-ECG significantly outperforms state-of-the-art distillation methods and proprietary VLMs highlighting its robust clinical fidelity and strong generalizability to diverse diagnostic scenarios. 2 Method This section details our proposed EVL-ECG framework in Figure 1 , which facilitates multi-level heterogeneous knowledge distillation between a large teacher and an efficient student VLM. Figure 1 : Overview of the proposed EVL-ECG distillation framework. Our approach is tailored to capture the complex temporal and spatial dependencies of ECG signals. The framework employs multi-head cross-attention to align heterogeneous ECG representations between a large-scale teacher and an efficient student. Furthermore, it integrates OT-based visual matching to preserve the global structural patterns of ECG leads, while Geometric Intra-Architecture Relation Matching distills the underlying diagnostic logic essential for cardiac arrhythmia detection. 2.1 Optimal Transport-based Visual Feature Matching In ECG images, spatial layout intrinsically encodes clinical information such as the standard 12-lead grid where specific patches correspond to distinct cardiac axes. Standard point-wise matching ignores this global spatial topology. To preserve the teacher’s collective geometry of visual descriptors, ensuring the student accurately maps the spatial progression of leads and waveforms without positional confusion, we employ OT regularization. Let \mathbf{t}_{i},\mathbf{s}_{j}\in\mathbb{R}^{D_{s}} denote the L_{2} -normalized visual tokens of the teacher (linearly projected to the student’s dimension) and student, respectively. We treat the V_{t} teacher and V_{s} student tokens as uniform empirical distributions: \mu=\frac{1}{V_{t}}\sum_{i}\delta_{\mathbf{t}_{i}} and \nu=\frac{1}{V_{s}}\sum_{j}\delta_{\mathbf{s}_{j}} . Defining the transport cost as the squared Euclidean distance C_{ij}=\|\mathbf{t}_{i}-\mathbf{s}_{j}\|_{2}^{2} , we solve for the optimal transport plan \mathbf{P}^{\ast}_{\varepsilon}\in\mathbb{R}_{+}^{V_{t}\times V_{s}} that minimizes the entropically regularized Sinkhorn distance: \mathbf{P}^{\ast}_{\varepsilon}=\arg\min_{\mathbf{P}\in\Pi(\mu,\nu)}\langle\mathbf{P},C\rangle-\varepsilon\mathcal{H}(\mathbf{P}) where \Pi(\mu,\nu) is the set of valid couplings with marginals \mu and \nu , and \mathcal{H}(\mathbf{P}) is the entropy of the transport plan. The OT-based distillation loss is then computed as the expected cost under this optimal coupling: \mathcal{L}_{ot}=\sum_{i=1}^{V_{t}}\sum_{j=1}^{V_{s}}P^{\ast}_{\varepsilon,ij}\|\mathbf{t}_{i}-\mathbf{s}_{j}\|_{2}^{2} By minimizing \mathcal{L}_{ot} , the student aligns its visual feature map with the teacher’s distribution, actively preventing the spatial misalignment of critical diagnostic regions such as misinterpreting precordial leads as limb leads. 2.2 Multi-Head Cross-Attention Alignment To bridge the sequence length disparity between the teacher’s dense representations H_{t}\in\mathbb{R}^{B\times L_{t}\times D_{t}} , which capture high-resolution temporal ECG morphologies, and the student’s compact latent space H_{s}\in\mathbb{R}^{B\times L_{s}\times D_{s}} , we employ Multi-Head Cross-Attention (MHCA). Treating the student’s hidden states as queries and the teacher’s states as keys and values allows the student to adaptively isolate and aggregate diagnostically relevant features, such as localized ectopic beats or subtle ST-segment deviations. Rather than enforcing strict positional matching, the student dynamically attends to the most critical ECG segments across the sequence. The aligned teacher representation is obtained by computing \hat{H}_{t}=\text{Concat}(\text{head}_{1},\dots,\text{head}_{h})W^{O} where the student features H_{s} serve as the query and the teacher features H_{t} act as both key and value, optimal h detailed in Table 4 . For each attention head, the operation is defined as \text{head}_{i}=\text{Softmax}\left(\frac{(H_{s}W_{i}^{Q})(H_{t}W_{i}^{K})^{\top}}{\sqrt{d_{k}}}\right)(H_{t}W_{i}^{V}) where W_{i}^{Q} , W_{i}^{K} , and W_{i}^{V} are learnable weight matrices. The final alignment loss ensures the student accurately reconstructs these clinical features by minimizing the mean squared error between its states and the attention-weighted teacher context: \mathcal{L}_{\mathrm{mhca}}=\frac{1}{B\cdot L_{s}}\sum_{b=1}^{B}\sum_{i=1}^{L_{s}}\|H_{s}^{(b,i)}-\hat{H}_{t}^{(b,i)}\|^{2}_{2} Theoretical Motivation. We demonstrate that the attention mechanism mathematically can be interpreted as an Entropic Barycentric Projection under an OT framework. In ECG interpretation, enforcing a strict 1-to-1 alignment is ill-posed when sequences differ in length ( L_{s}\neq L_{t} ) or when clinically relevant waveforms such as P waves, QRS complexes, and ST segments are temporally shifted or vary across leads. Further theoretical details are provided in Appendix B . 2.3 Geometric Intra-Architecture Relation Matching ECG interpretation relies fundamentally on the structural topology and temporal relationships between distinct waveform segments such as P-wave to QRS complex intervals, or ST-segment orientations. While point-wise alignment aids token reconstruction, it overlooks this global diagnostic logic. Inspired by relational KD ( 23 ) , we introduce a Geometric Intra-Architecture Relation Matching module to ensure the student maintains the teacher’s internal geometric reasoning regarding the ECG signal’s morphology. We define two relation potentials for hidden state sequences H : a distance-wise potential \psi_{\mathrm{D}} mean-normalized pairwise Euclidean distance to capture magnitude and interval-based signal relationships, and an angle-wise potential \psi_{\mathrm{A}} pairwise cosine similarity to capture the directional morphology and electrical axis of the ECG features. The distillation loss aligns these structural potentials between the projected teacher \hat{H}_{t} and student H_{s} across batch B and sequence length L_{s} : \mathcal{L}_{k}=\frac{1}{B\cdot L_{s}^{2}}\sum_{b=1}^{B}\sum_{i,j=1}^{L_{s}}\biggl\|\psi_{k}(\hat{H}_{t}^{(b,i)},\hat{H}_{t}^{(b,j)})\\ -\psi_{k}(H_{s}^{(b,i)},H_{s}^{(b,j)})\biggr\|^{2} (1) where

Read the complete text