Knowledge-enhanced Multimodal ECG Representation Learning with Arbitrary-Lead Inputs
Che Liu, Cheng Ouyang, Zhongwei Wan, Haozhe Wang, Wenjia Bai, Rossella Arcucci · 2025
Read Knowledge-enhanced Multimodal ECG Representation Learning with Arbitrary-Lead Inputs by Che Liu, Cheng Ouyang, Zhongwei Wan, Haozhe Wang, Wenjia Bai, Rossella Arcucci (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-17900.txt.
Introduction
Che Liu and co-authors (2025) teach a model to read heart tracings even when only some leads are available. Standard electrocardiograms use twelve leads, but clinics with fewer resources often record less. Models that pair tracings with reports struggle with tangled medical wording. The authors' framework, called K-MERL, uses a large language model to pull structured facts from each report. A lead-aware encoder learns from whatever leads a tracing has, with random leads hidden during training.
The team tested the model on six outside data sets it had never trained on. It set new best scores for zero-shot classification and probe tests on its learned features. With partial leads, its accuracy beat earlier methods by sixteen percent on average. The authors argue the approach suits real clinics, where equipment varies and expert labels are scarce, because the model arrives able to sort new tracings.
Catalog record
| CREATOR | Che Liu, Cheng Ouyang, Zhongwei Wan, Haozhe Wang, Wenjia Bai, Rossella Arcucci |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 46 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-17900
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Knowledge-enhanced Multimodal ECG Representation Learning with Arbitrary-Lead Inputs Che Liu, Cheng Ouyang, Zhongwei Wan, Haozhe Wang, Wenjia Bai, Rossella Arcucci Abstract Recent advances in multimodal ECG representation learning center on aligning ECG signals with paired free-text reports. However, suboptimal alignment persists due to the complexity of medical language and the reliance on a full 12-lead setup, which is often unavailable in under-resourced settings. To tackle these issues, we propose K-MERL , a knowledge-enhanced multimodal ECG representation learning framework. K-MERL leverages large language models to extract structured knowledge from free-text reports and employs a lead-aware ECG encoder with dynamic lead masking to accommodate arbitrary lead inputs. Evaluations on six external ECG datasets show that K-MERL achieves state-of-the-art performance in zero-shot classification and linear probing tasks, while delivering an average 16% AUC improvement over existing methods in partial-lead zero-shot classification 1 1 1 All data and code will be released upon acceptance. . 1 Introduction Recent advancements in deep learning have enabled automated classification of cardiovascular disease (CVD) using electrocardiograms (ECGs), one of the most crucial diagnostic tools. However, most methods are supervised, requiring large amounts of annotated data, which is costly and demands prohibitively extensive expert effort in annotation ( Liu et al., 2023a ; Huang and Yen, 2022 ) . To address this challenge, self-supervised multimodal learning has recently emerged as an effective approach for learning representative ECG features from accompanied free-text clinical reports ( Li et al., 2023 ; Pham et al., 2024 ; Liu et al., 2024 ) . To this end, MERL ( Liu et al., 2024 ) recently introduced the first comprehensive benchmark using the largest dataset MIMIC-ECG ( Gow et al., ) for pretraining, and six datasets ( Wagner et al., 2020 ; Liu et al., 2018 ; Zheng et al., 2022 ; Zheng et al., 2020 ) for evaluating downstream task performance, including zero-shot classification and linear probing. Despite outperforming signal-only self-supervised approaches, multi-modal approaches, including MERL ( Liu et al., 2024 ) , still have notable drawbacks: They directly align ECG signals with reports, introducing unnecessary noise due to the free-text nature of the reports, and failing to fully exploit the rich cardiac knowledge contained within the text. Additionally, they encode ECG in a lead-agnostic manner, overlooking the unique spatial and temporal characteristics of the individual 12 ECG leads. Moreover, they require all 12 leads to be available as input, limiting their ability to generalize across different lead combinations. This raises important practical concerns since full 12-lead ECG data is not always available in clinical environments due to factors such as patient mobility issues, the need for rapid assessments in emergencies, and limited resource in pre-hospital care environments ( Bray et al., 2021 ; Swor et al., 2006 ; Quinn et al., 2020 ; Nonogi et al., 2008 ; Kotelnik et al., 2021 ; Zhang and Frick, 2019 ; Nonogi et al., 2008 ) . To overcome the challenges listed above, we make the following contributions: (1) We propose a framework dubbed K nowledge-enhanced E CG M ultimodal R epresentation L earning ( K-MERL ), which extracts cardiac-related entities from free-text ECG reports, converting unstructured reports into structured knowledge to enhance self-supervised ECG multimodal learning. To the best of our knowledge, this is the first work to leverage structured cardiac entities extracted from clinical reports to improve ECG multimodal learning. (2) To effectively capture and leverage the lead-specific spatial and temporal characteristics of 12-lead ECGs, we explore various tokenization and positional embedding techniques. In particular, we design lead-specific tokenization and lead-specific spatial positional embeddings , enabling the framework to capture the distinctiveness of each lead. (3) To enable our framework to handle arbitrary combinations of input leads, we introduce a dynamic lead masking strategy. In addition, we propose an independent segment masking strategy to further capture lead-specific temporal patterns. (4) Our K-MERL framework demonstrates superior performance in zero-shot classification and linear probing on multiple downstream datasets in various lead combinations, from a single lead to all 12 leads. Figure 1: Comparison between classical ECG multimodal learning and our K-MERL framework. (a): The classical approaches (e.g., MERL ( Liu et al., 2024 ) ) are suboptimal: they processes all leads in a lead-agnostic manner and naively align ECG signals directly free-text reports. (b): K-MERL introduces lead-specific processing and lead & segment masking to capture spatial-temporal patterns unique to each lead. It also extracts cardiac-related entities from reports as structured knowledge and aligns them with ECG features to enhance multimodal learning, thereby reducing the complexity introduced by the grammatical structure of free-text reports. 2 Method 2.1 Overview To this end, we first utilize a general-purpose open-source large language model (LLM), such as Llama3.1 ( AI@Meta, 2024 ) , without domain-specific fine-tuning, to extract cardiac-related entities from free-text ECG reports. 2 2 2 Entity extraction is inherently simpler than high-level text comprehension in specialized domains, and has been shown effective with general-purpose LLMs ( Zhang et al., 2023 ) . This makes our approach adaptable and well-positioned to benefit from future advancements in LLMs. Additionally, we design a lead-aware ECG encoder with lead and segment masking strategies, allowing the model to handle arbitrary lead inputs while capturing lead-specific spatial-temporal patterns. Our overall framework is illustrated in Fig 1 (b), shown together with the previous state-of-the-art MERL that is based on naive cross-modal contrastive learning ( Liu et al., 2024 ) , in Fig 1 (a). While both approaches utilize contrastive learning with an ECG signal encoder \mathcal{F}_{\textrm{E}} processing signal inputs and a text encoder \mathcal{F}_{\textrm{T}} processing reports, our method introduces substantial innovations, including lead-specific processing, dynamic masking strategies, and the extraction of cardiac-related entities from free-text reports, significantly enhancing ECG multimodal learning. In the following sections, we introduce the model framework and lead-specific processing in Sec 2.2 , followed by the proposed masking strategies in Sec 2.3 . We then describe the pipeline for extracting cardiac-related entities as structured knowledge from ECG reports in Sec 2.4 . Finally, in Sec 2.5 , we explain the knowledge-enhanced ECG multimodal learning process, a synergy of the aforementioned components. 2.2 Lead-specific Processing To begin with, we define the symbols used in our framework: Given a training dataset \mathcal{X} consisting of N ECG-report pairs, we represent each pair as (\mathbf{e}_{i}^{l},\mathbf{t}_{i}) , where \mathbf{e}_{i}^{l}\in\mathcal{E} denotes the raw 12-lead ECG signals for lead l\in\{1,2,3,\dots,12\} of the i -th subject ( i=1,2,3,\dots,N ), and \mathbf{t}_{i}\in\mathcal{T} represents the associated free-text report. We then perform lead-specific processing, as illustrated in Fig 2 . Lead-specific Tokenization. Consider an input ECG signal e_{i}^{l} with 12 leads and a signal length denoted by S . We split the time-series signal into M non-overlapping segments, each segment of length \frac{S}{M} , and perform tokenization for them. In this way, each lead ECG is projected into a sequence of tokens: e_{i}^{l}\left[p_{1}\right],e_{i}^{l}\left[p_{2}\right],e_{i}^{l}\left[p_{3}\right],\ldots,e_{i}^{l}\left[p_{M}\right] where e_{i}^{l}\left[p_{m}\right] corresponds to the ECG token for the m -th segment for lead l . For 12 leads, the total number of tokens is 12\times M . Unlike MERL ( Liu et al., 2024 ) , which generates a single token for a 12-lead ECG temporal segment, we produce tokens separately for each individual lead to capture the lead-specific nature. Lead-specific Spatial Positional Embedding. We apply a learnable