How Do Electrocardiogram Models Scale?
Jiawei Li, Fabio Bonassi, Ming Jin, Stefan Gustafsson, Johan Sundström, Thomas B. Schön, Antônio H. Ribeiro · 2026
Read How Do Electrocardiogram Models Scale? by Jiawei Li, Fabio Bonassi, Ming Jin, Stefan Gustafsson, Johan Sundström, Thomas B. Schön, Antônio H. Ribeiro (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-17276.txt.
Introduction
Li and colleagues map scaling laws for heart-trace models by training over 120 models, from 20 thousand to 200 million parameters, on a large set of heart-trace records. Models taught with labels hit a data wall on familiar data, while models taught without labels scale steadily with size and data. ResNets beat transformers on parameter efficiency on new data, and label-free training wins on data efficiency and transfer to unseen clinical tasks.
Catalog record
| CREATOR | Jiawei Li, Fabio Bonassi, Ming Jin, Stefan Gustafsson, Johan Sundström, Thomas B. Schön, Antônio H. Ribeiro |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 86368 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.17276
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
How Do Electrocardiogram Models Scale? Jiawei Li, Fabio Bonassi, Ming Jin, Stefan Gustafsson, Johan Sundström, Thomas B. Schön, Antônio H. Ribeiro Abstract While scaling laws have established a fundamental framework for foundation models in natural language processing, their applicability to electrocardiogram (ECG) models remains poorly characterized. Indeed, recent studies do not always yield consistent downstream gains as one increases the model size or pre-training dataset size of ECG models, leaving the exact roles of architectural inductive biases, pre-training paradigms, and expected improvements with size largely unanswered. In this work, we systematically investigate neural and loss-to-loss scaling laws within the ECG domain. By pre-training over 120 120 models (ranging from 20 20 K to 200 200 M parameters) on the large-scale CODE dataset ( 2.3 2.3 M records), we decouple the effects of model architecture (ResNet vs. Transformer) and pre-training paradigm, namely supervised learning (SL) versus self-supervised learning (SSL). We found that ( i ) SL models are data-bottlenecked in-distribution, whereas SSL models scale robustly across both model and data sizes; ( ii ) for out-of-distribution (OOD) generalization, ResNets are 1.3 1.3 to 2.5 2.5 times more parameter-efficient than Transformers, while SSL is up to 16 16 times more data-efficient and achieves up to 7.6 7.6 times higher transfer efficiency than SL on unseen clinical tasks; ( iii ) across the observed scales, ResNet-based models generally achieve the lowest OOD loss, with SSL dominating on unseen clinical tasks and self-supervised Transformers overtaking at very large model sizes. Our results suggest that the path to effective ECG foundation models lies in the strategic alignment of architecture and paradigm rather than brute-force scaling. 1 Introduction Cardiac diseases are the main cause of mortality and morbidity worldwide ( James et al. 2018 ) , and the electrocardiogram (ECG) is a major diagnostic tool. It is low-cost, widely available, and used from basic care to emergency departments. ECG foundation models are deep learning architectures pre-trained on large-scale ECG datasets, typically via self-supervised or supervised learning paradigms ( McKeen et al. 2024 ; Li et al. 2024 ) . This approach has been widely successful in various fields, and for ECGs it has the potential to enable the development of screening and diagnosis models for low-prevalence diseases, where limited labeled data would otherwise hinder training. Recent research has scaled ECG model and dataset sizes to enhance representation learning, with quite diverse choices of architectures and paradigms. As we show in Figure 1 , the current landscape contains models trained using self-supervised (SSL) and supervised learning (SL), mainly based on transformers and convolutional neural networks (CNN), most notably the ResNet architecture ( He et al. 2016 ) . For instance, ECG-FM ( McKeen et al. 2024 ) scales a transformer-based model to 90.9\,\textrm{M} parameters under SSL, while ECGFounder ( Li et al. 2024 ) , a CNN-based model, scales the supervised training set to over 10\,\textrm{M} annotated ECG records. In the literature, scaling laws ( Kaplan et al. 2020 ; Brandfonbrener et al. 2025 ; Hernandez et al. 2021 ) characterize the predictable gains in model performance relative to model size, dataset size, and total compute. However, recent benchmarking ( Al-Masud et al. 2026 ) and large-scale hyperparameter optimization ( Nolin-Lapalme et al. 2026 ) of ECG foundation models suggest that clear scaling is not yet evident in this domain . In fact, Al-Masud et al. 2026 introduced a lightweight model with just 3.8\,\textrm{M} parameters that matches or outperforms pretrained models over 20 times its size. On the other hand, Coppola et al. 2024 showed that a 30\,\textrm{M} -parameter variant of HuBERT achieves superior downstream fine-tuning performance compared to its significantly larger 188\,\textrm{M} counterpart. Al-Masud et al. 2026 concluded that neither increased model size nor larger data scales consistently translate into superior transfer performance, revealing a distinct non-monotonicity in ECG scaling regimes. Similar conclusions were reached for electroencephalograms (EEG) ( Yang et al. 2026 ) . Experiment design often complicates these comparisons. For example, Nolin-Lapalme et al. 2026 observed that SSL outperforms SL when diagnosing new conditions, yet their SL baseline utilized only 1.51\,\text{M} parameters, making it significantly smaller than its SSL counterpart. We argue that this unpredictability arises because existing empirical observations do not clearly and systematically disentangle the effects of architectural inductive biases, hyperparameters, and pre-training paradigm . To address this problem, we systematically investigate the scaling behavior of ECG models by pre-training a suite of over 120 models, ranging from 20\,\textrm{K} to 200\,\textrm{M} parameters, on CODE ( Ribeiro et al. 2021 ) , currently the second-largest public ECG dataset ( 2.3\,\textrm{M} records, 1.6\,\textrm{M} patients). We further conduct experiments on MIMIC-IV ( Gow et al. 2023 ) ( 0.8\,\textrm{M} records, 927 labels) to study the effect of label scaling. Our study considers four representative ECG configurations to disentangle the effects of architectural inductive bias and pre-training paradigms: ResNet-SL, ResNet-SSL (via contrastive learning), Transformer-SL, and Transformer-SSL (via masked autoencoding). By evaluating these models across both in-distribution (ID) and out-of-distribution (OOD) scenarios, we found that the scaling effect is strictly dependent on the coupling between architecture and paradigm: Figure 1 : Average downstream performance across 10 datasets (see Table S.9 for full results). Underlined methods indicate self-supervised foundation models. The results do not show a clear scaling trend; instead, scaling behaviors diverge across different architectures and training paradigms. 1. In the ID scenario, SL models are primarily bottlenecked by dataset size rather than model size, whereas SSL models show no immediate evidence of scaling bottlenecks in either. 2. OOD scaling exhibits an architectural bias, with ResNets yielding 1.3 to 2.5 times higher parameter scaling efficiency than Transformers, and a paradigm bias, where SSL achieves up to 16 times higher data scaling efficiency than SL. 3. Across the observed scales, ResNet-based models generally achieve the lowest absolute OOD loss; in parallel, SSL exhibits superior transfer efficiency ( 7.6\times ) and lower OOD loss on unseen clinical tasks compared to SL, and Transformer-SSL ultimately prevails at large model sizes. 4. While scaling the label space in supervised pre-training improves ID performance, OOD generalization remains sensitive to both architecture and the specific choice of pre-training labels. 2 Related Work Scaling laws. Neural scaling laws ( Kaplan et al. 2020 ) describe how model performance improves as a power law in model size, dataset size, and training compute. Hoffmann et al. 2022 further formulated a parametric scaling law bounded by an irreducible error, establishing that optimal scaling efficiency is achieved when model size and training data size grow proportionally. These principles have been extended to the transfer setting, where pre-training and downstream distributions differ ( Hernandez et al. 2021 ) . Recently, Brandfonbrener et al. 2025 introduced loss-to-loss prediction, demonstrating that shifted power laws relate losses from pre-training to downstream tasks. Beyond the natural language domain, scaling laws have been extended to other domains ( Zhai et al. 2022 ; Yao et al. 2024 ; Chen et al. 2025 ) . For example, Zhai et al. 2022 characterized the performance–compute frontier for Vision Transformers, observing power-law scaling with a double-saturation effect at extreme compute. In the speech domain, scaling behavior has been explored for self-supervised and supervised models, though primarily with respect to model size ( Chen et al. 2025 )