Learning Longitudinal Health Representations from EHR and Wearable Data
Yuanyun Zhang, Han Zhou, Li Feng, Yilin Hong, Shi Li · 2026
Read Learning Longitudinal Health Representations from EHR and Wearable Data by Yuanyun Zhang, Han Zhou, Li Feng, Yilin Hong, Shi Li (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-12227.txt.
Introduction
Yuanyun Zhang, Han Zhou, Li Feng, Yilin Hong, and Shi Li learn health traits over time from records plus wearable data. Models trained on health records do well on many clinic tasks. Yet records are sparse, irregular, and driven by visits.
Wearables give dense, steady body signals, but lack labels and clinical sense. Joining the two could give traits that are both rich and grounded. Such traits could aid long-run risk and trend calls. The work builds that joint learning frame. Joint traits could track slow shifts in health state.
Catalog record
| CREATOR | Yuanyun Zhang, Han Zhou, Li Feng, Yilin Hong, Shi Li |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 71 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-12227
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Learning Longitudinal Health Representations from EHR and Wearable Data Yuanyun Zhang, Han Zhou, Li Feng, Yilin Hong, Shi Li Abstract Foundation models trained on electronic health records (EHRs) have shown strong performance across a wide range of clinical prediction tasks, yet they remain fundamentally limited by the sparse, irregular, and event-driven nature of clinical documentation. In parallel, wearable devices generate dense, continuous physiological time series that capture subclinical dynamics but lack semantic grounding and task-specific supervision. Existing approaches typically model these modalities in isolation or combine them through late fusion, failing to exploit their complementary statistical structure. In this work, we introduce a multimodal foundation model that jointly represents EHR data and wearable sensor streams as a unified continuous-time latent process. Our approach combines modality-specific encoders with a shared temporal backbone and is pretrained using self-supervised and cross-modal objectives that align dense physiological signals with sparse clinical events. By explicitly modeling time and enforcing asymmetric cross-modal prediction, the model learns representations that are both temporally coherent and clinically grounded. We evaluate the learned representations across clinical event forecasting, physiological state estimation, and longitudinal risk modeling tasks, using frozen probes to isolate pretraining effects. Across all settings, the proposed multimodal model consistently outperforms strong EHR-only and wearable-only foundation models, including MOTOR, CEHR-BERT, PaPaGei, and HiMAE, with particularly large gains at long prediction horizons and under realistic missing-modality scenarios. Representation analyses further show smoother latent trajectories and improved calibration relative to unimodal baselines. Our results demonstrate that joint EHR–wearable pretraining yields a qualitatively different class of patient representations, bridging sparse clinical observability and continuous physiological dynamics. This work suggests that multimodal foundation models are a necessary step toward faithful, transferable representations of longitudinal health. † † address: University of the Chinese Academy of Sciences, Columbia University 1 Introduction The past year has seen the rapid emergence of foundation models [ 1 , 2 , 3 ] as a unifying paradigm for representation learning across natural language, vision, and increasingly, biomedical data. In healthcare, this shift has been most visible in the success of large-scale models trained on electronic health records (EHRs) [ 4 , 5 , 6 ] , which leverage the longitudinal, high-dimensional, and semantically rich structure of clinical data to learn reusable representations for downstream prediction tasks. In parallel, the proliferation of consumer and clinical-grade wearable devices has produced a complementary stream of high-frequency physiological measurements, capturing aspects of human health that are largely invisible to traditional clinical documentation [ 7 , 8 ] . Despite their shared goal of characterizing patient state over time, EHRs and wearables have largely been modeled in isolation. This separation reflects not only historical data silos, but also deeper methodological challenges arising from mismatched temporal resolutions, observation mechanisms, and inductive biases. At a conceptual level, EHR data and wearable data encode distinct but interdependent views of health. EHRs provide sparse, event-driven observations mediated by clinical workflows, such as diagnoses, laboratory tests, medications, and procedures. These observations are irregularly sampled, often missing not at random, and semantically structured through medical ontologies and billing codes. Wearable data, by contrast, offers dense, continuous-time measurements of physiological signals such as heart rate, activity, sleep, and autonomic dynamics. These signals reflect underlying biological processes at timescales ranging from seconds to days, often preceding clinical manifestation by weeks or months. From an information-theoretic perspective, the two modalities occupy different regions of the signal-to-noise spectrum: EHRs are low-frequency but high-semantic-content measurements, while wearables are high-frequency but weakly labeled proxies of latent health state. Any model that seeks to represent patient trajectories faithfully must therefore reconcile these complementary regimes. The central thesis of this work is that jointly modeling EHR and wearable data within a unified foundation model framework enables the learning of richer, more temporally grounded patient representations than either modality alone. The motivation is not merely additive performance gains on downstream tasks, but a qualitative shift in what aspects of health can be represented. Many clinically meaningful phenomena, such as disease onset, exacerbation, recovery, and treatment response, unfold continuously over time but are only intermittently recorded in the EHR. Wearable signals can provide dense contextualization of these latent transitions, while EHR events anchor them to clinically interpretable states. In this sense, the modalities are not redundant but conditionally informative: wearable data disambiguates the temporal dynamics between EHR events, while EHR data provides semantic supervision that stabilizes and grounds wearable representations. However, integrating these modalities poses nontrivial challenges that preclude straightforward concatenation or late fusion. The data-generating processes differ fundamentally. Let x_{i}(t) denote the latent health state of patient (i) at time (t). EHR observations y_{i}^{\text{EHR}}(t_{k}) occur at discrete, irregular times t_{k} and are conditionally dependent on both x_{i}(t_{k}) and clinical decision processes, while wearable observations y_{i}^{\text{wear}}(t) form a dense stochastic process with different noise characteristics and partial observability. Any joint model must therefore contend with asynchronous sampling, modality-specific missingness, and scale disparities spanning several orders of magnitude in time. Moreover, supervision is weak and heterogeneous: EHR labels are coarse, delayed, and task-dependent, while wearable data is largely unlabeled. These properties challenge conventional supervised learning formulations and instead point toward self-supervised or weakly supervised pretraining objectives. Foundation models offer a natural abstraction for addressing these challenges. By pretraining a large-capacity model on multimodal longitudinal data using self-supervised objectives, one can decouple representation learning from task-specific supervision and enable transfer across a wide range of clinical and physiological tasks. Crucially, a multimodal foundation model can exploit cross-modal prediction as an inductive bias, using one modality to predict or contextualize the other. For example, wearable-derived representations may be trained to predict future EHR events, while EHR context may condition the interpretation of physiological dynamics. This cross-modal alignment encourages the model to learn latent factors that are stable across modalities and temporally coherent, approximating the underlying health state trajectory. The problem we address in this paper is therefore not simply how to combine EHR and wearable features, but how to design a scalable, temporally aware foundation model that respects the statistical structure of each modality while enabling meaningful interaction between them. We argue that success in this setting requires architectural choices that explicitly model time, uncertainty, and modality-specific inductive biases, as well as pretraining objectives that reflect the causal and temporal asymmetries inherent in healthcare data. By framing EHR–wearable integration as a representation learning problem rather than a task-specific engineering exercise, we aim to establish a principled foundation for multimodal patient modeling. In summary, this work is motivated by a gap between the richness of available longitudinal health data and the representational capacity of current models. EHR-only foundation models risk learning abstractions that are temporally underdetermined, while wearable-only models lack semantic grounding and clinical interpretability. A unified foundation model trained jointly on both modalities has the potential to bridge this gap, yielding representations that are both physiologically sensitive and clinically meaningful. The remainder of this paper develops this argument formally, introduces a multimodal foundation model architecture tailored to EHR and wearable data, and demonstrates its empirical advantages across a range of downstream tasks. 2 Related works 2.1 EHR Foundation Models Early EHR foundation models emerged from a straightforward but influential idea: treating structured clinical records as sequences amenable to Transformer-based representation learning [ 9