Clin-JEPA: A Multi-Phase Co-Training Framework for Joint-Embedding Predictive Pretraining on EHR Patient Trajectories

Yixuan Yang, Mehak Arora, Ryan Zhang, Baraa Abed, Junseob Kim, Md Hassanuzzaman, Kevin Zhu, Ayman Ali, Chengkun Yang, Victor Moas, Rishikesan Kamaleswaran · 2026

Read Clin-JEPA: A Multi-Phase Co-Training Framework for Joint-Embedding Predictive Pretraining on EHR Patient Trajectories by Yixuan Yang, Mehak Arora, Ryan Zhang, Baraa Abed, Junseob Kim, Md Hassanuzzaman, Kevin Zhu, Ayman Ali, Chengkun Yang, Victor Moas, Rishikesan Kamaleswaran (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-10840.txt.

Introduction

Yixuan Yang and ten colleagues present Clin-JEPA, a 2026 pretraining framework that learns from electronic health record trajectories. One model can then both forecast a patient's course and serve many risk-prediction tasks. Joint-embedding predictive architectures have shone in robotics and vision. Carrying them to medical records has been blocked by unstable training and predictors that never teach the encoder.

A five-phase curriculum warms up the predictor and refines encoder and predictor together. It then aligns the target network, syncs it hard, and finalizes the predictor. Each step tames collapse and drift. Built on a Qwen3-8B backbone, the co-trained model keeps rollouts stable where naive joint training diverges. Clinical machine-learning teams get a single backbone for forecasting and risk tasks without retraining for each job.

Catalog record

CREATORYixuan Yang, Mehak Arora, Ryan Zhang, Baraa Abed, Junseob Kim, Md Hassanuzzaman, Kevin Zhu, Ayman Ali, Chengkun Yang, Victor Moas, Rishikesan Kamaleswaran
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE51943
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.10840

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Clin-JEPA: A Multi-Phase Co-Training Framework for Joint-Embedding Predictive Pretraining on EHR Patient Trajectories Yixuan Yang, Mehak Arora, Ryan Zhang, Baraa Abed, Junseob Kim, Md Hassanuzzaman, Kevin Zhu, Ayman Ali, Chengkun Yang, Victor Moas, Rishikesan Kamaleswaran Abstract We present Clin-JEPA , a multi-phase co-training framework for joint-embedding predictive (JEPA) pretraining on electronic health record (EHR) patient trajectories. JEPA architectures have enabled latent-space planning in robotics and high-quality representation learning in vision, but extending the paradigm to EHR data—to obtain a single backbone that simultaneously forecasts patient trajectories and serves diverse downstream risk-prediction tasks without per-task fine-tuning—remains an open challenge. Existing JEPA frameworks either discard the predictor after pretraining (I-JEPA, V-JEPA) or train it on a frozen pretrained encoder (V-JEPA 2-AC), leaving the encoder unaware of the rollout signal that the retained predictor must use at inference; co-training the encoder and predictor under a shared JEPA prediction objective would supply this grounding, but naïve co-training is unstable, with representation collapse and online/target drift causing autoregressive rollout to diverge. Clin-JEPA ’s five-phase pretraining curriculum—predictor warmup, joint refinement, EMA target alignment, hard sync, and predictor finalization—addresses each failure mode by phase, stably co-training a Qwen3-8B-based encoder and a 92M-parameter latent trajectory predictor. On MIMIC-IV ICU data, three independent evaluations support the framework: (1) latent ℓ 1 \ell_{1} rollout drift uniquely converges ( − - 15.7%) over 48-hour horizons while baselines and ablations diverge ( + + 3% to + + 4951%); (2) the encoder learns a clinically discriminative latent geometry (deteriorating-patient cohorts displace 4.83 × \times further than stable patients in latent space, vs ≤ \leq 2.62 × \times for baseline encoders); (3) a single backbone outperforms strong tabular and sequence baselines on multi-task downstream evaluation. Clin-JEPA achieves mean AUROC 0.851 on ICareFM EEP and 0.883 on 8 binary risk tasks ( + + 0.038 and + + 0.041 vs baseline average). † † Code: https://github.com/YeungYathin/Clin-JEPA 1 Introduction A patient’s stay in the intensive care unit (ICU) is a high-dimensional dynamical system: every hour brings new vital signs, laboratory values, and clinical interventions whose effects on patient state must be accounted for in both forecasting and risk prediction. Existing electronic-health-record (EHR) modelling approaches address only part of this problem. Token-autoregressive language models trained on tokenised EHR sequences [ 23 , 28 , 29 , 21 ] can generate plausible event trajectories, but treat the patient as a sequence of static text events and never explicitly model the underlying continuous physiological state. Per-task foundation models for the ICU [ 6 ] achieve strong task-level performance, but require per-task feature engineering and fine-tuning with no unified representation that supports trajectory simulation. Recent latent world models in vision and robotics offer a promising direction. Joint-Embedding Predictive Architectures (JEPA) [ 20 , 2 , 5 ] pre-train representations by predicting masked or future content in a latent space rather than reconstructing pixels, and V-JEPA 2-AC [ 3 ] retains an action-conditioned predictor at inference to enable latent-space rollout for robotic planning. Adapting this paradigm to clinical EHR—where the analogous goal is a patient-state simulator that consumes free-text observations and interventions—is appealing but non-trivial. Existing JEPA designs leave the inference-time-simulator use case open. I-JEPA [ 2 ] and V-JEPA [ 5 ] discard the predictor after pretraining, so only the encoder is available downstream and trajectory simulation is impossible. V-JEPA 2-AC [ 3 ] retains the predictor but trains it on a frozen pretrained encoder, leaving the encoder unaware of the rollout signal that the retained predictor must use at inference. Co-training the encoder and predictor under a shared JEPA prediction objective would simultaneously close both gaps—producing an encoder whose representations are dynamically grounded for the predictor, and a predictor that operates in a representation space jointly optimized for it. However, naïve co-training is unstable: the encoder is dragged toward representation collapse by an untrained predictor, and the predictor’s autoregressive rollout diverges as it accumulates errors in a moving target latent space. We present Clin-JEPA , a multi-phase co-training framework that closes this gap. Our contributions are: (1) a five-phase pretraining curriculum (predictor warmup, joint encoder–predictor refinement, EMA target alignment, hard sync, and predictor finalization) that stably co-trains a Qwen3-8B-based encoder with a retained latent trajectory predictor on MIMIC-IV ICU data [ 17 ] ; (2) a text-based EHR representation that consumes raw clinical text and serves all downstream tasks from a single set of latent embeddings, requiring no missing-value imputation, no normalization, and no hand-engineered features; and (3) a three-axis empirical evaluation showing that Clin-JEPA uniquely converges over a 48-hour autoregressive horizon, learns a clinically discriminative latent geometry that separates deteriorating from stable patient cohorts, and outperforms strong tabular and sequence baselines on two standard ICU benchmarks. The framework is summarized in Figure 1 . 2 Related Work Clinical foundation models for EHR. Pretrained models for EHR span early encoder-only architectures such as Med-BERT [ 26 ] (structured ICD code sequences) and GatorTron [ 31 ] (clinical-text encoder), generative or autoregressive trajectory models [ 23 , 28 , 19 , 12 ] , and reasoning-enhanced or instruction-tuned LLMs for EHR analysis [ 29 , 21 , 10 ] . Multi-task ICU foundation models [ 6 , 9 , 27 ] achieve strong task-level performance through zero-shot in-context evaluation or per-task fine-tuning; recent empirical analysis [ 7 ] documents cross-institution transferability challenges. Across these architectures, none provides an explicit latent dynamical state representation suitable for autoregressive trajectory simulation. JEPA paradigm. Joint-embedding predictive architectures (JEPA) [ 20 ] were realized for vision in I-JEPA [ 2 ] and V-JEPA [ 5 ] , then extended with a retained action-conditioned predictor for robotic planning (V-JEPA 2-AC [ 3 ] ), to language (LLM-JEPA [ 16 ] ), and to vision-language (VL-JEPA [ 8 ] ). The closest medical adaptation, SMB-Structure [ 1 ] , jointly applies SFT and JEPA over masked future-token spans on longitudinal oncology EHR with an LLM encoder, but uses only the encoder’s embeddings for downstream tasks via linear probes. Latent world models for clinical decision support. A parallel line of work introduces explicit latent world models for ICU and oncology decision support. medDreamer [ 30 ] adopts a Dreamer-style [ 13 ] RSSM trained from scratch with a discrete action space for sepsis policy learning. Latent Physiology as Language [ 22 ] models patients as a continuous-time latent SDE with EHR events as control inputs, trained jointly with reconstruction, masked imputation, and intervention-aware rollout-consistency objectives. CLARITY [ 11 ] pairs a LoRA-adapted MRI foundation encoder with an action-conditioned latent predictor on brain-tumor and breast-cancer MRI cohorts. The Qazi et al. [ 25 ] survey reviews this emerging paradigm. None of the above jointly grounds an LLM encoder against the rollout signal of a retained latent predictor — the gap Clin-JEPA closes with a multi-phase co-training curriculum. 3 Clin-JEPA Framework The Clin-JEPA framework comprises two deployed components illustrated in Figure 1 : an LLM-based encoder that maps EHR text into a continuous latent space, and a Transformer that autoregressively predicts patient trajectories in that latent space. § 3.1 introduces the problem formulation and encoder; § 3.2 details the latent trajectory predictor. Five-Phase Co-Training Curriculum # Phase Encoder LoRA 1 Warmup frozen 2 Co-training trainable \star 3 Alignment frozen

Read the complete text