Wearable Foundation Models Should Go Beyond Static Encoders

Yu Yvonne Wu, Yuwei Zhang, Hyungjun Yoon, Ting Dang, Dimitris Spathis, Tong Xia, Qiang Yang, Jing Han, Dong Ma, Sung-Ju Lee, Cecilia Mascolo · 2026

Read Wearable Foundation Models Should Go Beyond Static Encoders by Yu Yvonne Wu, Yuwei Zhang, Hyungjun Yoon, Ting Dang, Dimitris Spathis, Tong Xia, Qiang Yang, Jing Han, Dong Ma, Sung-Ju Lee, Cecilia Mascolo (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-19564.txt.

Introduction

Wu and co-authors argue that wearable foundation models must move past static encoders. Current models map short sensor windows to fixed labels and do well on narrow tasks. They fail on chronic conditions that unfold over months or years. Three shifts are needed, the authors say. Data must become rich, long-term, and multimodal. Models must reason over long personal histories, not snapshots. Inference must become agentic, supporting plans and interventions under uncertainty. The goal is anticipatory health support.

Catalog record

CREATORYu Yvonne Wu, Yuwei Zhang, Hyungjun Yoon, Ting Dang, Dimitris Spathis, Tong Xia, Qiang Yang, Jing Han, Dong Ma, Sung-Ju Lee, Cecilia Mascolo
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE41568
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.19564

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Wearable Foundation Models Should Go Beyond Static Encoders Yu Yvonne Wu, Yuwei Zhang, Hyungjun Yoon, Ting Dang, Dimitris Spathis, Tong Xia, Qiang Yang, Jing Han, Dong Ma, Sung-Ju Lee, Cecilia Mascolo Abstract Wearable foundation models (WFMs), trained on large volumes of data collected by affordable, always-on devices, have demonstrated strong performance on short-term, well-defined health monitoring tasks, including activity recognition, fitness tracking, and cardiovascular signal assessment. However, most existing WFMs primarily map short temporal windows to predefined labels via static encoders, emphasizing retrospective prediction rather than reasoning over evolving personal history, context, and future risk trajectories. As a result, they are poorly suited for modeling chronic, progressive, or episodic health conditions that unfold over weeks, months or years. Hence, we argue that WFMs must move beyond static encoders and be explicitly designed for longitudinal, anticipatory health reasoning . We identify three foundational shifts required to enable this transition: (1) Structurally rich data , which goes beyond isolated datasets or outcome-conditioned collection to integrated multimodal, long-term personal trajectories, and contextual metadata, ideally supported by open and interoperable data ecosystems; (2) Longitudinal-aware multimodal modeling , which prioritizes long-context inference, temporal abstraction, and personalization over cross-sectional or population-level prediction; and (3) Agentic inference systems , which move beyond static prediction to support planning, decision-making, and clinically grounded intervention under uncertainty. Together, these shifts reframe wearable health monitoring from retrospective signal interpretation toward continuous, anticipatory, and human-aligned health support. † † affiliationnotice: Equal contribution 1 Introduction Figure 1 : Overview of a Wearable Foundation Model . The pipeline consists of three main stages: (a) Data Collection, involving heterogeneous physiological signals and contextual metadata; (b) Multimodal Pretraining, which utilizes self-supervised learning and cross-modality alignment to produce robust latent representations; and (c) Inference, where the model is applied to downstream tasks. Recent advances in wearable sensing have enabled a new class of machine learning systems for continuous, everyday health monitoring. Modern wearables, including smartwatches and fitness trackers, continuously collect multimodal physiological and behavioral signals, such as cardiovascular activity (ECG/PPG), motion (IMU), respiration, and ambient temperature, often over months or years. When paired with foundation models trained on large-scale data, these systems, which we refer to as Wearable Foundation Models (WFMs) , have demonstrated strong analytical performance on a range of well-scoped analytical tasks, including human activity recognition, sleep staging, and population-level health monitoring ( 43 ; 60 ) . For example, SensorLM ( 73 ) , trained on population-scale Fitbit data, exhibits strong zero-shot generalization for activity recognition. In parallel, large language model (LLM)-based health-assistants, such as ChatGPT Health 1 1 1 https://openai.com/index/introducing-chatgpt-health and Claude Healthcare 2 2 2 https://www.anthropic.com/news/healthcare-life-sciences , have begun integrating wearable signals into conversational systems intended to provide health-related guidance when clinicians are unavailable. Despite this progress, most existing WFMs remain fundamentally confined to static encoders, which frame wearable health modeling as a form of static pattern recognition over short temporal windows ( 1 ) . Specifically, they map short segments of physiological and behavioral data to predefined labels or scalar targets, such as activity classes, sleep stages, or fitness scores. Such a static paradigm closely mirrors paradigms that have proven effective in vision and speech recognition, where inputs are typically treated as independent, stationary, and semantically complete within short temporal windows ( 40 ; 60 ) . However, when applied to wearable health, this framing directly limits the system’s ability to support core health objectives, including longitudinal health tracking, personalized baseline estimation, and reasoning about gradual change, recovery, or disease progression. Importantly, these limitations are not isolated deficiencies of individual models or datasets; they are systematic consequences of a shared static encoder framing that permeates the entire WFM pipeline, spanning data collection, modeling, and inference. At the data level, the dominance of such static paradigm is reflected in how most wearable datasets are constructed. Most open-source wearable datasets are organized around short temporal windows, isolated sensing modalities, and outcome-conditioned labeling schemes, in support of narrowly defined downstream tasks ( 53 ; 28 ) . Although large volumes of passive sensor data may be collected, richer contextual information, such as behavior, environment, device usage, or self-reported state, is often sparse, inconsistently integrated, or entirely unavailable. Such contextual factors are critical for causal and situational grounding, enabling interpretation of why physiological patterns arise rather than merely what patterns are present ( 2 ) . As a result, WFMs trained on these datasets are structurally biased toward learning correlations, rather than reasoning about evolving, personalized health states over time. Moreover, many population-scale multimodal and longitudinal wearable datasets remain proprietary, limiting reproducibility, benchmarking, and cross-dataset evaluation ( 43 ; 14 ) . Consequently, existing wearable datasets largely support WFMs that function as passive observers, optimized for short-term pattern recognition rather than longitudinal health reasoning. At the modeling level, the static paradigm is reinforced by how WFMs represent multimodal and temporal information. Most dominant approaches inherit these dataset-level assumptions by treating inputs as independent temporal segments and often training separate foundation models for individual sensing modalities ( 20 ) . More recent work has explored foundation models spanning multiple sensor modalities and proposed a range of multimodal pretraining strategies ( 43 ; 40 ; 14 ) . However, these methods remain architecturally tied to fixed set of input modalities and temporal resolutions, typically relying on data- and task-specific multimodal fusion mechanisms. As a result, existing WFMs tend to internalize modality- and resolution-specific patterns during pretraining, making them difficult to generalize to heterogeneous sensor configurations, missing modalities, or variable sequence lengths encountered in real-world deployments. These architectures also struggle to natively incorporate contextual information, often neglecting it altogether or deferring it to ad hoc late fusion with language models. More fundamentally, the dominance of short-window, cross-sectional modeling prevents WFMs from effectively leveraging longitudinal health trajectories, undermining the primary advantage of wearable sensing: ubiquitous, long-term, everyday monitoring. Furthermore, at the inference level, most existing WFMs remain confined to functioning as discriminative predictors, mapping incoming signals to predefined labels or scores, rather than supporting agentic inference that reasons adaptively over evolving observations and history ( 70 ) . Under this framing, health monitoring is treated as a static prediction problem, in which outputs such as risk scores or activity classes are generated independently at each time step. While effective for benchmarked prediction tasks, this formulation provides limited support for real-world health scenarios that require more than pattern detection, specifically, reasoning about temporal evolution, accumulating evidence across observations, and determining when additional sensing, feedback, or intervention is warranted ( 69 ) . As a result, this predictive-only formulation constrains downstream capabilities such as adaptive intervention planning, personalized behavioral recommendations, and longitudinal decision support, thereby motivating a shift from static prediction toward agentic inference in wearable health systems. We argue that the dominant limitation of existing WFMs is not model capacity, but their shared reliance on a static encoder that frames wearable health as the retrospective mapping of short-term sensor windows to predefined targets. The next generation of foundation models for wearable health must move beyond short-horizon static encoders and be explicitly designed for longitudinal, anticipatory health reasoning. Moving beyond static encoders requires coordinated shifts across the full WFM pipeline, spanning data, modeling, and inference: • Structurally rich data , emphasizing

Read the complete text