VCR: Learning Valid Contextual Representation for Incomplete Wearable Signals

Yuxuan Weng, Wenhan Luo, Qijia Shao · 2026

Read VCR: Learning Valid Contextual Representation for Incomplete Wearable Signals by Yuxuan Weng, Wenhan Luo, Qijia Shao (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-18837.txt.

Introduction

Yuxuan Weng, Wenhan Luo and Qijia Shao tackle wearable health sensing when some sensor channels drop out. Rebuilding a whole missing signal can invent details the remaining sensors cannot support. Their VCR framework instead learns representations that survive gaps, splitting each modality into shared meaning and its own quirks. A missing-aware expert network adapts to whichever channels are present, and reconstructing only the shared part of a missing modality avoids hallucinations while beating strong baselines.

Catalog record

CREATORYuxuan Weng, Wenhan Luo, Qijia Shao
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE71912
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.18837

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

VCR: Learning Valid Contextual Representation for Incomplete Wearable Signals Yuxuan Weng, Wenhan Luo, Qijia Shao Abstract Wearable devices enable continuous health monitoring from multimodal signals, but real-world deployment is hindered by limited labeled data and pervasive sensor incompleteness. While large-scale self-supervised pretraining reduces label dependence, most existing methods assume full modality availability. Current approaches for handling modality missingness often reconstruct entire absent signals, which can encourage hallucinating modality-specific details that are not inferable from the observed sensor signals and degrade robustness. We propose VCR, a self-supervised framework that learns to extract valid representations robust to modality missingness. VCR employs an orthogonal tokenizer to enforce strict orthogonal disentanglement by rectifying latent manifolds and applying a geometric projection, separating each modality into shared semantics and modality-specific residuals. This design preserves complete information integrity while serving as a structural foundation for robust learning under modality missingness. The resulting tokens are processed by a missing-aware mixture-of-experts backbone that adapts to varying patterns of modality availability. By constraining the objective to reconstruct only the inferable shared components of missing modalities, VCR effectively mitigates hallucinations of non-inferable modality-specific details. Across multiple health monitoring tasks, VCR consistently improves performance and robustness under full, single-missing, and multiple-missing modality settings compared with strong supervised and self-supervised baselines. 1 Introduction Wearable devices enable continuous and non-invasive monitoring by capturing synchronized motion and physiological signals such as photoplethysmography (PPG), electrodermal activity (EDA), accelerometer (ACC), and skin temperature (TEMP). Beyond basic signal processing (e.g., estimating heart rate from PPG), these modalities support a broad range of downstream health applications, including emotion recognition 36 , 5 , sleep staging 50 , 24 , and activity recognition 39 , 28 . However, reliable generalization in real-world wearable deployments remains challenging due to two coupled factors: (i) label scarcity across diverse tasks 15 , and (ii) pervasive modality missingness 1 1 1 In this paper, “missingness” refers to the complete absence of a modality throughout the given window. caused by hardware constraints and user-dependent effects 31 (e.g., some wearables omit EDA/TEMP entirely, users disabling PPG to save power, and motion/firmware issues can cause ACC streams to drop out). Figure 1 : (a) Performance under full and single-modal missingness (see Experiments for multiple missingness). We report normalized Macro-F1 (value/max) and MAE (min/value) within each dataset to compare methods across tasks and datasets. Left: comparison to supervised baselines trained on labeled data only. Right: comparison to pretrained/self-supervised baselines. Higher is better. (b) Modality-specific information can be non-identifiable from observed sensors. Two activity windows may exhibit similar EDA, ACC, and TEMP signals while showing distinct PPG morphology. Recent progress in self-supervised learning and foundation models 45 , 34 , 52 has made label-efficient wearable intelligence increasingly feasible: models can be pretrained on large-scale unlabeled data streams and transferred to downstream tasks with limited labeled data 31 , 15 . However, most existing models implicitly assume full modality availability during pretraining, thereby limiting their real-world utility under heterogeneous modality missingness. While contrastive learning approaches 40 , 2 , 51 can handle missing modalities by aligning shared representations, their emphasis on cross-modal commonality may suppress modality-specific cues . Conversely, missingness-aware reconstruction 46 often relies on masked reconstruction of the entire missing signal. When the absent modalities contain modality-specific details that are not identifiable from the observed sensors, such objectives can encourage hallucination-like completions and inject spurious information into the learned representation. Meanwhile, task-specific modality-routing methods 48 , 18 may improve a particular downstream task but are not naturally reusable across multiple tasks. These limitations motivate a general-purpose pretrain model that remains valid and robust under train-on-full, test-on-missing scenarios. Achieving this goal requires addressing the following technical challenges. • Non-identifiability of Missing Modality-Specific Information. When a modality is absent, its modality-specific details cannot be reliably inferred from the remaining sensors as illustrated in Figure 1 (b). Nevertheless, reconstruction-based objectives can still incentivize generating them, leading to hallucination-like completions and injecting noise into representations. • Availability-Dependent Cross-Modal Dependencies. Different missing-modality patterns determine which inter-modal correlations are observable, making it challenging for a unified dense architecture to model all modality combinations without interference. • Information Loss Caused by Imbalanced Learning. Wearable modalities differ substantially in sampling rate, variance, and semantic content. As a result, masked reconstruction can be dominated by high-variance modalities and underutilize other modalities, while contrastive alignment can overemphasize shared semantics and underrepresent modality-specific cues. To address these obstacles, we propose VCR, a missingness-robust general-purpose framework for multimodal wearable sensor data, pre-trained on raw PPG, EDA, ACC, TEMP signals. VCR consists of two stages: an Orthogonal Tokenizer that separates each modality into shared and specific components, and a Missing-Aware MoE Backbone that learns contextual representations under diverse modality-availability patterns. Together, these components enable VCR to preserve information under full observations while avoiding unsupported reconstruction when modalities are missing. Our key contributions are as follows: • Validity-Aware Learning: To address the non-identifiability of missing modality-specific information, we propose an orthogonal tokenizer that rectifies encoder representations, applies whitening, and uses a learned orthogonal projection to decompose each modality into modality-shared semantics and modality-specific residuals. During structural missingness, our validity-aware objective reconstructs only inferable shared components of absent modalities, thereby avoiding unsupported modality-specific reconstruction and mitigating hallucination-like completions. • Missing-Aware MoE Backbone: To handle availability-dependent cross-modal dependencies, we introduce a sparse MoE backbone with learnable partition embeddings. This preserves modality/component identity and enables experts to specialize under different observed modality sets. • Two-Stage Information Preservation: To mitigate imbalanced and conflicting SSL objectives, we first train the tokenizer to preserve shared semantics via cross-modal contrast and specific details via intra-modal reconstruction. We then train the backbone with masked latent reconstruction, reducing domination by high-frequency or high-variance raw signals. We evaluate VCR on a comprehensive benchmark spanning three classification tasks (i.e., emotion detection, activity recognition, sleep staging), and one regression task (i.e., \text{VO}_{2} prediction). Across these tasks as shown in Figure 1 (a) , VCR achieves state-of-the-art (SOTA) performance under both full-modality and all missing-modality settings, demonstrating its effectiveness in preserving information integrity and ensuring robustness against modality missingness. We will make the code publicly available upon acceptance to support reproducibility and future research. 2 Method This section introduces VCR. We first formalize the problem and motivate disentanglement (Sec. 2.1 ), then describe the overall framework (Sec. 2.2 ), the orthogonal tokenizer (Sec. 2.3 ), and the missing-aware MoE backbone with training objectives (Sec. 2.4 ). 2.1 Preliminary Problem Formalization. We consider multimodal wearable time-series data collected from M sensing modalities, including PPG, EDA, ACC, and TEMP. After synchronization and non-overlapping windowing, each sample is denoted as X=\{\mathbf{x}^{m}\}_{m}^{M} . Here, \mathbf{x}^{m}\in\mathbb{R}^{T_{m}\times C_{m}} denotes the sequence for modality m , where T_{m} is the number of time steps (the product of window duration and sampling rate) and C_{m} is the number of channels. In actual deployments, not all modalities are always available due to heterogeneous device configurations, sensor failures, or user non-compliance. In this paper, a missing modality refers to the complete absence of an entire modality throughout the current window. We formalize this by defining the observed and missing modality sets as \mathcal{M}_{\text{obs}} and \mathcal{M}_{\text{miss}} , respectively, where \mathcal{M}_{\text{obs}},\mathcal{M}_{\text{miss}}\subseteq\mathcal{M}=\{1,\dots,M\} . Accordingly, we denote the set of observed modalities as \mathcal{X}_{obs}=\{\mathbf{x}^{k}|k\in\mathcal{M}_{obs}\} and the set of missing modalities

Read the complete text