Learning Transferable Sensor Models via Language-Informed Pretraining

Yuliang Chen, Arvind Pillai, Yu Yvonne Wu, Tess Z. Griffin, Lisa Marsch, Michael V. Heinz, Nicholas C. Jacobson, Andrew Campbell · 2026

Read Learning Transferable Sensor Models via Language-Informed Pretraining by Yuliang Chen, Arvind Pillai, Yu Yvonne Wu, Tess Z. Griffin, Lisa Marsch, Michael V. Heinz, Nicholas C. Jacobson, Andrew Campbell (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-11950.txt.

Introduction

Chen and co-authors introduce SLIP, a framework for sensor models that transfer across setups. Existing sensor-language models accept only fixed channel counts at fixed sampling rates, so new devices break them. SLIP pairs contrastive alignment with sensor-conditioned captioning, reusing a pretrained language model via cross-attention. A flexible patch embedder absorbs new channel counts and rates at test time, no retraining needed. Across 11 data sets it leads on retrieval, captioning, and question answering, with 77.14% average probing accuracy.

Catalog record

CREATORYuliang Chen, Arvind Pillai, Yu Yvonne Wu, Tess Z. Griffin, Lisa Marsch, Michael V. Heinz, Nicholas C. Jacobson, Andrew Campbell
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE63738
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.11950

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Learning Transferable Sensor Models via Language-Informed Pretraining Yuliang Chen, Arvind Pillai, Yu Yvonne Wu, Tess Z. Griffin, Lisa Marsch, Michael V. Heinz, Nicholas C. Jacobson, Andrew Campbell Abstract Modern sensing systems generate large volumes of unlabeled multivariate time-series data. This abundance of unlabeled data makes self-supervised learning (SSL) a natural approach for learning transferable representations. However, most existing approaches are optimized for reconstruction or forecasting objectives and often fail to capture the semantic structure required for downstream classification and reasoning tasks. While recent sensor–language alignment methods improve semantic generalization through captioning and zero-shot transfer, they are limited to fixed sensor configurations, such as predefined channel sets, signal lengths, or temporal resolutions, which hinders cross-domain applicability. To address these gaps, we introduce SLIP ( S ensor L anguage- I nformed P retraining), an open-source framework for learning language-aligned representations that generalize across diverse sensor setups. SLIP integrates contrastive alignment with sensor-conditioned captioning, facilitating both discriminative understanding and generative reasoning. By repurposing a pretrained decoder-only language model via cross-attention and introducing an elegant, flexible patch-embedder, SLIP supports different temporal resolutions and variable-length input at inference time without additional retraining. Across 11 datasets, SLIP demonstrates superior performance in zero-shot transfer, signal captioning, and question answering. It achieves a 77.14% average linear-probing accuracy, a 5.93% relative improvement over strong baselines, and reaches 64.83% accuracy in sensor-based question answering. All code and datasets are publicly available at https://github.com/yuc0805/SLIP . 1 Introduction Ubiquitous sensors continuously generate large volumes of multivariate time series data, motivating self-supervised pretraining as a way to learn transferable representations without costly annotations. Recent work shows that such pretraining can improve performance across diverse downstream tasks spanning health, activity recognition, and urban environment monitoring ( 40 ; 43 ; 18 ) . Alongside these efforts, general-purpose time series foundation models trained on heterogeneous sensor corpora using reconstruction or regression objectives, such as Chronos-2 ( 2 ) , have emerged. While effective for forecasting, these models primarily capture local temporal continuation and often fail to encode the semantic structure required for downstream classification and reasoning tasks (Figure 1 ). By contrast, sensor-specific pretraining frameworks tailored to individual modalities ( e.g. , ECG or PPG) achieve strong in-domain performance ( 31 ; 37 ; 26 ) , but typically struggle to generalize across different types of sensors or tasks. Recently, large language models (LLMs) have showcased strong semantic abstraction and generalization, particularly in zero-shot and few-shot settings. Consequently, recent work explores aligning LLMs with time-series and sensor representations to overcome the limitations of reconstruction-based or sensor-specific pretraining. For instance, HealthLLM ( 15 ) reformulates sensor signals as text to directly leverage pretrained LLMs; however, this often leads to substantial semantic and temporal information loss. To better preserve signal structure, methods such as Time-LLM ( 13 ) and ChatTS ( 52 ) map the time series into the LLM embedding space via projection layers and rely on the language model for contextual inference. A complementary line of work uses contrastive cross-modal architectures to directly align sensor and text modalities, enabling zero-shot retrieval alongside generation ( 28 ; 59 ) . Despite progress in sensor–language alignment, most methods tie the sensor encoder to a fixed input specification, such as a predefined channel set or temporal resolution, limiting transfer when sensor configurations change and necessitating retraining. For instance, SensorLM ( 59 ) is limited by its fixed 26-channel, 1440-minute pretraining, which hinders transfer to varying sensor layouts or temporal resolutions. To address these challenges, we propose SLIP, a unified language-informed sensor encoder trained via contrastive alignment to support heterogeneous sensor configurations, including multiple modalities and various temporal resolutions, and enable generalization to downstream tasks across different sensor domains. SLIP departs from prior sensor–language models in three key ways. First, to ensure broad cross-domain coverage, we pretrain on a mixture of heterogeneous time-series datasets, moving beyond the single-device specialization common in prior work. Second, to handle structural variations in sensor inputs, we introduce FlexMLP , a weight-sharing patch embedding mechanism that allows the model to dynamically adapt to different temporal resolutions without requiring retraining. Finally, instead of pretraining a decoder-only model ( 52 ; 16 ) , SLIP repurposes a pretrained decoder-only language model into an encoder–decoder architecture by decoupling it into a text encoder and a generative decoder, and extending the decoder with cross-attention to condition generation on sensor representations to enable unified sensor language understanding and open vocabulary generation across diverse downstream tasks. We summarize these capability differences against forecasting foundation models and recent sensor language models in Table 1 . Figure 1 : Illustrated example of the forecasting–classification gap . Chronos-2 achieves accurate forecasting on UCI-HAR with low error (MSE = 0.96), yet its learned representations lead to incorrect activity classification (walking downstairs vs. upstairs). This example illustrates that SSL-based models optimized for forecasting do not necessarily learn semantic representations that support downstream classification and understanding. Toward sensor–language models that capture semantic structure across heterogeneous sensor configurations, our contributions are as follows: (1) Unified language-aligned sensor modeling: We introduce SLIP, a unified sensor model that aligns heterogeneous multivariate time series with language, enabling a broad range of sensor–language tasks. Using the proposed FlexMLP , a lightweight mechanism for handling diverse sensor inputs, and repurposing decoder-only text models for efficient training, SLIP improves efficiency while maintaining cross-domain adaptability. (2) Comprehensive multi-domain evaluation: We evaluate SLIP across 11 diverse sensor datasets spanning human activity recognition, clinical diagnosis, stress prediction, and urban sensing, demonstrating consistent improvements in linear-probing classification, with an average accuracy of 77.15% compared to the strongest baseline, Normwear, at 72.82%, and competitive performance relative to supervised baselines at 76.2%. (3) Open-vocabulary reasoning and generation: We show that SLIP adapts effectively to open-vocabulary downstream tasks, with SLIP SFT achieving strong sensor question answering performance of 64.83% average across four benchmarks and generating high-fidelity sensor captions with a BERTScore of 0.887. (4) Curated sensor–language pretraining data: To support sensor–language alignment at scale, we curate a dataset of 600K sensor–caption pairs spanning over 1 billion time points across diverse sensing domains including health, environment, Internet of Things, energy, and transportation, which we will release upon acceptance together with model weights and code to support further research. Table 1 : Capability comparison of sensor text modeling approaches. We compare prior studies and SLIP across key capabilities, including temporal resolution adaptive sensing to handle different input sequence length and frequency, sensor-text retrieval to capture semantic meaning, sensor question answering (QA) , and open source availability. Study Resolution Adaptive Retrieval QA Open Source Chronos ( 3 ) ✗ ✗ ✗ ✓ Chronos2 ( 2 ) ✗ ✗ ✗ ✓ Sundial ( 22 ) ✗ ✗ ✗ ✓ SensorLM ( 59 ) ✗ ✓ ✗ ✗ Normwear ( 25 ) ✗ ✓ ✗ ✓ ChatTS ( 52 ) ✗ ✗ ✓ ✓ OpenTSLM ( 16 ) ✗ ✗ ✓ ✓ SLIP (ours) ✓ ✓ ✓ ✓ Figure 2 : Sensor-Language Informed Pretraining (SLIP) Architecture. 2 Related Work Time series and sensor foundation models. Sensor waveform signals derived from wearable devices are inherently time series data. Early efforts to learn transferable representations of such signals typically rely on large-scale self-supervised pretraining, using objectives such as masked reconstruction ( 9 ; 29 ) or contrastive

Read the complete text