ITGPT: Generative Pretraining on Irregular Timeseries

Antoine Honoré ⋆ , Ming Xiao ⋆ · 2026

Read ITGPT: Generative Pretraining on Irregular Timeseries by Antoine Honoré ⋆ , Ming Xiao ⋆ (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-16069.txt.

Introduction

Honoré and Xiao introduce ITGPT, an attention-based model for mixed-type time series sampled at irregular intervals with missing values. Such gaps are common in health care and predictive maintenance. Trained with self-taught and generative goals, it reached top results on a health-monitoring task and a maintenance task. It needed no resampling, feature fusion, or explicit imputation.

Catalog record

CREATORAntoine Honoré ⋆ , Ming Xiao ⋆
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37006
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.16069

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ITGPT: Generative Pretraining on Irregular Timeseries Thanks: This work was partially supported by the Wallenberg AI, Autonomous Systems and Software Program (WASP) funded by the Knut and Alice Wallenberg Foundation. The computations were enabled by the Berzelius resource provided by the Knut and Alice Wallenberg Foundation at the National Supercomputer Centre. Correspondence: [email protected]. Antoine Honoré ⋆ , Ming Xiao ⋆ Abstract Timeseries regression models often struggle to leverage large volumes of labeled multimodal data, particularly when the data are irregularly sampled or contain missing values. This is common in domains like healthcare and predictive maintenance, where data are collected from unreliable sources, and labeling requires expert knowledge or costly equipments. Transformer-based large language models have proven effective on structured data such as text through self-supervised learning (SSL) and generative pretraining (GPT) frameworks. However, such models lack the flexibility to efficiently process irregularly sampled multimodal timeseries data. In this paper, we introduce ITGPT, an attention-based architecture designed for handling multimodal, irregularly sampled timeseries by allowing training with both SSL losses and GPT-like objectives. We evaluate its performance on a healthcare task with the TIHM dataset, and a predictive maintenance task with the CompX dataset. Our results demonstrate that ITGPT achieves state-of-the-art performance without requiring resampling, feature fusion or explicit data imputation. Furthermore, when labels are scarce, ITGPT effectively leverages unlabeled data through SSL and GPT training, outperforming the purely supervised approach. This represents an important step towards efficiently using large and unstructured timeseries datasets for practical inference tasks. The code is available on GitHub: https://github.com/antoinehonore/itgpt . I Introduction Multimodality arises when multiple sources of information are used to characterize a system [ 13 ] . In many practical cases, sources might be de-synchronized, with different sampling frequencies and missing data. For instance, in predictive maintenance (PdM) the quality of a mechanical piece must often be established from sensors with various sampling frequencies, e.g. noise and vibration sensors [ 14 ] . Similarly in the medical field, a patient’s health status must be established using information from fast or slow sources, for instance electrocardiograms (up 500 samples per second) combined with blood gas analysis (one sample per day) [ 10 ] . Additionally, in these contexts, labels about the health status of a patient or a mechanical piece might require expert knowledge and/or expensive sensors which limits their availability for supervised learning schemes. Current models handle irregularly sampled multimodal data with computationally expensive or overly simplistic resampling, imputation and feature fusion approaches. These approaches lack the ability to predict future input data and thus, only allow supervised learning frameworks. Current approach for processing multimodal data often consider use cases related to image, text or speech processing. Attention-based models such as Transformers are established as strong models that allow training with few labels, using self-supervised learning (SSL) or generative pretraining (GPT) frameworks. These frameworks are key towards prediction models utilizing large amount of unlabeled data in PdM or healthcare contexts [ 12 , 7 ] . SSL is a machine learning paradigm that leverages unlabeled data to learn representations of input data by generating supervisory signals from the data itself [ 4 ] . Unlike traditional supervised learning, SSL does not require manual annotations, making it highly scalable and particularly valuable for domains where labeled data is scarce or expensive to obtain. A common approach involves predicting parts of the data from other parts, e.g. predicting the future of a timeseries from its past. SSL has been key for advancing domains such as computer vision and natural language processing, often serving as a pretraining step before fine-tuning on downstream tasks. The success of SSL reflects its ability to uncover structure and semantics in raw data without external supervision. GPT is a strategy in which models are initially trained in an unsupervised manner on large datasets, in order to learn the distribution of the input data, typically using SSL objectives. This method became prominent with the introduction of models where the transformer architecture was pretrained to predict the next token in a sequence, capturing a wide range of linguistic patterns and world knowledge [ 9 ] . After pretraining, the model can be fine-tuned on specific supervised tasks with relatively little additional data, leveraging the rich representations learned during the generative phase. This two-step process has set new benchmarks and forms the backbone of many state-of-the-art natural language processing systems such as ChatGPT, Mixtral and DeepSeek. In the context of irregularly sampled multimodal data, there are currently no model to efficiently leverage unlabeled data for learning in SSL or GPT frameworks. I-A Related works The problem of handling irregularly sampled multimodal data was addressed with Kalman filters in the context of linear state space models [ 3 ] . These approaches provide efficient closed form inference models, but rely on strong knowledge of the linear processes and measurement systems. Recurrent neural networks with input space augmentation was shown promising for handling unimodal irregularly sampled signals [ 1 ] . This nonetheless might lead to a too large input space dimension. Other RNNs with neural ODEs for modeling continuous time were also shown relevant, but computationally expensive. Other kernel-based Gaussian processes were also investigated, but scale quadratically with the number of training samples [ 2 ] . ITNet [ 5 ] can be used to produce a multivariate data timeseries with arbitrary time sampling instants. This output time series can then be used as an estimator for some quantity of interest in a supervised learning framework. However, this often requires large quantities of accurately labeled data thus ignoring possibly large datasets of unlabeled data. While this is straight to do on regularly sampled timelines, there is currently no standard method for multimodal and irregularly sampled timeseries data. I-B Contributions Our contributions are as follows: 1. We propose a framework called ITGPT which extends on ITNet to allow GPT-like objectives with irregularly sampled multimodal input data. 2. We evaluate our model on two recent datasets for a healthcare and predictive maintenance tasks. 3. We study our model when varying key hyperparameters: the model depth and the mixing layers. 4. We show experimentally that our model benefits from SSL or GPT training frameworks when a few labeled samples are available for training. II Methods In this section we start by introducing the timeseries regression problem and the multimodal input data in II-A . For completeness we introduce the ITNet model in II-B . The ITGPT architecture is finally described in II-C . We use the notation \forall N\in\mathbb{N},\quad[N]=\{1,\dots,N\} . II-A Problem formulation We introduce the problem of timeseries regression based on multimodal and irregular timeseries. Let M\in\mathbb{N} the number of observed modalities. Suppose that we have access to N\in\mathbb{N} independent realizations of the modalities of a system, in addition to a target timeseries. For an observation i\in[N] , the data of modality m\in[M] are denoted: \tilde{X}_{m}^{(i)}=[\tilde{\mathbf{x}}_{m,1}^{(i)},\dots,\tilde{\mathbf{x}}_{m,L_{m}^{(i)}}^{(i)}]^{\top}\in\mathbb{R}^{L_{m}^{(i)}\times d_{m}}, (1) where (L_{m}^{(i)},d_{m})\in\mathbb{N}^{2} denote respectively the number of samples and the data dimension, and \tilde{\mathbf{x}}_{m,t}^{(i)}\in\mathbb{R}^{d_{m}} . In addition, we assume that we have access to a sorted sequence of sampling timestamps for each modality and observation: \boldsymbol{\tau}_{m}^{(i)}=[t_{m,1}^{(i)},\dots,t^{(i)}_{m,L_{m}^{(i)}}]^{\top}\in\mathbb{R}^{L_{m}^{(i)}}. (2) Importantly, all timestamps across modalities have the same time reference, i.e. identical timestamps encountered in different modalities refer to the same time

Read the complete text