Measuring Pre-training Data Quality without Labels for Time Series Foundation Models

Songkang Wen, Vasilii Feofanov, Jianfeng Zhang · 2024

Read Measuring Pre-training Data Quality without Labels for Time Series Foundation Models by Songkang Wen, Vasilii Feofanov, Jianfeng Zhang (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-06368.txt.

Introduction

Wen, Feofanov, and Zhang wrote a 2024 paper on judging training data for time series foundation models. These models aim to handle many tasks, but they need broad pre-training data, which is hard to gather for time series sorting tasks. Labels for judging data quality are often missing. The authors look for a label-free way to grade data sets.

They propose a score called contrastive accuracy, drawn from how the trained model sorts matching and non-matching pairs in its inner space. In their tests, data sets with higher scores led to better results on later tasks. The measure could guide the search for data that lifts a model's general skill. That would save costly full training runs on poor data.

Catalog record

CREATORSongkang Wen, Vasilii Feofanov, Jianfeng Zhang
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE22 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-06368

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Measuring Pre-training Data Quality without Labels for Time Series Foundation Models Songkang Wen, Vasilii Feofanov, Jianfeng Zhang Abstract Recently, there has been a growing interest in time series foundation models that generalize across different downstream tasks. A key to strong foundation models is a diverse pre-training dataset, which is particularly challenging to collect for time series classification. In this work, we explore the performance of a contrastive-learning-based foundation model as a function of the data used for pre-training. We introduce contrastive accuracy, a new measure to evaluate the quality of the representation space learned by the foundation model. Our experiments reveal the positive correlation between the proposed measure and the accuracy of the model on a collection of downstream tasks. This suggests that the contrastive accuracy can serve as a criterion to search for time series datasets that can enhance the pre-training and improve thereby the foundation model’s generalization. 1 Introduction Nowadays, the success of large foundation models ( Bommasani et al., 2021 ) such as GPT-4 ( Achiam et al., 2023 ) or Llama ( Touvron et al., 2023 ) cause dramatic changes in the research and applications. Instead of training a tailored model for a particular task, the foundation model is pre-trained on a collection of datasets with the goal to generalize simultaneously across various downstream tasks either by fine-tuning the model or directly using its output. This workflow effectively simplifies the choice of the model architecture while reducing the requirement for amount of labeled data. The wave of foundation models has now reached the time series domain, including forecasting ( Rasul et al., 2023a ; Woo et al., 2024 ) and classification models ( Lin et al., 2024 ) . When it comes to real deployment of a time series foundation model (TSFM), a very important question is whether the pre-training dataset is sufficiently diverse, so the model generalizes well to new downstream tasks. Usually, this is verified by directly evaluating the performance on several supervised downstream tasks thereby requiring availability of annotated data samples and introducing a high cost for assesing the quality of different pre-trained datasets. Therefore, in this paper, we ask the following research question: Can we evaluate the quality of pre-training data in an unsupervised manner? By focusing on time series classification and contrastive pre-training, we show that it is possible to leverage important information from the representation space learned by the TSFM. More precisely, if new data points are not similar to pre-training data, their embeddings will tend to not satisfy the uniformity property of contrastive learning ( Wang and Isola, 2020 ) , directly impacting the foundation model’s performance. Based on this observation, we introduce a new metric called contrastive accuracy that evaluates how well spread the data points are in the embedding space. We empirically show that the proposed metric positively correlates with the model’s accuracy on the tasks unseen during pre-training, allowing us to use it for measuring the quality of pre-training examples without the need to regularly test the TSFM on supervised downstream tasks. 2 Related Work Time series representation learning has got a high attention in recent years, resulting in a wealth of excellent works. Many of them use the contrastive learning scheme for pre-training, including TS2Vec ( Yue et al., 2022 ) and TS-TCC ( Eldele et al., 2021 ) with a CNN backbone and TF-C ( Zhang et al., 2022 ) with a ResNet backbone, demonstrating good performance on classification datasets, e.g, the UCR collection ( Dau et al., 2019 ) . With the success of large language models (LLMs), popularity of the transformer architecture for time series analysis increases ( Nie et al., 2023 ; Ilbert et al., 2024 ) , while the development of foundation models is becoming a prevalent task. Numerous TSFMs have been proposed recently, including Lag-Llama ( Rasul et al., 2023b ) , Time-LLM ( Jin et al., 2023 ) , One Fits All ( Zhou et al., 2023 ) , Brant ( Zhang et al., 2023 ) , MOIRAI ( Woo et al., 2024 ) . While some of them try to adapt LLM for time series data, others train foundation models from scratch on a large volume of time series data. In our work, we are based on the latter approach, studying how to evaluate the quality of pre-training data, which, to our knowledge, has not been addressed before. Finally, we would like to notice that our framework reminds unlabeled performance estimation ( Donmez et al., 2010 ) where the goal is to predict model’s performance on unlabeled test data. However, most of these approaches focus on single-task classification for computer vision problems ( Hendrycks and Gimpel, 2016 ; Yu et al., 2022 ; Xie et al., 2024 ) , implying a fundamentally different methodology. To the best of our knowledge, time series foundation models have never been considered in this domain before. 3 Proposed Method First, in Section 3.1 , we introduce the setup we are working in. Then, we present briefly the architecture of our foundation model and how it was pre-trained in Section A , respectively. Finally, in Section 3.3 , we propose a new evaluation metric called the contrastive accuracy. 3.1 Problem Setup We consider the task of unsupervised pre-training where an unlabeled pre-training set \mathrm{X}_{\text{0}} is given with m pre-training sequences of length t . The goal is to design a foundation model F:\mathbb{R}^{t}\to\mathbb{R}^{d_{\text{hid}}} that projects any time series \mathbf{x}\in\mathbb{R}^{t} to a discriminative hidden space \mathbb{R}^{d_{\text{hid}}} . In this work, the quality of the foundation model is evaluated on a collection of downstream tasks \mathcal{D}=\{D_{i}\}_{i=1}^{p} , where D_{i} consists of observations \mathrm{X}_{i} and labels \mathrm{Y}_{i} . For each downstream task, we perform a train-test split, append F with a linear head, fine-tune it on the training set and compute the accuracy score on the test set. The performance of F with a pre-training dataset \mathrm{X}_{\text{0}} that is averaged over all downstream tasks is denoted by \mathcal{P}_{\text{test}}(\mathrm{X}_{0}) . Similarly, \mathcal{P}_{\text{train}}(\mathrm{X}_{0}) denotes the average performance when it is evaluated on the training set. 3.2 Architecture and Pre-training Similarly to Nie et al., 2023 and Lin et al., 2024 , we use the ViT ( Dosovitskiy et al., 2021 ) as the backbone, but our implementation slightly differs from theirs. Instead of dividing a sequence into disjoint patches, we employ a single CNN layer allowing them to be overlapped. Similarly to other TSFMs, we reshape any time series to a fixed sequence length equal to 512. More details on the architecture are given in Appendix A . For pre-training of the foundation model, we use the contrastive learning that aims to train a such encoder that outputs similar representations for two random augmentations of the same sample (positive pair) and dissimilar representations for augmentations of two different samples (negative pair). More formally, let \mathcal{T} be a considered space of transformations (augmentations) such that \forall\phi\in\mathcal{T},\mathbf{x}\in\mathcal{X} we have \phi(\mathbf{x})\in\mathcal{X} . In our experiments, we have considered the RandomCropResize with a crop length varying from 70\% to 80\% . To measure similarity of two embeddings, we first project the output of the foundation model F(\mathbf{x}) to a lower dimension using a MLP projector g:\mathbb{R}^{d_{\text{hid}}}\to\mathbb{R}^{d^{\prime}_{\text{hid}}} and then compute the cosine

Read the complete text