Revisit Time Series Classification Benchmark: The Impact of Temporal Information for Classification
Yunrui Zhang ✉, Gustavo Batista, Salil S. Kanhere · 2025
Read Revisit Time Series Classification Benchmark: The Impact of Temporal Information for Classification by Yunrui Zhang ✉, Gustavo Batista, Salil S. Kanhere (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-20264.txt.
Introduction
Yunrui Zhang, Gustavo Batista, and Salil S. Kanhere question a basic belief about sorting time series data. Their two thousand twenty-five paper tests the UCR archive, the standard benchmark for this task. The field assumes that the order of values over time is what sets time series apart. The authors scramble that order and watch what happens to accuracy.
For a large share of the datasets, scrambling changed almost nothing. Many sets are tabular in nature, or rest on tabular traits, not on timing. That flaw can make timing-focused methods look better or worse than they are. The team builds UCR Augmented, a fresh benchmark that rewards true use of time order. Tests across seven families of classifiers showed clear gaps on it.
Catalog record
| CREATOR | Yunrui Zhang ✉, Gustavo Batista, Salil S. Kanhere |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 31 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-20264
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Revisit Time Series Classification Benchmark: The Impact of Temporal Information for Classification Yunrui Zhang ✉, Gustavo Batista, Salil S. Kanhere Abstract Time series classification is usually regarded as a distinct task from tabular data classification due to the importance of temporal information. However, in this paper, by performing permutation tests that disrupt temporal information on the UCR time series classification archive, the most widely used benchmark for time series classification, we identify a significant proportion of datasets where temporal information has little to no impact on classification. Many of these datasets are tabular in nature or rely mainly on tabular features, leading to potentially biased evaluations of time series classifiers focused on temporal information. To address this, we propose UCR Augmented, a benchmark based on the UCR time series classification archive designed to evaluate classifiers’ ability to extract and utilize temporal information. Testing classifiers from seven categories on this benchmark revealed notable shifts in performance rankings. Some previously overlooked approaches perform well, while others see their performance decline significantly when temporal information is crucial. UCR Augmented provides a more robust framework for assessing time series classifiers, ensuring fairer evaluations. Our code is available at https://github.com/YunruiZhang/Revisit-Time-Series-Classification-Benchmark . 1 Introduction Classification is one of the most successful tasks in machine learning, making it unsurprising that time series classification is also among the most extensively researched topics in time series analysis. Each year, dozens of methods are proposed, and hundreds of application papers are published. However, considering that most time series classification methods focus on univariate and fixed-length time series, the question arises: What distinguishes time series classification problems from their tabular counterparts? We argue that time series classifiers should ideally be capable of exploiting the correlations or patterns between data points while also tolerating misalignment across different instances to improve classification accuracy. Most time series classification methods uncover temporal information by extracting relevant subsequences interpreted as “shapes” or “features”. This includes many state-of-the-art techniques such as the shapelet transform classifier [ 9 ] , the random convolutional kernels classifier [ 4 ] , and feature extraction approaches such as Catch22 [ 12 ] . The proposed methods are benchmarked on many time series classification datasets, usually from the UCR time series classification archive [ 3 ] . Nonetheless, despite a strong emphasis on extracting and utilizing temporal information for time series classification, limited research has focused on the significance of temporal information as a contributing factor. To address this gap, we propose a test to evaluate the contribution of temporal information in time series classification datasets by disrupting their temporal structure. By comparing the performance of time series classifiers on the original and disrupted datasets, we find that removing temporal information does not significantly impact classification for a large portion of datasets in the UCR archive. This is primarily because these datasets are tabular in nature or too well-segmented, reducing the reliance on temporal information for classification. The observation that temporal information is not a critical feature for classification in these datasets suggests that the current benchmark is sub-optimal for evaluating a classifier’s ability to extract and utilize temporal information—the main focus of time series classification. To address this, we propose a new benchmark UCR Augmented, based on the UCR time series classification archive [ 3 ] , designed to emphasize the importance of temporal information and overcome the limitations of the original archive. Using this benchmark, we evaluate seven categories of time series classifiers representing diverse approaches and analyze their performance. Our findings reveal notable shifts in the relative rankings of these classifiers, with some previously overlooked methods performing strongly under the new benchmark. In contrast, tabular and phase-dependent methods that performed well on the original benchmark show significantly weaker performance on the proposed benchmark. The contributions of this paper can be summarized as follows: 1. Proposing a test for identifying datasets where temporal information does not significantly contribute to classification. 2. Identifying a significant proportion of the UCR archive datasets where temporal information has minimal impact and analyzing the cause. 3. Introducing a new benchmark that evaluates the classifier’s ability to extract and utilize the temporal information. 4. Benchmarking major categories of time series classifiers on the proposed benchmark and analyzing changes in their relative performance compared to the UCR archive to provide insights. 2 Related Work Time series classification assigns discrete labels to data sequences, with temporal order in varying indexes being crucial features, setting it apart from tabular data classification. The UCR time series classification archive [ 3 ] has significantly contributed to the field’s growth by providing a standard benchmark. This archive includes 128 heterogeneous univariate time series datasets from various application domains, with 112 having equal sequence lengths. The current research in time series classification typically introduces new methods and assesses their performance using the UCR time series classification archive [ 3 ] . This includes shape-based methods that employ different techniques for extracting and comparing shapes for classification [ 9 ] and feature-based approaches that convert time series data into features [ 12 ] . All these methods emphasize the extraction of temporal information from time series data. However, no prior work has examined the importance of temporal information for classifying time series in the benchmark archive. Bagnall et al. [ 2 ] benchmarked various classifiers on the UCR time series archive, finding that some standard tabular data classifiers outperformed the DTW-1NN algorithm, which accounts for temporal information. Dhariyal et al. [ 6 ] found that tabular methods could outperform time series classifiers on a subset of the UCR archive. Henderson et al. [ 8 ] showed that features like mean and standard deviation performed well on specific datasets. These findings, where classifiers ignoring temporal information still achieve good results, made us question the universal importance of temporal information in time series classification. Hu et al. [ 10 ] shows that many time series datasets are unrealistically well-aligned and lack non-representative subsequences, unlike real-life tasks. 3 Preliminaries Definition 1 Time Series A time series is an ordered set of n real-valued variables. For a time series X it can be defined as X=x_{1},\ldots,x_{n} . Definition 2 Subsequence For a time series X of length n , a subsequence S_{z} of X is a sub-interval of X of length m that m<n with the starting position of z . S_{z}=x_{z},\ldots x_{z+m} for 1\leq z\leq n-m+1 Definition 3 Temporal Information For a time series X and all possible subsequences, the temporal information \mathcal{T} for each data point s_{t} in a subsequence S is the conditional probability p(s_{t}\mid s_{1\ldots t-1,t+1\ldots m}) where m is the length of the subsequence. As discussed in the introduction, time series classification differs from tabular data classification due to the critical role that temporal information plays in classification. As explained in Definition 3 , temporal information can be seen as the correlation between data points in subsequences, regardless of the starting point of the subsequence. In practice, temporal information is often interpreted as a unique shape or feature of the subsequence, irrespective of its index. 4 Testing the Impact of Temporal Information In this section, we propose the Temporal Information Removal Test to assess whether temporal information is a contributing factor for each