Enhancing Time Series Classification with Diversity-Driven Neural Network Ensembles Javidan Abdullayev, Maxime Devanne, Cyril Meyer, Ali Ismail-Fawaz, Jonathan Weber, Germain Forestier Abstract Ensemble methods have played a crucial role in achieving state-of-the-art (SOTA) performance across various machine learning tasks by leveraging the diversity of features learned by individual models. In Time Series Classification (TSC), ensembles have proven highly effective whether based on neural networks (NNs) or traditional methods like HIVE-COTE. However most existing NN-based ensemble methods for TSC train multiple models with identical architectures and configurations. These ensembles aggregate predictions without explicitly promoting diversity which often leads to redundant feature representations and limits the benefits of ensembling. In this work, we introduce a diversity-driven ensemble learning framework that explicitly encourages feature diversity among neural network ensemble members. Our approach employs a decorrelated learning strategy using a feature orthogonality loss applied directly to the learned feature representations. This ensures that each model in the ensemble captures complementary rather than redundant information. We evaluate our framework on 128 datasets from the UCR archive and show that it achieves SOTA performance with fewer models. This makes our method both efficient and scalable compared to conventional NN-based ensemble approaches. I Introduction Time Series Classification (TSC) is a fundamental problem in machine learning with applications across various domains including healthcare [ 28 ] , human activity recognition [ 25 ] , social security [ 38 ] , remote sensing [ 26 ] , etc.. Increasing availability of large-scale time series datasets, such as the UCR archive [ 7 ] , has led to the development of more advanced classification methods. Recent progress in deep learning has significantly improved TSC performance by leveraging convolutional neural networks (CNNs) which can automatically extract meaningful temporal features [ 23 ] . Currently state-of-the-art (SOTA) deep learning models for TSC including InceptionTime [ 13 ] , H-InceptionTime [ 17 ] and LITETime [ 15 ] achieve high accuracy using ensemble learning. The Time suffix in their names signifies that these models are ensembles of five identical neural networks, each trained independently. These methods combine predictions from their individual members to improve classification performance. However they do not explicitly enforce feature diversity within the ensemble just relying instead on random initialization to introduce variation. This often leads to feature redundancy and limits the potential gains from ensembling. In ensemble learning, diversity among individual models is key for improving generalization [ 9 ] . Traditional approaches such as HIVE-COTE [ 22 ] achieve diversity by ensembling heterogeneous models with different architectures. However, in this work, we take a different approach by focusing on homogeneous neural network ensembles. Our goal is to promote diversity among models with the same architecture by encouraging them to learn distinct feature representations. This is an important but often overlooked aspect of deep ensembles where models trained independently often converge to similar solutions. To address this issue we propose a decorrelated learning framework that explicitly promotes feature diversity within ensembles by penalizing redundant representations. Inspired by knowledge distillation [ 1 ] , we introduce a feature orthogonality loss that forces ensemble models to learn complementary rather than overlapping features. This loss minimizes the cosine similarity between feature vectors produced by different ensemble members. As a result, each model captures unique aspects of the input, reducing redundancy across the ensemble. This enhances generalization without increasing computational complexity or modifying the base model architecture. The overall idea is exemplified using the BirdChicken dataset from the UCR archive, as shown in Figure 1 . On the left, two base LITE models trained separately with different initializations produce highly similar feature maps. Conversely, in our proposal, a second decorrelated LITE model is guided to learn diverse features compared to the first base model trained previously. Figure 1 clearly illustrates on the right side that the decorrelated model learned different features. This differentiation enables the decorrelated ensemble to achieve 100% test accuracy which exceeds the 90% accuracy achieved by the base ensemble. Furthermore, it also exceeds the best performance of SOTA models such as InceptionTime [ 13 ] which achieves a maximum test accuracy of 95% using an ensemble of five models. It suggests that learning diverse features in an ensemble of deep models can result in better generalization and classification performance. Fig. 1: Comparison of ensemble model performances and feature maps on the BirdChicken dataset, from standard and decorrelated training. We assessed our approach on 128 datasets from the UCR archive, demonstrating its effectiveness across diverse real-world time series data. As a result our approach achieves performance comparable to LITETime [ 15 ] but with fewer models which offers a more efficient solution. Our main contributions in this work are: • We propose a novel diversity-driven ensemble framework that explicitly promotes feature diversity and improves deep ensemble effectiveness for TSC. • We conduct a comprehensive empricial validation across 128 UCR datasets, showing that our method achieves SOTA-level performance with fewer models and improves efficiency. • We provide quantitative and qualitative diversity analysis to demonstrate increased feature diversity using Fréchet Inception Distance (FID) scores and t-SNE visualizations of learned convolutional filters. The rest of the paper is organized as follows: Section II provides background information and discusses related work on deep learning-based TSC and ensemble methods. Section III describes our proposed decorrelated learning framework. Section IV presents the experimental setup, dataset details and evaluation metrics, followed by a detailed analysis of results. Finally Section V concludes the paper and discusses potential future directions. II Background and Related Work In this work we focus on TSC task where recent studies demonstrated that deep neural networks (DNNs) leveraging 1D temporal convolutions achieve impressive performance [ 24 ] . Many traditional TSC algorithms especially those not based on deep learning rely heavily on feature engineering as part of the classification task. While this approach has shown strong performance in certain cases [ 23 ] it often requires domain expertise and poses challenges in scalability and automation. On the other hand deep learning models integrate feature extraction and classification into a single pipeline and optimize both jointly. This capability allows them to scale efficiently to large datasets and take advantage of hardware acceleration such as GPUs. SOTA deep learning models for TSC often employ ensemble learning where multiple models of the same architecture are trained independently with different initializations and their predictions are combined [ 15 ] . While ensembling has been shown to improve accuracy [ 9 ] , this approach does not explicitly enforce feature diversity among individual models. As a result, ensembles may suffer from redundant feature representations which limits their overall effectiveness. In this work, we address this issue by focusing on increasing diversity among ensemble members. Specifically, we introduce a decorrelated feature loss that actively encourages diverse feature learning across models during training. Our approach optimizes ensemble diversity by ensuring that individual models learn complementary instead of redundant representations which leads to improved generalization and classification performance. II-A Definitions A univariate time series is defined as an ordered set of numerical values that represents evolution of a specific quantity over time. A time series dataset is denoted as \mathcal{D}={(\mathbf{X}_{i},\mathbf{Y}_{i})}_{i=1}^{N} where N represents the number of samples, \mathbf{X}_{i} is an individual time series and \mathbf{Y}_{i} is its corresponding label vector. The label vector follows a one-hot encoding scheme where \mathbf{Y}_{i}\in\mathbb{R}^{C} represents a class label c from a set of C predefined categories. Time series classification (TSC) task involves assigning a class label to a given sample based on its temporal characteristics. The objective is to train a model that can effectively identify patterns, trends and dependencies within time series data. Formally, the task consists of learning a mapping function f:\mathbf{X}\rightarrow\mathbf{Y} that accurately classifies each input time series \mathbf{X}_{i} into one of the predefined categories in \mathbf{Y}_{i} . II-B Deep Learning for Time Series Classification Time series analysis has been a fundamental area of research for many years with various machine learning techniques applied to tasks like TSC. Early approaches often relied on similarity-base methods such as Dynamic Time Warping (DTW) [ 8 ] between time series or classification models such as Random Forests [ 21 ] and Support Vector Machines (SVMs) [ 4 ] . A key limitation of these traditional methods is that feature extraction is typically treated as a separate process from classification which can lead to information loss and increase the complexity of the overall pipeline. In recent years deep learning has gained significant popularity in the TSC due to its ability to learn complex patterns from raw time series data without requiring manual feature engineering. One of the first deep learning models applied to TSC was the Multi-Layer Perceptron (MLP) [ 12 ] but its fully connected nature made it inefficient in capturing temproal dependencies. A major step forward came with the introduction of 1D CNNs which proved highly effective for extracting meaningful features from time series data. The Fully Convolutional Network (FCN) [ 34 ] was one of the first deep learning models to achieve strong results in TSC. FCN consists of three convolutional blocks, each containing a convolutional layer, batch normalization and activation functions. Unlike standard CNN architectures, FCN does not use pooling layers which allows it to preserve the original time series length and retain temporal relationships more effectively. Following FCN, researchers introduced ResNet for TSC [ 34 ] which applies nine convolutional layers but also incorporates residual connections. These connections improve gradient flow, mitigate information loss and make training deeper models more stable. In 2020, Inception-based model were introduced for TSC in the form of InceptionTime [ 13 ] , drawing inspiration from Google’s Inception v4 [ 29 ] . InceptionTime consists of six Inception modules where each module applies convolutions of different kernel sizes to capture patterns at varying temporal resolutions. Building on InceptionTime, an improved model called Hybrid Inception (H-Inception) [ 17 ] was developed, incorporating custom convolutional filters in the initial layers to enhance feature extraction and classification performance. Recently, LITE (Light Inception with Boosting Techniques) was proposed as a more efficient alternative to Inception-based model [ 15 ] . LITE significantly reduces parameter count to just 2.34% of that of InceptionTime while maintaining competitive classification performance. The model consists of three convolutional layers, combining custom, multiplexed, dilated and depthwise separable convolutions to reduce computational cost while preserving predictive accuracy To achieve SOTA performance, ensemble versions of these models, InceptionTime, H-InceptionTime and LITETime, are widely used in TSC. Each model is trained five times under the same setup but with different initializations and their predictions are combined to produce a final ensemble output. While ensembling improves classification accuracy these approaches do not explicitly enforce diversity among the individual classifiers. Instead they rely solely on random initialization to introduce variation. However, this does not guarantee diverse feature learning and often leads to redundant representations. II-C Ensemble Learning Ensemble learning is a powerful machine learning technique that improves performance by combining predictions from multiple individual models. It leverages their collective strengths to mitigate individual weaknesses and reduce generalization errors [ 9 ] . Common ensemble strategies include bagging where models are trained on different subsets of the training data [ 6 ] , boosting which iteratively focuses on difficult-to-classify samples [ 10 ] and stacking where predictions of base models are combined through a meta-learner [ 36 ] . In deep learning, ensemble approaches have been highly successful in improving generalization and reducing overfitting, particularly in complex tasks such as image recognition and time series classification. The work [ 12 ] demonstrated the effectiveness of deep neural network ensembles for time series classification by highlighting the importance of combining diverse predictions for improved accuracy. The recent model CoCaLite [ 3 ] which utilizes ensemble strategies to balance computational efficiency and predictive accuracy has demonstrated SOTA performance. It is important to note that most existing works overlook critical aspects of feature diversity within ensembles. Various techniques such as decorrelated learning and feature orthogonality have been introduced to explicitly promote diversity during training [ 19 ] . These methods are particularly relevant for deep neural networks where similar architectures and training setups can lead to redundant feature extraction and limits ensemble benefits. For instance, the kernels tailored for time series similarity have been employed with ensemble models to enhance temporal pattern recognition [ 20 ] . The key advantage of ensembling lies in its ability to exploit the diversity of features learned by each individual model. Because of the stochastic nature of deep learning, each model may learn slightly different features which collectively enhance overall performance of the ensemble. In convolutional neural networks this diversity is largely attributed to the variation in features learned by each model. Our experiments have shown that when the models in an ensemble learn very similar features, the performance gains are minimal. This observation is really intuitive and underscores the importance of feature diversity in the success of ensemble learning. II-D Decorrelated Learning To the best of our knowledge, most SOTA deep learning-based TSC models rely on training multiple instances of the same model and ensembling their outputs [ 13 , 17 , 15 ] . Base models within these ensembles are trained independently without mechanisms for sharing information or coordinating their learning. This independent training process often leads the model to converge similar, nearby local minima which results in reduced diversity. Both theoretical and experimental studies suggest that generalization ability of an ensemble can be greatly enhanced if base models are negatively correlated [ 31 ] . In contrast, decorrelated learning explicitly introduces shared training mechanisms that encourage diversity among ensemble members. Instead of training models independently, decorrelated learning ensures that training process of each model is influenced by others in the ensemble. By explicitly minimizing correlations between filters, features or predictions decorrelated learning ensures that each model is guided to learn distinct patterns. This approach helps the features produced by different models to complement one another rather than overlap. This shared training paradigm is particularly valuable in ensemble settings where diversity among individual models plays a curcial role in improving overall performance [ 18 ] . Techniques such as Orthogonality Loss have been developed to enforce feature decorrelation by minimizing cosine similarity between filters [ 37 ] . These methods are particularly relevant for tasks where redundant feature extraction limits performance, as is often observed in deep neural networks trained on datasets with identical architectures. Most existing approaches in CNN-based models apply orthogonality loss to convolutional filters to achieve diversity in the feature space [ 2 ] . In this work, we demonstrate that for TSC tasks applying orthogonality loss to convolutional filters does not lead to meaningful diversity in features. To address this, we propose applying orthogonality loss directly to the feature outputs. This approach ensures a higher level of feature diversity which is critical for improving ensemble performance. Additionally, decorrelation has been shown to be effective in unsupervised representation learning, helping to mitigate representational collapse and improve downstream task performance [ 19 ] . These findings reinforce the value of decorrelated learning in addressing redundancy and enhancing the discriminative power of deep learning models. Our method promotes diversity explicitly at the level of convolutional features, a strategy we have found effective for ensemble learning in TSC tasks. We utilize a diversity-driven auxiliary loss to improve diversity among ensemble models during training. The core idea of our decorrelated learning approach is to train ensemble models sequentially, ensuring that each model learns features distinct from those learned by previously trained models. In this paper, we focus on exploring the impact of decorrelated learning in convolution-based TSC models to deepen our understanding of its effects. III Proposed Method In this section we introduce our diversity-driven ensemble learning framework for TSC. We first describe the transition from filter-level orthogonality to feature-level orthogonality, highlighting its advantages in promoting meaningful diversity among ensemble members. Next we provide a detailed description of our proposed framework, outlining the sequential training process and incorporation of feature orthogonality constraints. Finally we present the mathematical formulation of the feature orthogonality loss which serves as the core of our method. Fig. 2: Proposed decorrelated learning framework for a 2-model ensemble where the decorrelated model is trained with feature orthogonality loss to enhance diversity. III-A Rethinking Diversity: From Filter Orthogonality to Feature Orthogonality The primary objective of this work is to enhance diversity among ensemble models while maintaining the performance of individual models. One possible approach to promote feature diversity is to use techniques such as Deep Negative Correlation Classification (DNCC) [ 40 ] where diversity is encouraged during training by explicitly forcing one model to focus on samples that are misclassified or have low confidence in another model. However, in our experiments with LITE model [ 15 ] we observed that it achieves 100% training accuracy on majority of datasets from UCR archive This indicates that the model effectively fits training data and leaves minimal room for misclassified or low-confidence samples to enhance diversity through misclassification-based methods. Given this limitation we explored alternative methods for promoting diversity, focusing on feature and filter-level diversity. From the literature we identified several works that employ filter orthogonality loss to promote diversity among ensemble models, particularly in image classification tasks [ 33 , 37 , 2 ] . The central idea behind these methods is to enforce orthogonality between convolutional filters, encouraging them to learn distinct representations. From our experiments we realized that filter orthogonality often shifts discriminative patterns rather than creating diverse representations. While filters become orthogonal, feature maps remain similar, limiting the benefits of decorrelation. Thus filter-level orthogonality alone does not guarantee meaningful diversity in ensembles of time series classifiers. To address this challenge, we propose a feature diversity loss function and a diversity-driven optimization strategy aimed at encouraging each model in the ensemble to extract distinct features from the input data. Unlike existing approaches that target filter diversity our method focuses on promoting diversity directly in the feature space itself. Specifically, we introduce an orthogonality loss function that operates on the feature outputs. The goal is to explicitly enforce orthogonality between the feature representations produced by different models in the ensemble. This loss function encourages each model to learn orthogonal features thereby reducing redundancy in the feature space. By encouraging orthogonal feature representations we aim to promote diversity among individual models within the ensemble, ultimately improving the generalization and overall performance. This approach ensures that each model contributes unique information to the ensemble, leading to a more robust and generalizable classification system. III-B Framework Overview In this work we introduce a sequential training framework designed to enhance diversity in neural network ensembles for TSC. Our approach explicitly encourages each model in the ensemble to learn distinct feature representations thereby improving generalization. Unlike conventional ensemble methods where models are trained independently our framework introduces decorrelated models which are explicitly guided to learn features that are orthogonal to those of previously trained models. The training follows a sequential process where each base model in the ensemble (n) is trained to minimize feature redundancy with respect to all earlier models (