A mini-batch training strategy for deep subspace clustering networks

Yuxuan Jiang, Chenwei Yu, Zhi Lin, Xiaolan Liu · 2025

Read A mini-batch training strategy for deep subspace clustering networks by Yuxuan Jiang, Chenwei Yu, Zhi Lin, Xiaolan Liu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-19917.txt.

Introduction

Yuxuan Jiang, Chenwei Yu, Zhi Lin, and Xiaolan Liu give a mini-batch strategy for deep subspace clustering in two thousand twenty-five. Subspace clustering groups data that live on low dimensional sheets. Deep forms pair an autoencoder with a self expressive layer. That layer needs the whole data set, which blocks mini-batch runs.

A memory bank keeps global features so batches can stand in. High resolution images thus train at scale in their frame. A decoder free variant swaps autoencoding for contrastive learning. It fine tunes big pretrained encoders with less overhead. Tests match full batch scores and beat top rivals on two sets.

Catalog record

CREATORYuxuan Jiang, Chenwei Yu, Zhi Lin, Xiaolan Liu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE29 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-19917

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

A mini-batch training strategy for deep subspace clustering networks Yuxuan Jiang, Chenwei Yu, Zhi Lin, Xiaolan Liu Abstract Mini-batch training is a cornerstone of modern deep learning, offering computational efficiency and scalability for training complex architectures. However, existing deep subspace clustering (DSC) methods, which typically combine an autoencoder with a self-expressive layer, rely on full-batch processing. The bottleneck arises from the self-expressive module, which requires representations of the entire dataset to construct a self-representation coefficient matrix. In this work, we introduce a mini-batch training strategy for DSC by integrating a memory bank that preserves global feature representations. Our approach enables scalable training of deep architectures for subspace clustering with high-resolution images, overcoming previous limitations. Additionally, to efficiently fine-tune large-scale pre-trained encoders for subspace clustering, we propose a decoder-free framework that leverages contrastive learning instead of autoencoding for representation learning. This design not only eliminates the computational overhead of decoder training but also provides competitive performance. Extensive experiments demonstrate that our approach not only achieves performance comparable to full-batch methods, but outperforms other state-of-the-art subspace clustering methods on the COIL100 and ORL datasets by fine-tuning deep networks. 1 Introduction In recent years, advances in technology have led to the generation of massive high-dimensional data for computer vision and machine learning. Such data pose challenges, including difficulty in analysis and high computational costs. Fortunately, these high-dimensional data often lie in a union of low-dimensional subspaces. For example, face images captured under varying lighting conditions reside in a nine-dimensional subspace Basri and Jacobs 2003 , and the motion trajectories of rigidly moving objects in a video belong to different subspaces of at most three dimensions Tomasi and Kanade 1992 . To effectively uncover these underlying structures, subspace clustering has emerged as a powerful technique for grouping data points based on their intrinsic subspace membership. Over the past several decades, it has had wide applications in computer vision ( Yang et al. 2008 ) Vidal et al. 2008 Li et al. 2015 Wang et al. 2023 , image processing Hong et al. 2006 and systems theory Vidal et al. 2003 . Some classical subspace clustering methods, such as Sparse Subspace Clustering (SSC) Vidal 2009 and Low-Rank Representation (LRR) Liu et al. 2010 , cluster data drawn from multiple low-dimensional linear or affine subspaces embedded in a high-dimensional space. However, real-world datasets usually contain nonlinear relationships. Some works have tried to nonlinearly map input data into latent space for clustering using kernel tricks Patel and Vidal 2014 Patel et al. 2015 or autoencoders, but these two-step approaches separate feature learning from clustering objectives. Consequently, the learned representations are not well-adapted to clustering tasks, limiting their effectiveness. To bridge this gap, Ji et al. 2017 introduce the DSC framework where a self-expressive layer resides between the encoder and the decoder, enabling joint optimization of feature learning and clustering. Despite its conceptual simplicity, this method requires processing the entire dataset in a single forward pass, which necessitates downsampling and full-batch training. These constraints limit its applicability to deeper networks and high-resolution data. Subsequent work Cai et al. 2022 Li et al. 2024 attempts to circumvent this issue by replacing the self-expressive module with alternative mechanisms, though at the cost of increased implementation complexity. In contrast, our work generalizes DSC to mini-batch training via a memory bank that preserves global feature correlations, enabling scalable training with deeper architectures. Conventional deep subspace clustering methods adopt shallow networks trained from scratch, while modern pre-trained models like ResNets He et al. 2016 and Vision Transformers (ViTs) Dosovitskiy et al. 2020 offer richer representations, yet adapting them to subspace clustering is impractical under full-batch training. Although we can build decoders and fine-tune these encoders via mini-batch gradient descent, it yields only marginal improvements, as the features remain unaligned with clustering objectives. Leveraging our mini-batch training strategy, we prove the effectiveness of fine-tuning large-scale pretrained models in subspace clustering. Besides, since the training data for subspace clustering are usually insufficient for training large decoders, we propose a decoder-free architecture that substitutes autoencoding with contrastive learning Chen et al. 2020 , making it more efficient for fine-tuning deep networks. Experiments on benchmark datasets demonstrate that our method achieves state-of-the-art performance among subspace clustering approaches. 2 Related work 2.1 Mini-batch training The adoption of mini-batch optimization in training deep neural networks has been widely studied due to its computational and statistical advantages. Unlike full-batch gradient descent, mini-batch methods partition the dataset into smaller subsets, enabling efficient memory usage and parallelizable computations, which is critical for large-scale vision tasks LeCun et al. 2002 . Empirical evidence suggests that mini-batch training introduces stochasticity, acting as a regularizer to mitigate overfitting while maintaining convergence stability compared to pure batch gradient descent(BGD). For instance, ResNet He et al. 2016 and other vision architectures leverage mini-batch SGD with momentum to achieve robust feature learning across diverse datasets. Further analysis by Smith et al. Smith and Le 2017 highlights the trade-off between batch size and gradient variance, demonstrating that smaller batches enhance generalization by escaping sharp minima—a key factor in vision model robustness. 2.2 Self-expressiveness Self-expressiveness Vidal 2009 denotes the property that a given data point can be represented as a linear combination of points from the same subspace. Suppose we have data \mathbf{X}\in\mathcal{R}^{N\times d} , self-expressiveness can be represented as \mathbf{X}=\mathbf{C}\mathbf{X} , where \mathbf{C} is the self-representation coefficient matrix. By applying l_{1} or l_{2} regularization on \mathbf{C} , \mathbf{C} has a block diagonal structure (after permutation), and each block corresponds to one particular subspace. To avoid each data point being represented by itself, there is another restriction: diag(\mathbf{C})=\mathbf{0} . Once \mathbf{C} is obtained, an affinity matrix can be constructed by |\mathbf{C}|+|\mathbf{C}^{T}| or other heuristics, and used for spectral clustering for final results. Since self-expressiveness only holds for linear subspaces, Ji et al. 2017 propose a deep subspace clustering network (DSC) which consists of an autoencoder and a self-expressive layer. The autoencoder helps to learn a latent space through nonlinear mapping, and the self-expressive layer is used to mimic the self-expressiveness property. Under this framework, we can learn representations that are well-suited for subspace clustering and a self-representation coefficient matrix that can be used for spectral clustering. 2.3 Contrastive learning with a Memory Bank In unsupervised learning, contrastive learning often employs a pretext task known as instance discrimination Wu et al. 2018 . For a given image, two data augmentations are applied to generate two variants: \mathbf{X} , which serves as the anchor, and \mathbf{X}^{+} , which forms a positive pair with \mathbf{X} . Images from different instances, denoted as \mathbf{X}^{-} , are treated as negative samples. The objective of this framework is to minimize the distance between the positive pair representations while maximizing the distance between the anchor and all negative pairs. The performance of contrastive learning benefits from the number of negative samples. To this end, either a large batch size Chen et al. 2020 or a memory bank is typically used Wu et al. 2018 He et al. 2020 . However, using a large batch size often comes with the challenge of excessive memory usage, which is typically unaffordable on most devices. As an alternative, a memory bank that stores the representations of a large number of samples

Read the complete text