SCPL: Enhancing Neural Network Training Throughput with Decoupled Local Losses and Model Parallelism
Ming-Yao Ho, Cheng-Kai Wang, You-Teng Lin, Hung-Hsuan Chen · 2026
Read SCPL: Enhancing Neural Network Training Throughput with Decoupled Local Losses and Model Parallelism by Ming-Yao Ho, Cheng-Kai Wang, You-Teng Lin, Hung-Hsuan Chen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-00062.txt.
Introduction
Ming-Yao Ho and coauthors propose SCPL, a training method that splits the long gradient chain of backpropagation into short local pieces. Standard end to end training is costly and slow for firms, since gradients must flow through every layer in order. By giving each layer its own supervised contrastive loss, SCPL lets layers compute gradients at the same time, lifting throughput through model parallelism. Tests against backpropagation, early exit, GPipe, and associated learning show gains in speed and quality, the authors report, and they release code for repeat work.
Catalog record
| CREATOR | Ming-Yao Ho, Cheng-Kai Wang, You-Teng Lin, Hung-Hsuan Chen |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 69046 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602.00062
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SCPL: Enhancing Neural Network Training Throughput with Decoupled Local Losses and Model Parallelism Ming-Yao Ho, Cheng-Kai Wang, You-Teng Lin, Hung-Hsuan Chen Abstract Adopting large-scale AI models in enterprise information systems is often hindered by high training costs and long development cycles, posing a significant managerial challenge. The standard end-to-end backpropagation (BP) algorithm is a primary driver of modern AI, but it is also the source of inefficiency in training deep networks. This paper introduces a new training methodology, Supervised Contrastive Parallel Learning (SCPL), that addresses this issue by decoupling BP and transforming a long gradient flow into multiple short ones. This design enables the simultaneous computation of parameter gradients in different layers, achieving superior model parallelism and enhancing training throughput. Detailed experiments are presented to demonstrate the efficiency and effectiveness of our model compared to BP, Early Exit, GPipe, and Associated Learning (AL), a state-of-the-art method for decoupling backpropagation. By mitigating a fundamental performance bottleneck, SCPL provides a practical pathway for organizations to develop and deploy advanced information systems more cost-effectively and with greater agility. The experimental code is released for reproducibility. 1 1 1 https://github.com/minyaho/scpl/ K eywords training throughput \cdot contrastive learning \cdot pipeline \cdot parallel training \cdot neural network 1 Introduction Deep neural networks have become prominent. Although they have excellent predictive power, the practical adoption of these large-scale models in enterprise environments – for applications ranging from decision support and market forecasting to customer relationship management – faces a significant challenge: the immense computational cost and time required for training. This challenge is not only technical; it represents a key managerial barrier that hinders the agile development and deployment of advanced information systems, impacting the return on investment of AI initiatives. Recent industry reports underscore the severity of these financial barriers. For instance, the average cost of computing for enterprise AI is projected to increase by 89% between 2023 and 2025 ( 11 ) . Furthermore, training costs for frontier AI models have historically increased by 2.4 times annually since 2016 ( 7 ) . In this high-cost landscape, the traditional inefficiency of backpropagation, where sequential dependencies leave expensive hardware underutilized during training, becomes an unsustainable waste of resources. The scenario where each layer waits for gradient information from the layers closer to the target before proceeding with the gradient computation is called backward locking , which severely limits the potential speedup of parallel processing, as the sequential nature of gradient calculations becomes a bottleneck for parallel training. This paper proposes an innovative methodology, Supervised Contrastive Parallel Learning (SCPL), to address the issue of backward locking. As a result, SCPL realizes model parallelism , which not only partitions a large neural network into segments processed across multiple computing units (e.g., GPUs) but also enables parallel processing on these devices. This is similar but different from most of today’s model parallelism tools or packages (e.g., the definitions used by Amazon SageMaker 2 2 2 https://docs.aws.amazon.com/sagemaker/latest/dg/model-parallel-intro.html ), which allocate different model components to different GPUs but do not necessarily run these GPUs simultaneously. SCPL decouples the long gradient flow of a deep neural network. The forward path transforms an input x into the corresponding prediction \hat{y} as a usual neural network in this design. However, the gradient flow on the backward path is blocked between different components (layers). Instead of using a global objective, SCPL assigns a local objective to each component and forces each gradient flow to remain within one component. When allocating these local objectives to different GPUs, SCPL can compute the local gradients without waiting for the gradients in the neighboring layers. We conduct experiments on multiple open datasets, including natural language processing and computer vision tasks, using famous network structures such as the LSTM, Transformer, vanilla convolutional neural network, VGG, and ResNet. Our results show that SCPL increases training throughput ( 1.92\times in practice) while maintaining comparable test accuracies compared to backpropagation (BP), Early Exit, GPipe, and Associated Learning (AL) ( 28 ; 13 ) , a state-of-the-art methodology for decoupling BP. We released the code and Docker images with a step-by-step guide for reproducibility. The rest of the paper is organized as follows. In Section 2 , we review previous work on model parallelism and data parallelism. In Section 3 , we introduce SCPL and its properties. Section 4 compares SCPL with BP, Early Exit, GPipe, and AL with respect to their training time and test accuracies. We discuss why SCPL may work, its limitations, and compare and discuss the architectural similarities and differences of SCPL and GPipe in Section 5 . We conclude our contribution in Section 6 . 2 Related Work 2.1 Data Parallelism vs. Model Parallelism Data parallelism (DP) and model parallelism (MP) are two strategies used in distributed deep learning. DP involves distributing batches of training data across multiple devices or processors; each device computes the gradients independently. Devices usually synchronize gradients at the end of every iteration or multiple iterations ( 26 ; 16 ) . DP is well-suited for scenarios where the model can fit within the memory of each device. On the other hand, MP mainly tackles the issue of training models that are too large to fit into the memory of a single device. MP divides the model into segments that are processed on different devices. However, the interdependence of the gradient calculations across layers complicates the parallelization process. The layers in one device may need the gradients from other devices to proceed with its gradient calculations, creating a synchronization bottleneck that limits the potential parallelism. As a result, naïve model parallelism (which will be called “NMP” below) results in poor training efficiency ( 9 ) . 2.2 Model Parallelism Strategies Table 1: A comparison of the properties of the representative related models ( H : number of layers/components) Method Supervised? Parallel model training? Length of gradient flows BP ( 25 ) Y N O(H) AL ( 28 ) Y Y (but not implemented) O(1) GPipe ( 9 ) Y Y O(H) LoCo ( 29 ) N Y (but not implemented) O(1) GIM ( 17 ) N Y (but not implemented) O(1) SCPL (ours) Y Y O(1) A core challenge in model parallelism, backward locking, stems from the inherent characteristics of backpropagation (BP). Therefore, the various methods that attempt to find alternative BP methods could be candidates for achieving model parallelism. Studies in this line include target propagation (TP) ( 15 ; 19 ; 18 ; 3 ) , gradient prediction ( 12 ) , and local objective assignments ( 28 ; 13 ; 10 ) . Although many of these methods do not require gradient computation in training, most still need to update the parameters layer-by-layer, so it is challenging to realize model parallelism. Associated learning (AL) is one of the few methods capable of simultaneously updating parameters in different layers. However, the authors of AL released only a sequential implementation, making it difficult to experiment and verify the parallelization capacity of AL. NMP has a low training efficiency, but MP’s training efficiency can be improved by pipelining, which overlaps GPU computations and minimizes idle time between different stages of computation. The most representative model in this line is Google’s GPipe ( 9 ) ,