Characterization of GPU TEE Overheads in Distributed Data Parallel ML Training
Jonghyun Lee, Yongqin Wang, Rachit Rajat, Murali Annavaram · 2025
Read Characterization of GPU TEE Overheads in Distributed Data Parallel ML Training by Jonghyun Lee, Yongqin Wang, Rachit Rajat, Murali Annavaram (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-11771.txt.
Introduction
Jonghyun Lee and co-authors measure the cost of secure GPU training in 2025. New graphics chips can run work inside a trusted vault that hides data from the cloud host. That privacy is vital for training models on sensitive data. The price in speed for split training across many chips was not well charted.
The team profiled distributed training inside these vaults in depth. Split training shares gradient updates by a ring method with several stages. Only the chip package is trusted, so data must be guarded each time it leaves. Their measures expose where vault training loses speed and why.
Catalog record
| CREATOR | Jonghyun Lee, Yongqin Wang, Rachit Rajat, Murali Annavaram |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 45 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-11771
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Characterization of GPU TEE Overheads in Distributed Data Parallel ML Training Jonghyun Lee, Yongqin Wang, Rachit Rajat, Murali Annavaram Abstract Confidential computing (CC) or trusted execution enclaves (TEEs) is now the most common approach to enable secure computing in the cloud. The recent introduction of GPU TEEs by NVIDIA enables machine learning (ML) models to be trained without leaking model weights or data to the cloud provider. However, the potential performance implications of using GPU TEEs for ML training are not well characterized. In this work, we present an in-depth characterization study on performance overhead associated with running distributed data parallel (DDP) ML training with GPU Trusted Execution Environments (TEE). Our study reveals the performance challenges in DDP training within GPU TEEs. DDP uses ring-all-reduce, a well-known approach, to aggregate gradients from multiple devices. Ring all-reduce consists of multiple scatter-reduce and all-gather operations. In GPU TEEs only the GPU package (GPU and HBM memory) is trusted. Hence, any data communicated outside the GPU packages must be encrypted and authenticated for confidentiality and integrity verification. Hence, each phase of the ring-all-reduce requires encryption and message authentication code (MAC) generation from the sender, and decryption and MAC authentication on the receiver. As the number of GPUs participating in DDP increases, the overhead of secure inter-GPU communication during ring-all-reduce grows proportionally. Additionally, larger models lead to more asynchronous all-reduce operations, exacerbating the communication cost. Our results show that with four GPU TEEs, depending on the model that is being trained, the runtime per training iteration increases by an average of 8x and up to a maximum of 41.6x compared to DDP training without TEE. I introduction In recent years, machine learning (ML) providers have increasingly relied on cloud computing platforms to deploy large-scale ML workloads on multi-GPU machines for their scalability and cost-effectiveness. However, running ML training on these cloud servers introduces vulnerabilities to proprietary or sensitive data and models, including risks from physical attacks on the underlying hardware and compromised host operating systems. To address these security concerns, various hardware-enforced Trusted Execution Environments (TEEs) have been introduced. Intel Software Guard Extensions (SGX), ARM TrustZone, AMD Secure Encrypted Virtualization (SEV), and Intel Trusted Domain Extensions (TDX). These TEEs enforce isolation by rejecting virtual memory translation requests of unauthorized entities to the guest machine’s memory. Additionally, DRAM encryption and authentication safeguards against potential hardware attacks such as memory bus probing and data replay attacks. Since modern ML workloads prefer to exploit massive parallelism in GPUs, recently, GPUs also started supporting TEEs for private ML training. Notably, solutions such as Graviton [ 28 ] , HIX [ 15 ] , and NVIDIA Confidential Computing (CC) [ 24 ] extend TEE capabilities to commodity GPUs. NVIDIA H100 GPUs introduced the first commercially enabled GPU TEE. NVIDIA CC has demonstrated minimal overhead when data transmission to and from GPU TEEs is limited [ 24 ] . However, modern ML models are often trained using multiple GPUs in the distributed data parallel (DDP) approach where each GPU receives a fraction of the total training data. The ML model itself learns from all the data partitions by frequently synchronizing gradients from all the GPUs and updating the model parameters from the gradients averaged over all the partitions. In the context of secure training using multiple GPU TEEs, this synchronization involves encrypting/decrypting and authenticating gradients before aggregation. Despite the importance of TEE-enabled private training in DDP, the overhead associated with gradient synchronization in GPU TEEs remains underexplored in the existing literature. In this work, we analyze the overhead of private DDP training using multiple NVIDIA H100 GPU TEEs. Fig. 1 : Multi-GPU Confidential Computing system topology. As NVLink and PCI-e are not trusted, secure channel is established via shared key between two entities: CPU-to-GPU (black and blue key) and GPU-to-GPU (red key). Fig. 2 : Computation flow of Distributed Data Parallelism training of each GPU. All-reduce (AR) operation is launched asynchronously overlapped with training backward pass to hide System Topology: Figure 1 illustrates the general multi-GPU system topology for secure ML training in NVIDIA CC. The system consists of a CPU TEE and GPU TEEs connected via PCI-e and NVLink. CPU TEE, implemented using an Intel TDX or AMD SEV-SNP, loads the GPU driver in its secure enclave and ensures secure CPU-GPU data movement using the shared keys between each GPU TEE (shown as GPU-specific black and blue keys). Each GPU TEE ensures isolated computation on the GPU. Multiple GPUs are connected using NVLink, a proprietary NVIDIA interface, for inter-GPU communication. However, NVLink is not trusted and can be tampered with by adversaries. Hence, data transfers between GPU TEEs require senders to encrypt the data and generate a tag before transmitting the data on the NVLink (using the red key shown in figure). Upon arrival of the encrypted data, the receivers must decrypt the data and verify the tag. This additional protection establishes a secure communication channel within the multi-GPU system. While ML libraries, such as PyTorch Distributed Library, provide various parallelization models to enable efficient large-scale ML training, we focus on Distributed Data Parallelism (DDP). DDP, a synchronous distributed training, is among the most widely used parallelism strategies due to its ease of implementation, scalability, and broad applicability across diverse benchmarks [ 18 ] . In DDP, all participating GPUs maintain the same model parameters among all participating devices, using all-reduce operations [ 18 ] to synchronize during backward propagation in each training iteration. In the DDP training process, we assume all the training datasets reside on the cloud CPU TEE before training starts; this is a common practice for cloud training. DDP training process in multi-GPU follows the steps outlined in Figure 2 : 1) The CPU TEE performs preprocessing of the input data 2) The CPU TEE sends the data through the secure channel, where the GPU TEEs receive the encrypted data 3) Each GPU TEE performs the training process 4) GPU TEEs synchronize gradients (via all-reduce) using the NVLink using the secure channel. The secure channels in steps 2 and 4 are TEE overheads to ensure data confidentiality and integrity. In current implementations step 4 (all-reduce) occurs asynchronously during the training backward pass of step 3, overlapping communication overhead with computation. As gradients are computed layer by layer, the communication of one layer’s gradients is overlapped with the computation of the next layer’s gradient. While the overlapping is effective in CC-off (non-private training) settings, inter-GPU communication overheads introduced with CC-on make it impractical to overlap fully with the backward computation in DDP. Figure 2 shows the flow of DDP training of a single GPU in CC-on setting. The bottom figure shows the flow of overall training, and the top figure shows the flow of a single all-reduce operation. As you can see, when data leaves the trusted boundary (outside CPU and GPU packages), data must be encrypted/decrypted and authenticated. TEE overheads of inter-GPU communication: Compared with multi-GPU training without GPU TEEs, GPU TEEs suffer from substantial performance degradation due to additional requirements for secure data transfer. During steps 2 and 4, data is transmitted via PCI-e and NVLink, respectively, both of which are considered untrusted physical links. To ensure data confidentiality and integrity, the senders must