Performance of Confidential Computing GPUs

Antonio Martínez Ibarra 1 , Julian James Stephen 2 , Aurora González Vidal 1 , K. R. Jayaram 2 , Antonio Fernando Skarmeta Gómez 1 · 2025

Read Performance of Confidential Computing GPUs by Antonio Martínez Ibarra 1 , Julian James Stephen 2 , Aurora González Vidal 1 , K. R. Jayaram 2 , Antonio Fernando Skarmeta Gómez 1 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-16501.txt.

Introduction

Antonio Martínez Ibarra and four co-authors measured the cost of running AI inference on privacy-shielded graphics processors, in a 2025 study. Confidential computing encrypts data in use, which suits sensitive workloads, but the price in speed was unclear. The team ran relaxed batch inference for several large language models on one high-end NVIDIA chip inside a single virtual machine.

They varied traffic load, traffic shape, scheduling rules, and service targets while models swapped in and out of memory. Without the shield, latency ran 20 to 30 percent lower and service targets were met 15 to 20 percent more often. Throughput without the shield ran 45 to 70 percent higher, and chip use was about half again as high. The gap traces mainly to encryption work when models load, the authors find.

Catalog record

CREATORAntonio Martínez Ibarra 1 , Julian James Stephen 2 , Aurora González Vidal 1 , K. R. Jayaram 2 , Antonio Fernando Skarmeta Gómez 1
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE27 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-16501

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Performance of Confidential Computing GPUs Thanks: This study was supported by grant RYC2023-043553-I, funded by MICIU/AEI/10.13039/501100011033 and ESF+ as well as the HORIZON-MSCA-2021-SE-01-01 project Cloudstars (g.a. 101086248). The publication is also part of NEREIDAS TSI-100120-2024-13 funded by EU NextGenerationEU (PRTR). Antonio Martínez Ibarra 1 , Julian James Stephen 2 , Aurora González Vidal 1 , K. R. Jayaram 2 , Antonio Fernando Skarmeta Gómez 1 Abstract This work examines latency, throughput, and other metrics when performing inference on confidential GPUs. We explore different traffic patterns and scheduling strategies using a single Virtual Machine with one NVIDIA H100 GPU, to perform relaxed batch inferences on multiple Large Language Models (LLMs), operating under the constraint of swapping models in and out of memory, which necessitates efficient control. The experiments simulate diverse real-world scenarios by varying parameters such as traffic load, traffic distribution patterns, scheduling strategies, and Service Level Agreement (SLA) requirements. The findings provide insights into the differences between confidential and non-confidential settings when performing inference in scenarios requiring active model swapping. Results indicate that in No-CC mode, relaxed batch inference with model swapping latency is 20–30% lower than in confidential mode. Additionally, SLA attainment is 15–20% higher in No-CC settings. Throughput in No-CC scenarios surpasses that of confidential mode by 45–70%, and GPU utilization is approximately 50% higher in No-CC environments. Overall, performance in the confidential setting is inferior to that in the No-CC scenario, primarily due to the additional encryption and decryption overhead required for loading models onto the GPU in confidential environments. © 2025 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works. I Introduction Confidential computing addresses the need to protect data in use by performing computations within a hardware-based, attested Trusted Execution Environment (TEE) [ 1 ] . These secure and isolated environments prevent unauthorized access or modification, ensuring data integrity, data confidentiality, and code integrity [ 2 ] . Confidential Virtual Machines (CVMs), a key implementation of confidential computing, operate within such TEEs to shield their code and data from hypervisors, host operating systems, and other components of the TEE’s host environment [ 3 ] . The proliferation of artificial intelligence (AI) has increased the necessity of protecting not only data but also the models themselves, including their architectures, weights, activations, and outputs [ 4 ] . Secure AI inference is particularly challenging because models process sensitive data while requiring substantial computational resources [ 5 ] . The additional overhead of running AI inference within confidential environments—due to encryption, decryption, and attestation mechanisms—further complicates performance optimization. Efficient workload scheduling in CVMs is crucial for balancing security and performance while meeting Service Level Agreements (SLAs). While confidential computing on CPUs has been extensively studied, research on the performance of confidential GPUs and GPU-based TEEs for AI inference remains relatively scarce. GPUs play a critical role in accelerating AI workloads, particularly in deep learning, making their secure deployment a pressing concern. Unlike CPUs, GPUs introduce additional complexities, including memory access patterns, multi-stream parallelism, and specialized encryption mechanisms, which can significantly impact performance in confidential computing environments. Understanding the trade-offs between security and performance in confidential GPU inference is essential for optimizing AI deployments in production environments. To address this gap, this study investigates the performance implications of executing AI inference workloads in Confidential Compute (CC) mode versus Non-Confidential (No-CC) mode on GPUs. Experiments will evaluate key trade-offs, including latency, throughput, SLA attainment, and GPU utilization, using realistic AI workloads. While Large Language Models (LLMs) serve as a representative case study, the insights gained extend to a broader range of deep learning applications in fields such as computer vision, scientific computing, and healthcare. This analysis lays the groundwork for improving scheduling strategies and provisioning AI workloads efficiently in confidential GPU environments. II Background and Related Work II-A Types of Inference Although many machine learning (ML) engineers try to shorten inference time through techniques like quantization, specialized model compilation, etc., in practice, inference can be classified into different types based on response latency requirements: • Hard Real-time Inference: In these applications, failure to perform inference within the required timeframe will likely either be life-threatening or cause a substantial monetary loss. This type of inference involves dedicated hardware which is not shared and is consequently not “interesting” from a performance optimization and scheduling point of view. It is used in latency-critical applications such as autonomous vehicles, high-frequency trading, and satellite systems. • Soft Real-time Inference: Also demands fast response times but can tolerate some level of latency without compromising overall system reliability or safety. This type of inference is widely used in online services like chatbots and applications such as voice assistants and recommendation systems. Not meeting SLAs here can lead to customer dissatisfaction but seldom loss of life or large monetary damages. • Relaxed Inference: Allows for more flexible inference times, ranging from minutes to hours. Relaxed inference is widely used, either (i) when the objective is to keep processing costs low by using spot cloud instances, examples include invoice processing, understanding receipts and processing expense reports (for which there is usually 24+ hours, if not a few days), or (ii) in cases where the same data item is fed to multiple models in sequence, e.g., in medical diagnosis or weather prediction. • Best-effort Inference: Occurs in scenarios where no dedicated resources are allocated for inference. This is commonly found in peer-to-peer systems and community-driven environments, where inference is performed over a wide-area network with variable availability. This type of inference does not have SLAs. II-B Trusted Execution Environments (TEEs) TEEs provide secure execution environments with dedicated memory and storage protections, enabling computations on sensitive data while preventing unauthorized access. CPU-based TEEs, such as Intel TDX and AMD SEV, utilize hardware extensions to manage and encrypt memory, ensuring data protection during execution [ 6 ] . Confidential GPUs, such as the NVIDIA H100 [ 7 ] , extend CPU-based TEEs by enabling secure data transfer and control signal communication between the CPU and GPU. This key difference is shown in Fig. 1 . Without CC mode, data transfers between the CPU and GPU can be leaked by an un-trusted hypervisor. The H100 GPU relies on secure boot [ 8 ] and hardware attestation [ 9 , 10 ] . Secure boot ensures that only verified firmware and software are permitted to initialize the GPU, mitigating risks from malicious code injections. Hardware attestation provides cryptographic proof of the GPU’s secure state, allowing external systems to verify the integrity of the execution environment [ 7 , 11 ] . To enable the CC capabilities of the NVIDIA H100 GPU, specific host side requirements must also be met: (i) CPU TEE: The host system must support CPU TEEs such as AMD SEV-SNP or Intel TDX to securely manage CVMs, and (ii) Secure Hypervisor: A hypervisor with CC support (e.g., KVM) to facilitate CPU, memory and other resource allocations securely. These components work in tandem with

Read the complete text