Story of Two GPUs: Characterizing the Resilience of Hopper H100 and Ampere A100 GPUs

Shengkun Cui, Archit Patke, Hung Nguyen, Aditya Ranjan, Ziheng Chen, Phuong Cao, Gregory Bauer, Brett Bode, Catello Di Martino, Saurabh Jha, Chandra Narayanaswami, Daby Sow, Zbigniew T. Kalbarczyk, Ravishankar K. Iyer · 2025

Read Story of Two GPUs: Characterizing the Resilience of Hopper H100 and Ampere A100 GPUs by Shengkun Cui, Archit Patke, Hung Nguyen, Aditya Ranjan, Ziheng Chen, Phuong Cao, Gregory Bauer, Brett Bode, Catello Di Martino, Saurabh Jha, Chandra Narayanaswami, Daby Sow, Zbigniew T. Kalbarczyk, Ravishankar K. Iyer (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-11901.txt.

Introduction

Shengkun Cui and co-authors compare two generations of AI chips under real supercomputer strain in a paper from 2025. Delta, a large AI system, runs over a thousand A100 and H100 GPUs. Two and a half years of logs cover nearly twelve million GPU hours. Error patterns at that scale guide both buyers and builders.

The newer H100's memory failed more often per chip, about three times the error rate of the A100. Its larger memory outruns its own recovery tricks. Core hardware reliability, though, improved markedly in the H100. On both chips, errors often killed whole jobs for lack of recovery paths. The study's story is mixed progress: stronger cores, weaker memory, fragile jobs.

Catalog record

CREATORShengkun Cui, Archit Patke, Hung Nguyen, Aditya Ranjan, Ziheng Chen, Phuong Cao, Gregory Bauer, Brett Bode, Catello Di Martino, Saurabh Jha, Chandra Narayanaswami, Daby Sow, Zbigniew T. Kalbarczyk, Ravishankar K. Iyer
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE63 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-11901

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Story of Two GPUs: Characterizing the Resilience of Hopper H100 and Ampere A100 GPUs Shengkun Cui, Archit Patke, Hung Nguyen, Aditya Ranjan, Ziheng Chen, Phuong Cao, Gregory Bauer, Brett Bode, Catello Di Martino, Saurabh Jha, Chandra Narayanaswami, Daby Sow, Zbigniew T. Kalbarczyk, Ravishankar K. Iyer Abstract . This study characterizes GPU resilience in Delta 1 1 1 Delta is an HPC system operated by the National Center for Supercomputing Applications (NCSA) at the University of Illinois Urbana-Champaign. , a large-scale AI system that consists of 1,056 A100 and H100 GPUs, with over 1,300 petaflops of peak throughput. We used 2.5 years of operational data (11.7 million GPU hours) on GPU errors. Our major findings include: (i) H100 GPU memory resilience is worse than A100 GPU memory, with 3.2 × 3.2\times lower per-GPU MTBE for memory errors, (ii) The GPU memory error-recovery mechanisms on H100 GPUs are insufficient to handle the increased memory capacity, (iii) H100 GPUs demonstrate significantly improved GPU hardware resilience over A100 GPUs with respect to critical hardware components, (iv) GPU errors on both A100 and H100 GPUs frequently result in job failures due to the lack of robust recovery mechanisms at the application level, and (v) We project the impact of GPU node availability on larger-scales and find that significant overprovisioning of 5% is necessary to handle GPU failures. Conference: The International Conference for High Performance Computing, Networking, Storage and Analysis; November 16–21, 2025; St Louis, MO, USA The International Conference for High Performance Computing, Networking, Storage and Analysis (SC ’25), November 16–21, 2025, St Louis, MO, USA DOI: 10.1145/3712285.3759821 ISBN: 979-8-4007-1466-5/2025/11 CCS: Computer systems organization Reliability CCS: Computer systems organization Availability CCS: Computer systems organization Redundancy CCS: Computing methodologies Discrete-event simulation CCS: General and reference Measurement CCS: General and reference Reliability † † cc-license: by 1. Introduction Large-scale HPC systems are important not only for scientific workloads ( Skinner and Kramer, 2005 ) but also for data analytics ( Alam et al., 2022 ) and machine learning (ML) ( Huyen, 2022 ) . The main components of these systems are specialized accelerators, such as GPUs, that enable acceleration of computations, such as ML training ( Touvron et al., 2023 ; Anil et al., 2023 ) , ML inference ( Li et al., 2024 ) , and simulations ( Phillips et al., 2005 ; Shainer et al., 2009 ) . This paper studies the resilience of A100 GPUs and compares the result with H100 GPUs of the GH200 Grace Hopper Superchips 2 2 2 GH200 Superchip, hereafter. , together with their associated memory: 40 GB HBM2e on each A100 GPU, and 96 GB HBM3 on each H100 GPU, respectively. The A100 GPU nodes and H100 GPUs (GH200 Superchip) nodes are operated as two independent systems sharing a storage cluster running the Lustre file system, allowing us to study and compare the two systems (refer to as Delta ). The workflow on Delta ( Bode et al., 2025 ) involves users from universities nationwide and presents a spectrum of HPC and ML workloads. The study uses 2.5 years of data on critical GPU errors collected across the stated GPUs, encompassing 9.6 million GPU hours of A100 GPUs and 2.1 million GPU hours of H100, a combined 11.7 million GPU hours. This study assesses (i) the resilience of GPU hardware and memory components; (ii) the error propagation paths in GPU memory, GPU hardware, and NVLink interconnect; and (iii) the impact of the observed GPU errors on user jobs. Figure 1 shows an example error propagation path for an uncorrectable double-bit memory error in H100 GPU that caused user job failure. The complete recovery process required node draining and GPU reset, which took 19 hours following the error detection. A code snippet showing GSP Error Caused App Failure. Figure 1. A double-bit memory error (XID 48) occurred, and it is uncorrectable by the SECDED ECC HBM3 memory. Due to this double-bit error, the user job scheduled on that GPU failed, as reflected in the scheduler logs. Subsequently, this uncorrectable memory error requires a node draining and reset to complete the row remapping recovery action. The total recovery process for this incident took 19 hours, during which the node was unavailable for accepting new jobs. This incident shows that a GPU error can lead to user job failures and significantly impact node availability. A code snippet showing GSP Error Caused App Failure. Our major findings include: (i) H100 shows 3.2\times lower per-GPU mean time between errors (MTBE) compared to A100 for uncorrectable ECC memory errors. The per-GB MTBE of the H100’s HBM3 memory is 24% lower ( \sim\!8.5 M hours) than the A100’s HBM2e memory ( \sim\!11.3 M hours). We conjecture that the reduction in memory resilience stems from H100’s higher memory capacity. (ii) The GPU memory error-recovery mechanisms on A100 and H100 GPUs (e.g., memory row remapping, error containment) ( NVIDIA Corporation, 2024 ) improve GPU memory resilience and reduce service interruption. We observed that these mechanisms mitigate (e.g., using memory row remapping) 92% of uncorrectable ECC memory errors on H100 GPUs. However, the memory error-recovery mechanism is insufficient to handle the increase in memory capacity and the corresponding increase in row remapping events on the H100 GPUs. (iii) H100 GPUs demonstrate significantly improved hardware resilience over A100 GPUs, with respect to critical components such as GSP 3 3 3 A GPU system processor (GSP) is an onboard co-processor that offloads driver tasks from the CPU for latency and performance improvement. , NVLink, and PMU SPI 4 4 4 PMU on an NVIDIA’s GPU regulates the frequency, voltage, and power of the GPU based on various factors such as temperature and power cap. SPI stands for serial peripheral interface, which serves as the communication channel between peripheral hardware. , which were major sources of job failures in A100 systems. We attribute this to driver-level enhancements and tighter integration ( NVIDIA, 2023 ; NVIDIA Corporation, 2023 ) , which contribute to improved resilience. Specifically, comparing H100 and A100 GPU hardware, we observed (a) a significant reduction of GSP errors on H100 (only 3 cases in our measurement period) and (b) the elimination of PMU SPI error propagations, which on A100 GPUs can lead to MMU errors 88% of times with 90% leading to user job failures, (c) no NVLink errors on H100 GPUs during the measurement period. (iv) GPU errors on both A100 and H100 GPUs frequently result in job failures due to the lack of robust recovery mechanisms at the application level. Except for MMU and NVLink errors, other GPU errors cannot be handled by application-level mechanisms, resulting in close to 100\% job failure rate. The underlying cause of job failures differs by GPU type: hardware errors are the predominant cause in A100 GPUs, whereas memory errors are the primary cause in H100 GPUs. Overall, we find that application-based recovery strategies are largely ineffective; hence, there is a compelling need to improve resilience at the GPU memory and hardware level. (v) The overall availability per-GPU node is approximately 99.4\% for A100 GPUs and 99.3\% for H100 GPUs, corresponding to a downtime between 9–10 minutes per day. We projected the impact of this measured availability on larger scales via emulation. For

Read the complete text