When GPUs Fail Quietly: Observability-Aware Early Warning Beyond Numeric Telemetry

Michael Bidollahkhani , Freja Nordsiek , Julian M. Kunkel · 2026

Read When GPUs Fail Quietly: Observability-Aware Early Warning Beyond Numeric Telemetry by Michael Bidollahkhani , Freja Nordsiek , Julian M. Kunkel (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-28781.txt.

Description

Bidollahkhani, Nordsiek, and Kunkel propose an early-warning framework for quiet GPU failures in computing clusters. Some grow as slow thermal or efficiency drift. Others strike at the driver or interconnect level with no numeric warning. In those cases, the main signal is missing device metrics. Their method models thermal drift and monitoring pipeline decay together. Tests on production telemetry from GWDG nodes show detachment failures cause structural collapse first. Joint modeling gave longer warning lead time than GPU-only detection.

Catalog record

CREATORMichael Bidollahkhani , Freja Nordsiek , Julian M. Kunkel
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE53043
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.28781

Browse the full library · About the Tower of Babel