When GPUs Fail Quietly: Observability-Aware Early Warning Beyond Numeric Telemetry
Michael Bidollahkhani , Freja Nordsiek , Julian M. Kunkel · 2026
Read When GPUs Fail Quietly: Observability-Aware Early Warning Beyond Numeric Telemetry by Michael Bidollahkhani , Freja Nordsiek , Julian M. Kunkel (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-28781.txt.
Description
Bidollahkhani, Nordsiek, and Kunkel propose an early-warning framework for quiet GPU failures in computing clusters. Some grow as slow thermal or efficiency drift. Others strike at the driver or interconnect level with no numeric warning. In those cases, the main signal is missing device metrics. Their method models thermal drift and monitoring pipeline decay together. Tests on production telemetry from GWDG nodes show detachment failures cause structural collapse first. Joint modeling gave longer warning lead time than GPU-only detection.
Catalog record
| CREATOR | Michael Bidollahkhani , Freja Nordsiek , Julian M. Kunkel |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 53043 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.28781