SprayCheck: Finding Gray Failures in Adaptive Routing Networks
Jakob Krebs, Daniel Amir, Shir Landau Feibish, Mark Silberstein · 2026
Read SprayCheck: Finding Gray Failures in Adaptive Routing Networks by Jakob Krebs, Daniel Amir, Shir Landau Feibish, Mark Silberstein (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-03702.txt.
Description
Jakob Krebs, Daniel Amir, Shir Landau Feibish, and Mark Silberstein present SpraySense, a system for detecting gray failures in the huge networks used for distributed machine learning training. Gray failures degrade performance without fully breaking links, and they are notoriously hard to find in clusters of tens of thousands of graphics processors. Adaptive routing, which spreads traffic across many paths, makes them worse.
SpraySense works passively, exploiting the statistical behavior of adaptive routing and load balancing together with flow-level information. Discrepancies in how packets spray across paths reveal trouble early. Detecting failures before they seriously hurt applications enables preemptive rerouting and protects expensive training runs.
Catalog record
| CREATOR | Jakob Krebs, Daniel Amir, Shir Landau Feibish, Mark Silberstein |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 69 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.03702