SprayCheck: Finding Gray Failures in Adaptive Routing Networks

Jakob Krebs, Daniel Amir, Shir Landau Feibish, Mark Silberstein · 2026

Read SprayCheck: Finding Gray Failures in Adaptive Routing Networks by Jakob Krebs, Daniel Amir, Shir Landau Feibish, Mark Silberstein (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-03702.txt.

Description

Jakob Krebs, Daniel Amir, Shir Landau Feibish, and Mark Silberstein present SpraySense, a system for detecting gray failures in the huge networks used for distributed machine learning training. Gray failures degrade performance without fully breaking links, and they are notoriously hard to find in clusters of tens of thousands of graphics processors. Adaptive routing, which spreads traffic across many paths, makes them worse.

SpraySense works passively, exploiting the statistical behavior of adaptive routing and load balancing together with flow-level information. Discrepancies in how packets spray across paths reveal trouble early. Detecting failures before they seriously hurt applications enables preemptive rerouting and protects expensive training runs.

Catalog record

CREATORJakob Krebs, Daniel Amir, Shir Landau Feibish, Mark Silberstein
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE69 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.03702

Browse the full library · About the Tower of Babel