CCL-D: A High-Precision Diagnostic System for Slow and Hang Anomalies in Large-Scale Model Training DOI: 10.1145/3774934.3786429 ISBN: 979-8-4007-2310-0/2026/01 Conference: Proceedings of the 31st ACM SIGPLAN Annual Symposium on Principles and Practice of Parallel Programming; January 31 – February 4, 2026; Sydney, NSW, Australia Proceedings of the 31st ACM SIGPLAN Annual Symposium on Principles and Practice of Parallel Programming (PPoPP ’26), January 31 – February 4, 2026, Sydney, NSW, Australia CCS: Hardware Error detection and error correction CCS: Computing methodologies Neural networks

Yida Gu, Fakang Wang, Jianhao Fu, Zhenhang Sun, Qianyu Zhang, Hairui Zhao, Xingchen Liu, Yang Tian, Wenjing Huang, Zedong Liu, Yifan Chen, Jinwu Yang, Yueyuan Zhou, Qian Zhao, Haoxu Li, Tao Wang, Feng Yu, Zhan Wang, Guangming Tan, Dingwen Tao · 2026

Read CCL-D: A High-Precision Diagnostic System for Slow and Hang Anomalies in Large-Scale Model Training DOI: 10.1145/3774934.3786429 ISBN: 979-8-4007-2310-0/2026/01 Conference: Proceedings of the 31st ACM SIGPLAN Annual Symposium on Principles and Practice of Parallel Programming; January 31 – February 4, 2026; Sydney, NSW, Australia Proceedings of the 31st ACM SIGPLAN Annual Symposium on Principles and Practice of Parallel Programming (PPoPP ’26), January 31 – February 4, 2026, Sydney, NSW, Australia CCS: Hardware Error detection and error correction CCS: Computing methodologies Neural networks by Yida Gu, Fakang Wang, Jianhao Fu, Zhenhang Sun, Qianyu Zhang, Hairui Zhao, Xingchen Liu, Yang Tian, Wenjing Huang, Zedong Liu, Yifan Chen, Jinwu Yang, Yueyuan Zhou, Qian Zhao, Haoxu Li, Tao Wang, Feng Yu, Zhan Wang, Guangming Tan, Dingwen Tao (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-04478.txt.

Description

Yida Gu and nineteen co-authors present a diagnostic system for the slow or hanging communication faults that plague large-scale model training. As clusters grow, the libraries that coordinate thousands of processors hit faults from tangled hardware, software, and environmental causes, and manual diagnosis can take days.

Their system pairs a lightweight rank-level probe with an automated analyser. The probe traces communication traffic across layers in real time. The analyser detects anomalies and locates the faulty processor rank. Deployed for a year on a four thousand processor cluster, it covered nearly all known slow or hang cases. It pinpointed affected ranks within minutes, with high precision.

Catalog record

CREATORYida Gu, Fakang Wang, Jianhao Fu, Zhenhang Sun, Qianyu Zhang, Hairui Zhao, Xingchen Liu, Yang Tian, Wenjing Huang, Zedong Liu, Yifan Chen, Jinwu Yang, Yueyuan Zhou, Qian Zhao, Haoxu Li, Tao Wang, Feng Yu, Zhan Wang, Guangming Tan, Dingwen Tao
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE51 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.04478

Browse the full library · About the Tower of Babel