Joint Training on AMD and NVIDIA GPUs
Jon Hu, Thomas Jia, Jing Zhu, Zhendong Yu · 2026
Read Joint Training on AMD and NVIDIA GPUs by Jon Hu, Thomas Jia, Jing Zhu, Zhendong Yu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-18007.txt.
Description
Jon Hu and colleagues present a technical solution for training large language models across mixed clusters of AMD and NVIDIA graphics processors. Growing compute and system demands have made single-vendor homogeneous clusters insufficient. Their compatibility-oriented approach uses CPU-forwarding communication, with different communication back ends chosen for different parallel groups and multi-network-interface parallel data transfer. A second, faster approach uses device-direct communication.
That approach adds a CPU-offloading peer-to-peer mechanism that moves data directly between graphics processors from different vendors without staging through host memory. In experiments on two large language models, device-direct communication reached up to ninety-eight percent of the throughput of a homogeneous NVIDIA system while preserving training stability and correctness. The work shows that mixed-vendor training can nearly match single-vendor speed without giving up reliability.
Catalog record
| CREATOR | Jon Hu, Thomas Jia, Jing Zhu, Zhendong Yu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 17 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-18007