Joint Training on AMD and NVIDIA GPUs

Jon Hu, Thomas Jia, Jing Zhu, Zhendong Yu · 2026

Read Joint Training on AMD and NVIDIA GPUs by Jon Hu, Thomas Jia, Jing Zhu, Zhendong Yu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-18007.txt.

Description

Jon Hu and colleagues present a technical solution for training large language models across mixed clusters of AMD and NVIDIA graphics processors. Growing compute and system demands have made single-vendor homogeneous clusters insufficient. Their compatibility-oriented approach uses CPU-forwarding communication, with different communication back ends chosen for different parallel groups and multi-network-interface parallel data transfer. A second, faster approach uses device-direct communication.

That approach adds a CPU-offloading peer-to-peer mechanism that moves data directly between graphics processors from different vendors without staging through host memory. In experiments on two large language models, device-direct communication reached up to ninety-eight percent of the throughput of a homogeneous NVIDIA system while preserving training stability and correctness. The work shows that mixed-vendor training can nearly match single-vendor speed without giving up reliability.

Catalog record

CREATORJon Hu, Thomas Jia, Jing Zhu, Zhendong Yu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE17 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-18007

Browse the full library · About the Tower of Babel