H2: Towards Efficient Large-Scale LLM Training on Hyper-Heterogeneous Cluster over 1,000 Chips Conference: ; 2025;

Ding Tang, Jiecheng Zhou, Jiakai Hu, Shengwei Li, Huihuang Zheng, Zhilin Pei, Hui Wang, Xingcheng Zhang · 2025

Read H2: Towards Efficient Large-Scale LLM Training on Hyper-Heterogeneous Cluster over 1,000 Chips Conference: ; 2025; by Ding Tang, Jiecheng Zhou, Jiakai Hu, Shengwei Li, Huihuang Zheng, Zhilin Pei, Hui Wang, Xingcheng Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-17548.txt.

Introduction

Ding Tang and co-authors present a framework for training large language models on clusters that mix over a thousand chips from different makers, in a paper from 2025. Such mixed clusters promise more computing power, but chips differ in software, math routines, and links, so standard training tools waste them.

The framework, called H2, has three parts. A shared interface keeps the same program behavior on every chip. A direct device-to-device link library speeds talk between unlike chips. A smart pipeline plan splits the work to balance compute, memory, and transfer load. On a model with one hundred billion parameters, the system beat matched single-vendor training by a clear margin. The authors claim very large mixed clusters are practical.

Catalog record

CREATORDing Tang, Jiecheng Zhou, Jiakai Hu, Shengwei Li, Huihuang Zheng, Zhilin Pei, Hui Wang, Xingcheng Zhang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE54 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-17548

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

H2: Towards Efficient Large-Scale LLM Training on Hyper-Heterogeneous Cluster over 1,000 Chips Conference: ; 2025; Ding Tang, Jiecheng Zhou, Jiakai Hu, Shengwei Li, Huihuang Zheng, Zhilin Pei, Hui Wang, Xingcheng Zhang Abstract . Recent advancements in large language models (LLMs) necessitate extensive computational resources, prompting the use of diverse hardware accelerators from multiple vendors. However, traditional distributed training frameworks struggle to efficiently utilize hyper-heterogeneous clusters comprising thousands of chips due to significant disparities in software stacks, operator implementations, communication libraries, and hardware capabilities. To address these challenges, we propose H2, which stands for HyperHetero and is a systematic framework enabling efficient training of LLMs on clusters with over 1,000 heterogeneous chips. H2 incorporates DiTorch 1 1 1 https://github.com/DeepLink-org/ditorch , a unified PyTorch-compatible interface ensuring program consistency across chips, and DiComm, a device-direct RDMA communication library optimized for heterogeneous environments. Furthermore, we introduce HeteroPP with HeteroAuto, an adaptive pipeline parallelism strategy that dynamically balances computational load, memory limitations, and communication overhead. Evaluations on a 100-billion-parameter LLM demonstrate that our approach consistently achieves a superlinear speedup, outperforming baseline homogeneous training solutions by up to 16.37% in our experiments. These findings validate the feasibility and efficiency of hyper-heterogeneous training at unprecedented scales. 1. Introduction Figure 1. Comparison of chip specifications between capability-incremental and our hyper-heterogeneous scenario.In traditional heterogeneous scenarios, as indicated by the black dashed circles in the figure, chips show a trend of increasing capabilities in computation, communication, and memory. In contrast, in hyper-heterogeneous scenarios, as indicated by the red dashed circles in the figure, the capabilities of chips in these three aspects do not follow any specific pattern. Recent advances in artificial intelligence have led to the fast development of large-scale models such as GPT-4 ( Achiam et al. 2023 ) , LLaMA ( Touvron et al. 2023 ) , and DeepSeek ( Guo et al. 2025 ; Liu et al. 2024 ) , which require substantial computational resources. The training of these models typically involves prolonged operation on expansive computing clusters to ensure convergence and optimal performance. Historically, such clusters have been built around single-chip servers ( An et al. 2024 ) . However, as hardware requirements continue to escalate, numerous semiconductor companies have introduced their own distinctive chip designs ( Otterness and Anderson 2020 ) , and each chip is developed according to proprietary design principles, resulting in both software isolation and different hardware specifications . The real-world application of heterogeneous chips is stead-ily becoming a reality. First, relying solely on a single type of chip may be insufficient due to limitations in production capacity or the specific suitability for certain computational tasks ( Milojicic et al. 2021 ) . Consequently, many organizations are now deploying heterogeneous clusters composed of multiple types of chips ( Weng et al. 2022 ) . Furthermore, with the ongoing procurement of new chips over time ( Elster and Haugdahl 2022 ; Tirumala and Wong 2024 ; Choquette et al. 2021 ) , it is urgently important to integrate legacy chips with newer ones efficiently to reduce idle costs. Hyper-heterogeneous computing refers to an emerging computing paradigm characterized by significant hardware and software diversity within large-scale computing clusters. Specifically, hyper-heterogeneous environments exhibit three key distinguishing features. First, the hardware specifications—including computational power, memory capacity, and communication bandwidth—of different chips can vary drastically without clear patterns or predictable relationships. Second, chips from different vendors typically exhibit pronounced software isolation, utilizing separate software stacks and distinct communication libraries, thereby complicating interoperability and software integration. Third, the quantities of different chip types within a single cluster may be highly imbalanced, further increasing complexity in workload scheduling and resource management. Unlike conventional heterogeneous computing systems, which usually involve fewer chip variants with more uniform programming models and communication protocols, hyper-heterogeneous systems are characterized by a significantly higher degree of complexity and integration challenges. Existing heterogeneous approaches have predominantly concentrated on designing training algorithms for hardware setups that often involve different generations of chips from the same vendor ( Jia et al. 2022 ; Um et al. 2024 ) . For example, Metis searches for heterogeneous parallelism strategies on Nvidia T4, P100, and V100, and Whale accelerates heterogeneous parallel training with Nvidia P100 and V100. This design choice leads to two notable implications. First, there is an inherent performance gap between pairs of chips; one type typically outperforms the other across a range of specifications such as computational power, memory capacity, and communication bandwidth as shown in Figure 1 . This discrepancy can significantly impact workload distribution and overall system efficiency. Second, these setups benefit from a naturally unified software stack ( NVIDIA 2025a ) and communication library ( NVIDIA 2025b ) (e.g., CUDA and NCCL), which minimizes the need for adjustments that would otherwise be required to bridge isolated software environments within heterogeneous systems. In addition, most heterogeneous training approaches have been designed for small-scale scenarios or validated only on clusters with a limited types of chips ( Xu et al. 2024 ) . Consequently, their scalability and performance in larger, more diverse computing environments remain largely unproven. In this work, we address the scenario of efficiently training extremely large models in hyper-heterogeneous computing environments. To uniformly leverage chip resources from different vendors while ensuring scalability, we highlight the necessity of developing new systems and algorithms specifically designed for hyper-heterogeneous scenarios. Such systems must effectively address the following issues: First, the technical isolation among different types of chips are considered from three aspects: software, communication and operators. The software isolation comes from the fact that chips from different vendors are supported by distinct technology stacks and, in many cases, their own training frameworks. Although some chips offer compatibility via adaptations to popular interfaces ( Wu 2023 ; Abadi et al. 2016 ) (e.g., PyTorch), semantic inconsistencies between these solutions pose significant challenges for developers aiming for a unified development environment. Furthermore, the issue of communication isolation adds another layer of complexity. Different chips utilize diverse internal communication libraries, and the variations in server topologies—such as the number of network interface cards and respective bandwidths—make it difficult to develop a standardized communication library that can be uniformly applied ( Li et al. 2019a ) . Finally, operator isolation arises from the differing hardware design logics across chips. As each chip implements its own acceleration strategies and operator development methodologies, there often remain discrepancies in the support and precision of common operators. Although most chip manufacturers have implemented adaptations for widely-used large-scale models, gaps persist in both operator availability and computational accuracy alignment. Second, heterogeneous training in large-scale scenarios brings several challenges. Driven by the rapid advancement of large-model techniques ( Radford et al. 2018 ; Devlin et al. 2019 ) , the computational resources necessary for training a state-of-the-art base model have grown extraordinarily demanding ( Narayanan et al. 2021 ) . This scale of computational requirement naturally results in an extremely large space for parallel execution strategies, making the optimization process considerably complex. Therefore, it is imperative not only to develop efficient parallel algorithms, but also to ensure that the algorithmic search process itself remains computationally efficient to reduce the high cost of large-scale computing resources ( Zheng et al. 2022 ; Li et al. 2022 ) .

Read the complete text