Mind the Gap: Revealing Inconsistencies Across Heterogeneous AI Accelerators
Elliott Wen 1, Sean Ma 1, Evan Tempero 1, Bruce Sham 1, Yousong Song 1, Hong Jia 1, Daniel Luo 2, Jiayi Hua 2, Jens Dietrich 3, Kaiqi Zhao 4 Jiaxing Shen 5 · 2025
Read Mind the Gap: Revealing Inconsistencies Across Heterogeneous AI Accelerators by Elliott Wen 1, Sean Ma 1, Evan Tempero 1, Bruce Sham 1, Yousong Song 1, Hong Jia 1, Daniel Luo 2, Jiayi Hua 2, Jens Dietrich 3, Kaiqi Zhao 4 Jiaxing Shen 5 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-11601.txt.
Introduction
Elliott Wen and ten co-authors run the first large empirical check of whether the same machine learning model behaves identically on AI chips from different makers. NVIDIA dominates cloud data centers, but AMD, Intel, Apple, and Huawei hardware is marketed as a compatible, cheaper alternative.
An automated pipeline built over one hundred thousand model variants from four thousand real models and ran them on five enterprise accelerators. Newer non-NVIDIA platforms supported noticeably fewer operators, and output mismatches above five percent appeared, traced to operator implementations, handling of extreme numerical values, and instruction differences. Teams porting models between vendors now have measured evidence of where silent divergence creeps in.
Catalog record
| CREATOR | Elliott Wen 1, Sean Ma 1, Evan Tempero 1, Bruce Sham 1, Yousong Song 1, Hong Jia 1, Daniel Luo 2, Jiayi Hua 2, Jens Dietrich 3, Kaiqi Zhao 4 Jiaxing Shen 5 |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 38 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-11601
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Mind the Gap: Revealing Inconsistencies Across Heterogeneous AI Accelerators Elliott Wen 1, Sean Ma 1, Evan Tempero 1, Bruce Sham 1, Yousong Song 1, Hong Jia 1, Daniel Luo 2, Jiayi Hua 2, Jens Dietrich 3, Kaiqi Zhao 4 Jiaxing Shen 5 Abstract While NVIDIA remains the dominant provider of AI accelerators within cloud data center, emerging vendors such as AMD, Intel, Mac, and Huawei offer cost-effective alternatives with claims of compatibility and performance. This paper presents the first empirical study investigating divergence in machine learning model across heterogeneous AI accelerators. Utilizing an automated pipeline, we synthesize over 100,000 variant models derived from 4,000 real-world models and execute them across five different enterprise-grade accelerators. Our findings suggest that newer AI platforms from Mac and Huawei support at least 17% fewer operators than NVIDIA. These platforms also exhibit a higher rate of output discrepancies (exceeding 5%), which stem from differences in operator implementations, handling of exceptional numerical values, and instruction scheduling. They are also more susceptible to failures during model compilation-based acceleration, and in some cases, the compiled models produce outputs that differ noticeably from those generated using the standard execution mode. In addition, we identify 7 implementation flaws in PyTorch and 40 platform-specific issues across vendors. These results underscore the challenges of achieving consistent machine learning behavior in an increasingly diverse hardware ecosystem. I Introduction Machine learning (ML) is a rapidly expanding field with growing influence across various disciplines. To meet the computational demands of ML models training and inference, many cloud service providers now offer access to hardware such as graphics processing units and specialized AI accelerators. Although NVIDIA-based solutions are most widely adopted, other platforms such as AMD, Intel, Mac and Huawei are gaining traction. They claim to offer sufficient computational performance for general machine learning tasks, while at a more competitive price point. In addition, they emphasize support for existing ML frameworks, particularly PyTorch, one of the most widely used frameworks in academia and industry [ 8 ] . As hardware platforms become increasingly diverse, concerns about their consistency and reliability are growing, particularly for developers seeking to avoid vendor lock-in and adopt more cost-effective cloud AI hardware services. In this paper, we present, to the best of our knowledge, the first empirical study assessing divergence in ML model execution across heterogeneous AI accelerator. We aim to answer the following research question: How consistent are the outputs of ML models across different hardware platforms, and what are the potential sources of inconsistency? To address this, we employ a differential testing approach; we execute 100,000 synthesized models with the same randomly generated inputs across various hardware platforms. We then investigate significant discrepancies to uncover potential issues in the underlying hardware or software stack. To support this investigation, we implement an automated pipeline, as illustrated in Figure 1 . The pipeline begins with corpus curation, where we crawl approximately 4,000 real-world models from various machine learning domains. These curated models are then converted into computational graphs, which serve as input to a model synthesizer. The synthesizer generates a large number of variant models by iteratively merging random subgraphs from the corpus and applying mutations to individual nodes, inspired by prior work [ 5 ] . The variant models are executed across different hardware platforms, and any output discrepancies are recorded. We then analyze these discrepancies to pinpoint potential sources of divergence. Fig. 1: Our pipeline to uncover the behavior inconsistency across different hardware platforms We conduct our experiments on five enterprise-grade AI accelerator hardware commonly accessible through cloud, including the NVIDIA H200 , AMD MI300X , Intel Max 1100 , Huawei Ascend 910B , and Mac M4 Pro . We unveil the following key research findings: 1. Relatively new platforms, such as Mac and Huawei, exhibit lower model compatibility due to unsupported data types or unimplemented operations. Our analysis reveals that Mac and Huawei support 34% and 17% fewer operators than NVIDIA, respectively. As such, code modifications are often necessary. Most missing operators on Huawei and Mac involve quantized/sparse operations, attention, NLP embeddings, and network training. 2. Using NVIDIA as the baseline, Intel and AMD generally produce more consistent results, with output agreement rates of 99.6% and 99.8%, respectively. In contrast, the Ascend and Mac platforms show significantly lower consistency, with agreement rates dropping to 95% and 86%. These discrepancies primarily stem from flawed operator implementations, differences in handling exceptional values, variations in instruction scheduling, and instances of undefined behavior. 3. Some platforms exhibit reduced reliability when PyTorch’s compilation-based acceleration features are enabled. On Mac and Huawei devices, approximately 2% of models fail to compile, and about 1.1% yield results that differ from those produced by the default execution mode. II Background and Related Work Fig. 2: PyTorch Model Representation: From High-Level Python Code to Computation Graph We open-source our testing pipeline together with our model corpus. Using this pipeline, we uncover 7 implementation flaws in the PyTorch framework, such as memory access violations and program hangs caused by infinite optimization loops in the just-in-time compiler. We also identified behavioral inconsistencies or implementation flaws across different platforms: 1 on NVIDIA, 4 on AMD, 3 on Intel, 11 on Mac, and 13 on Huawei. ML Model and Computation Graph: To aid reader comprehension of the rest of this paper, we first provide a brief overview of ML frameworks. We focus on PyTorch due to its broad adoption in both academic research and industry applications. Moreover, Pytorch is the only framework that consistently receives active development and optimization support from all five hardware vendors discussed in this paper. In Pytorch, models are defined in Python by subclassing the nn.Module class. The model’s layers are specified within the init method, while the forward computation logic is implemented in the forward method. Once defined, input data can be passed through the model to generate predictions. Figure 2 (a) illustrates a simple model that takes two input tensors, x1 and x2 , adds them together, and passes the result through a linear layer followed by a ReLU activation function. Under the hood, these high-level Python tensor operations are dispatched to PyTorch’s low-level C++ backend, known as ATen . Each operation is mapped to a corresponding ATen operator. In our example, the tensor addition, linear transformation, and ReLU activation are translated into aten::add , aten::addmm , and aten::relu , respectively. These ATen operators form a directed acyclic computation graph as shown in Figure 2 (b), where each operation corresponds to a node. An edge is created between two nodes if the output of one operation is used as the input to another. The input and output tensors of the graph can be treated as special types of nodes, representing the entry and exit points of the computation. Internally, ATen operators are further redispatched to hardware-specific implementations, such as CUDA kernels relu_cuda and addmm_cuda to leverage hardware acceleration. Framework Bug Detection: ML frameworks can exhibit various types of bugs, such as incorrect operator implementations or memory safety violations [ 12 ] . Existing approaches for bug detection typically involve feeding data into curated or synthesized models to identify framework-level errors or inconsistencies. For instance, CRADLE [ 13 ] targets cross-framework inconsistencies by comparing outputs from a