SGEMM-cube: Emulating FP32 GEMM on Ascend NPUs Using FP16 Cube Units with Precision Recovery
Weicheng Xue, Baisong Xu, Kai Yang, Yongxiang Liu, Dengdeng Fan, Pengxiang Xu, Yonghong Tian · 2025
Read SGEMM-cube: Emulating FP32 GEMM on Ascend NPUs Using FP16 Cube Units with Precision Recovery by Weicheng Xue, Baisong Xu, Kai Yang, Yongxiang Liu, Dengdeng Fan, Pengxiang Xu, Yonghong Tian (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-23387.txt.
Introduction
Weicheng Xue and co-authors fake high-precision matrix math on fast low-precision hardware in a 2025 paper. Ascend AI chips multiply matrices quickly in half precision but lack a good full-precision path, which many workloads still need. Their method, SGEMM-cube, splits each full-precision number into a half-precision head and a scaled leftover.
Three half-precision multiplications rebuild the product, skipping only the tiny leftover-times-leftover term. Error analysis shows the skip costs no more than normal rounding in short sums. The team also handle leftover underflow and tune memory blocking for the chip's hand-managed memory. On an Ascend 910A the method ran far more accurately than native half precision and hit about seventy-seven percent of the three-pass peak speed.
Catalog record
| CREATOR | Weicheng Xue, Baisong Xu, Kai Yang, Yongxiang Liu, Dengdeng Fan, Pengxiang Xu, Yonghong Tian |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 38 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-23387
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SGEMM-cube: Emulating FP32 GEMM on Ascend NPUs Using FP16 Cube Units with Precision Recovery Weicheng Xue, Baisong Xu, Kai Yang, Yongxiang Liu, Dengdeng Fan, Pengxiang Xu, Yonghong Tian Abstract Low-precision matrix engines, such as FP16 cube, offer high throughput but lack support for full-precision computation. In this work, we propose SGEMM-cube, a high-performance algorithm for emulating FP32 general matrix-matrix multiplication (GEMM) using only FP16 computation units on a representative AI accelerator. The method decomposes each FP32 operand into two FP16 values and compensates for numerical errors through a tunable scaling strategy. A detailed analysis of numerical errors, including underflow conditions and precision loss, guides the selection of scaling parameters to preserve up to 22 bits of mantissa accuracy. We further investigate the effect of computation order on accuracy and demonstrate that a term-wise accumulation scheme improves numerical stability over conventional FP32 GEMM in low-exponent regimes. Finally, a cache-aware blocking strategy and double-buffered pipeline are introduced to overlap memory transfers with computation, enabling SGEMM-cube to achieve up to 77% of the theoretical FP32-equivalent peak performance on Ascend 910A NPU lacking native FP32 support. Extensive numerical experiments confirm that our method not only recovers the accuracy of native FP32 GEMM but also exhibits superior numerical stability under certain conditions, due to its structured and error-aware computation order. † † highlights: Emulate FP32 GEMM using FP16-only matrix engines with high precision Tunable scaling and decomposition strategy to preserve up to 22-bit mantissa Term-wise accumulation improves numerical stability in low-exponent regimes Cache-aware blocking and pipelining achieves 77% of peak performance 1 Introduction The evolution of specialized accelerators is reshaping the landscape of high-performance computing (HPC). Driven by the rapid progress in deep learning, modern computer systems increasingly rely on heterogeneous architectures that integrate domain-specific hardware. Processors such as NVIDIA GPUs with Tensor Cores, Google’s TPUs, and emerging Neural Processing Units (NPUs) deliver significant speedups for matrix-dominated workloads, offering orders of magnitude higher throughput than traditional CPUs. These accelerators achieve high performance by concentrating their compute resources on low-precision formats such as half-precision floating point (FP16), bfloat16 (BF16), and INT8. While these formats are sufficient for many machine learning tasks, they are often inadequate for deep learning, scientific and engineering applications [ 1 , 2 , 3 , 4 , 5 , 6 ] , which demand the accuracy and dynamic range of FP32 or FP64 arithmetic. This mismatch presents a growing challenge: the most powerful computing platforms are optimized for formats that are incompatible with the precision requirements of many established scientific workloads. To bridge this gap, this work investigates how to efficiently harness the compute density of low-precision hardware for workloads that require FP32-level accuracy. Specifically, we target scenarios where native FP32 matrix multiplication (GEMM) is unsupported or inefficient, such as on Ascend 910A NPUs that expose high-performance FP16 tensor engines but lack equivalent FP32 compute units. We propose SGEMM-cube , a software–hardware co-designed framework that enables high-throughput and high-accuracy FP32 GEMM using only FP16 computation. At its core, SGEMM-cube combines a numerical emulation technique with a performance-optimized implementation tailored to the memory hierarchy of a representative AI accelerator. The main contributions of this work are: 1. A general algorithm for FP32 emulation: We present SGEMM-cube, an algorithm that decomposes each FP32 operand into a pair of FP16 values, a high-order component and a residual, and recombines their products to reconstruct FP32-level results, preserving up to 22 bits of mantissa accuracy. 2. Error analysis and numerical control: We perform a detailed study of precision loss due to rounding and underflow, and introduce two mitigation strategies: (1) a tunable amplification factor applied to residual terms, and (2) a term-wise accumulation scheme that improves numerical stability over traditional reduction orders. 3. Architecture-aware optimization: We develop a set of performance strategies including L1-cache-aware tiling and a software-level double-buffered pipeline. These techniques overlap memory movement and computation, effectively hiding the cost of decomposition and enabling high hardware utilization. 4. Empirical validation: We evaluate SGEMM-cube on Ascend 910A, a representative AI accelerator with FP16-only GEMM engines. Results show that SGEMM-cube achieves up to 77% of the FP32-equivalent theoretical peak and, in some cases, exceeds the numerical accuracy of native sgemm implementations under low-exponent inputs. 2 Background and Related Work The growing gap between the numerical precision requirements of scientific computing and the capabilities of modern AI-oriented accelerators has prompted renewed interest in mixed-precision algorithms. While many scientific applications still demand FP32 or FP64 precision for numerical stability, contemporary hardware increasingly favors lower-precision formats such as FP16 or even FP8 to maximize throughput and energy efficiency. For example, the FP8 Tensor Core throughput of the NVIDIA H100 is roughly 60 \times higher than its FP64 throughput, and the Ascend 910A NPU [ 7 ] delivers \sim 256 TF/s in FP16 but lacks efficient support for FP32 GEMM. These trends highlight both a challenge and an opportunity: how to emulate high-precision matrix computations using low-precision arithmetic units without compromising accuracy . A natural approach to address the precision–performance tradeoff is through mixed-precision techniques. The idea of performing most computations in lower precision and then refining results in higher precision has a long history in numerical linear algebra, particularly in iterative refinement solvers. These methods rely on selectively using high-precision arithmetic in numerically sensitive operations, while exploiting faster low-precision operations elsewhere to gain performance. In contrast to such hybrid strategies, recent work has explored a more direct direction: emulating high-precision GEMM (e.g., FP32) using only low-precision hardware such as FP16. Rather than mixing precisions, these methods decompose each high-precision operand into two or more low-precision components, typically capturing the high and low bits of the significand, and reformulate the original matrix product as a sum of multiple low-precision GEMMs. This line of work aims to preserve high numerical accuracy while operating entirely on accelerators that lack efficient FP32 units. The foundational work by Ozaki et al. [ 8 ] formalized this idea by expressing the product of two FP64 matrices as a series of lower-precision GEMMs with a priori error bounds. Although originally targeted at double precision, this decomposition strategy laid the groundwork for FP32-from-FP16 emulation schemes. Markidis et al. [ 9 ] were among the first to explore such techniques on GPUs, proposing a two-pass approach based on splitting FP32 inputs into FP16 components. Their method performed an initial FP16 GEMM followed by a corrective residual GEMM. However, their use of round-towards-zero (RZ) conversion introduced a systematic 2-bit precision loss compared to full FP32 accuracy, owing to uncorrected rounding errors in the low-order bits. To reduce this error, Feng et al. [ 10 ] proposed Egemm-TC, a refinement that adopted a more careful decomposition and a rounding strategy closer to round-to-nearest (RN). Their implementation included warp-level cache reuse and register pipeline optimizations, yielding a 3.13 \times speedup over cuBLAS FP32 GEMM on NVIDIA Turing GPUs. However, the decomposition still failed to fully account for the implicit leading bit in FP32 representation, limiting the worst-case precision to \sim 21 bits. Ootomo and Yokota [ 11 ] later identified another key source of error: the use of Tensor Cores for accumulation. Tensor Core units perform multiplications in FP16 but may accumulate results using internal FP32 registers with RZ rounding [ 12 ]