W4A16 Mixed-Precision Matrix Multiplication on Decoupled Architecture: Kernel Design and Memory Bottleneck Analysis for Ascend NPUs

Yuanhong He, Peiyu Niu, Jun Chen, Chenchen Zhang, Chao Yang · 2026

Read W4A16 Mixed-Precision Matrix Multiplication on Decoupled Architecture: Kernel Design and Memory Bottleneck Analysis for Ascend NPUs by Yuanhong He, Peiyu Niu, Jun Chen, Chenchen Zhang, Chao Yang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-16536.txt.

Introduction

Yuanhong He and four coauthors tune mixed grade matrix times on Ascend chips. Their 2026 preprint deals with W4A16, a mix of weight and act grades. Weights use four bits while acts use twice times four bits. The mix cuts the store size of large models much. It keeps loss of truth low if the sums run well.

Ascend 910 chips pose hard bounds for such mixed sums. Store moves, not math, oft set the true speed cap. The team maps those move caps and shapes sums to fit. Fit sums would let big models run on less gear. The work serves low cost model use at scale.

Catalog record

CREATORYuanhong He, Peiyu Niu, Jun Chen, Chenchen Zhang, Chao Yang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE17 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-16536

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

W4A16 Mixed-Precision Matrix Multiplication on Decoupled Architecture: Kernel Design and Memory Bottleneck Analysis for Ascend NPUs Yuanhong He, Peiyu Niu, Jun Chen, Chenchen Zhang, Chao Yang Abstract As Large Language Models (LLMs) scale, weight-only quantization (W4A16: 4-bit weights, 16-bit activations) becomes critical for reducing memory footprint with minimal accuracy loss. However, its efficient deployment on Huawei’s Ascend 910 Neural Processing Unit (NPU) is challenging due to limited native mixed-precision support and the accelerator’s decoupled compute architecture. To enable quantization on such architecture, we present the first practical W4A16 matrix multiplication kernel tailored for the Ascend 910 NPU. Our design leverages vector cores for on-the-fly INT4-to-FP16 dequantization, cube cores for high-throughput GEMM, and Split-K parallelization to mitigate memory latency. Performance evaluations across diverse matrix shapes and batch sizes show our method outperforms data-parallel approaches when K ≫ \gg N, a typical scenario in LLM decoding. Specially, our method can achieve a speedup ranging from 1.01 × 1.01\times to 1.74 × 1.74\times . In addition, our profile reveals the primary bottleneck is not dequantization compution itself, but extra global memory transfer for the weight, making W4A16 only reaching a maximum speedup of 1.48 × 1.48\times over native FP16 × \times FP16 matrix multiplication in PyTorch. In the long run, our method lays a solid foundation and provides insightful views for the efficient deployment of quantized large language models on various domain-specific accelerators. K eywords Quantization \cdot Large Language Models \cdot Parallel Programming \cdot NPU 1 Introduction Quantization [ 7 ] has emerged as a pivotal technique for deploying large language models (LLMs) [ 13 , 15 , 9 ] on resource-constrained hardware. Recent studies, such as SINQ [ 10 ] and QQQ [ 16 ] , have achieved significant reductions in memory footprint, bandwidth requirements, and computational cost without incurring substantial losses in model accuracy. Among various quantization schemes, weight-only strategies—wherein weights are quantized to fixed-point integers while activations remain in floating-point format—have garnered growing attention due to their capability to balance efficiency and fidelity. In particular, the W4A16 configuration (4-bit weights and 16-bit activations) [ 2 , 8 ] has demonstrated remarkable potential: it retains high activation precision to preserve the dynamic range, while aggressively compressing weights to minimize storage and memory access overhead. However, unlocking the full potential of W4A16 quantization necessitates specialized kernel implementations that can efficiently unpack and dequantize 4-bit weights into FP16/BF16 formats prior to matrix multiplication. General Matrix Multiplication (GEMM) routines are ill-suited for such preprocessing-intensive, irregular data layouts and often fail to fuse dequantization with computation. This deficiency results in suboptimal memory bandwidth utilization and redundant data movement—challenges that are particularly pronounced on domain-specific architectures such as Huawei’s Ascend NPU [ 17 ] . While the cube units of the Ascend NPU are highly optimized for native GEMM operations, they lack tailored support for the additional memory-bound dequantization steps required by 4-bit weight formats. To address this gap, we present the first practical implementation of a W4A16 matrix multiplication kernel customized for Huawei’s Ascend NPU. Notably, while NVIDIA platforms benefit from highly optimized open-source libraries (e.g., Bitblas [ 14 ] and Marlin [ 3 ] ) that support low-bit quantized inference, the Ascend software stack currently provides no native kernel support for mixed-precision matrix multiplication. Our implementation integrates on-the-fly weight dequantization with compute scheduling, enabling the direct execution of W4A16 mixed-precision matrix multiplication kernels on Ascend NPUs. 2 Background 2.1 Quantization in Deep Neural Networks Quantization is a technique that maps high-precision floating-point parameters to lower-bit integer representations, primarily to reduce model size and computational cost. Among various approaches, weight-only quantization has emerged as a highly effective strategy for compressing and accelerating Large Language Models. In this paradigm, only the model weights are quantized—often to 4 bits or lower—while activations remain in full precision (e.g., FP16). This design preserves model accuracy more effectively than full quantization by avoiding error propagation through low-precision intermediate computations. Consequently, weight-only methods are particularly suitable for deployment on memory-constrained devices, as they dramatically reduce the model’s memory footprint. A widely used formulation for weight quantization is uniform affine quantization [ 7 ] , defined as: x_{q}=\text{round}\left(\frac{x}{s}\right)+z (1) where x is the original value, x_{q} the quantized integer, s the scale factor, and z the zero-point offset. For symmetric quantization (common in weights), z=0 . 2.2 Motivation and Challenges W4A16 [ 2 , 8 ] quantization (4-bit weights, 16-bit activations) offers an trade-off between model efficiency and accuracy. By storing weights in 4-bit integers, it reduces model size by approximately 4× compared to FP16 representations. However, a key implementation challenge remains: mainstream hardware—including GPUs and NPUs—lacks native support for mixed-precision arithmetic operations. In practice, 4-bit values must be packed into larger data types such as INT32 for storage and transfer, then dequant prior to computation. This dequant process introduces additional overhead and can become a notable performance bottleneck unless carefully optimized through efficient pipline and memory access patterns. Thus, while W4A16 is promising for efficient inference, its practical speedup depends heavily on low-level kernel optimization and hardware-aware implementation. 2.3 Ascend 910 NPU Architecture The Huawei Ascend 910 NPU is a high-performance AI processor designed for both training and inference workloads, featuring a many-core architecture centered around specialized AI Cores. Each AI Core integrates dedicated compute, storage, and data movement units to accelerate matrix-intensive and vector-intensive operations prevalent in deep neural networks [ 4 ] . Prior works on NVIDIA GPUs [ 11 ] have explored similar W4A16 optimizations using Tensor Cores and shared memory tricks. However, the Ascend architecture’s distinct memory hierarchy, instruction set, and lack of warp-level primitives necessitate a ground-up redesign—motivating our platform-specific kernel development [ 3 ] . An AI core is organized around three primary components: compute units, the on-chip memory hierarchy, and Memory Transfer Engines (MTEs). As shown in Figure 1 , the compute units consist of a high-throughput cube core (AIC) for matrix multiplication (e.g., FP16 GEMM with 16\times 16\times 16 tiles), vector cores (AIV) for SIMD-like operations on data, and lightweight scalar cores handling control flow and instruction dispatch. Data reuse and minimized off-chip traffic are enabled by a dedicated on-chip memory hierarchy, which includes the L1 Buffer, the L0A/L0B input buffers, the L0C accumulation buffer, and the Unified Buffer (UB). Finally, high-bandwidth data movement between global memory (GM) and these on-chip buffers is managed by Memory Transfer Engines, which often perform on-the-fly data format conversions such as quantization or dequantization.The Ascend 910 operates in decoupled mode, the cube core and the vector core each operate under their own dedicated scalar scheduling units, which are separately deployed on the respective cores. An AI core is composed of one cube core and two vector cores on Ascend 910. Programming for Ascend 910 is typically performed via the Ascend C programming model [ 5 ] , which abstracts the underlying hardware while exposing explicit control over data placement and pipeline scheduling. Efficient kernel implementation thus requires co-design of computation tiling, memory access patterns, and inter-unit synchronization to fully saturate the cube units—the primary throughput bottleneck in matrix multiplication workloads. Figure 1: Ascend NPU architecture. 3 Method We propose an efficient implementation of W4A16 mixed-precision matrix multiplication kernel in Huawei Ascend 910 NPU, leveraging the heterogeneous architecture

Read the complete text