nncase: An End-to-End Compiler for Efficient LLM Deployment on Heterogeneous Storage Architectures
Hui Guo, Qihang Zheng, Chenghai Huo, Dongliang Guo, Haoqi Yang, Yang Zhang · 2025
Read nncase: An End-to-End Compiler for Efficient LLM Deployment on Heterogeneous Storage Architectures by Hui Guo, Qihang Zheng, Chenghai Huo, Dongliang Guo, Haoqi Yang, Yang Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-21571.txt.
Introduction
Hui Guo, Qihang Zheng, and their co-authors present nncase, an open-source compiler that prepares large language models to run efficiently on varied hardware. Deployments face memory architectures that differ sharply between chips, and traditional compilers cope through fragmented workflows that are costly to adapt per target.
At the center of nncase is an e-graph-based term rewriting engine. E-graphs store many equivalent versions of a computation at once, sidestepping the phase ordering problem, where early optimization choices block better later ones. Three modules sit on top: one adapts code to different vector units, one searches parallel strategies with communication costs in mind, and one schedules work for on-chip cache reuse. In evaluations, nncase outperformed mainstream frameworks such as MLC LLM and Intel IPEX on Qwen3 models. On plain CPUs it matched the hand-optimized llama.cpp, a notable result for automated compilation.
Catalog record
| CREATOR | Hui Guo, Qihang Zheng, Chenghai Huo, Dongliang Guo, Haoqi Yang, Yang Zhang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 61 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-21571
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
nncase: An End-to-End Compiler for Efficient LLM Deployment on Heterogeneous Storage Architectures Hui Guo, Qihang Zheng, Chenghai Huo, Dongliang Guo, Haoqi Yang, Yang Zhang Abstract The efficient deployment of large language models (LLMs) is hindered by memory architecture heterogeneity, where traditional compilers suffer from fragmented workflows and high adaptation costs. We present nncase, an open-source, end-to-end compilation framework designed to unify optimization across diverse targets. Central to nncase is an e-graph-based term rewriting engine that mitigates the phase ordering problem, enabling global exploration of computation and data movement strategies. The framework integrates three key modules: Auto Vectorize for adapting to heterogeneous computing units, Auto Distribution for searching parallel strategies with cost-aware communication optimization, and Auto Schedule for maximizing on-chip cache locality. Furthermore, a buffer-aware Codegen phase ensures efficient kernel instantiation. Evaluations show that nncase outperforms mainstream frameworks like MLC LLM and Intel IPEX on Qwen3 series models and achieves performance comparable to the hand-optimized llama.cpp on CPUs, demonstrating the viability of automated compilation for high-performance LLM deployment. The source code is available at https://github.com/kendryte/nncase . 1 Introduction As large language models (LLMs) such as GPT-4 [ 34 ] and Llama [ 44 ] scale beyond the trillion-parameter mark, they impose exponential demands on computation and storage. This surge presents a fundamental challenge: navigating the heterogeneity of memory architectures. Contemporary deployment landscapes are bifurcated into distributed systems—spanning multi-server clusters and specialized accelerators (e.g., NVIDIA H100 arrays, Cerebras WSE [ 27 , 6 ] )—and single-node setups on standalone devices. Despite their topological differences, these architectures share critical constraints: (1) Deep Memory Hierarchies: Ranging from KB-level SRAM/L1 caches to GB-level global HBM/DRAM. (2) Heterogeneous computation Units: The coexistence of scalar, vector (e.g. AVX-512 [ 40 ] ) and matrix (e.g. Intel AMX [ 47 ] ) units. (3) The Memory Wall: With the memory bandwidth growing only 20% annually versus a 100% surge in compute power, data movement has become the primary bottleneck for LLM inference. Efficient deployment requires orchestrating the synergy between memory and computation, specifically targeting hierarchical reuse and heterogeneous unit adaptation. However, conventional compilers often necessitate distinct optimization pipelines for uniform and non-uniform memory architectures. This dichotomy introduces prohibitive adaptation costs and creates technical barriers to global optimization. To bridge this gap, we present nncase, an open-source end-to-end compilation framework designed to unify LLM deployment across heterogeneous memory architectures. The cornerstone of nncase is its "unified distributed compilation paradigm." By modeling all targets via a Non-Uniform Memory Access (NUMA) abstraction, nncase decouples the compilation workflow from physical topology, achieving a "compile once, adapt everywhere" capability. The framework is underpinned by an e-graph-based term rewriting engine that employs equality saturation. Unlike the greedy strategies of traditional deep learning compilers, our engine circumvents the phase ordering problem, exploring a comprehensive design space without compromising semantic integrity. Furthermore, by integrating a Roofline-based cost model, nncase performs multi-objective optimization to balance memory footprint, latency, and communication overhead. Using the capacity of the e-graph to represent equivalence, nncase enables three distinct optimization passes: 1) Auto Vectorize: Addressing the challenge of mapping tensors to various hardware units, this module introduces MetaPackOperation and FoldNopPack rules. It simultaneously generates multiple packed layout candidates within the e-graph, dynamically adjusting packing factors to eliminate redundant layout permutations. This allows the compiler to strike an optimal balance between data layout conversion and computing unit saturation. 2) Auto Distribution: To manage data parallelism, we adopt the SBP (Split, Broadcast, Partial-value) [ 58 ] abstraction. Operating on the principle that "nodes with consistent SBP attributes are equivalent," this module embeds the distributed strategy search space directly into the e-graph. It automatically balances communication costs against computational efficiency, adapting seamlessly to both multi-device clusters and single-node logical distributions. 3) Auto Schedule: Addressing the complexity of on-chip memory management, we introduce the nncase Tensor Template (NTT) Library. This library abstracts hardware primitives into atomic microkernels (µkernels). We then decompose scheduling into structural optimization (e.g., Loop Fusion) through Monte Carlo Tree Search (MCTS) [ 32 ] and parametric optimization (e.g., Tiling) through Mixed-Integer Nonlinear Programming (MINLP) [ 4 ] . This hierarchical approach achieves register-level efficiency while avoiding the prohibitive search times of exhaustive methods. We evaluated nncase using the Qwen3 [ 56 ] model family (0.6B and 1.7B) on an AMD Ryzen 9 5900X platform in single-core and multi-core configurations. Experimental results demonstrate that nncase significantly outperforms mainstream frameworks such as MLC LLM [ 33 ] and Intel IPEX [ 20 ] , while matching or surpassing the performance of the hand-optimized library llama.cpp [ 12 ] . In summary, the contributions of this paper include the following: • We design and open-source nncase , a unified compilation framework that bridges the gap between uniform and non-uniform memory architectures. • We propose an e-graph-based Auto Vectorize mechanism that resolves tensorization challenges across diverse compute units. • We develop an Auto Distribution method that embeds distributed strategy search into the e-graph, enabling topology-agnostic parallelism. • We introduce a hierarchical Auto Schedule system backed by the NTT Library, automating the optimization of on-chip memory reuse and instruction throughput. 2 Background and Motivation 2.1 Data Layout Optimization Data layout optimization remains a fundamental bottleneck for deep learning compilers targeting heterogeneous architectures. Modern processors have evolved from purely scalar designs to sophisticated hybrids integrating both Vector Processing Units (e.g., AVX-512, NEON [ 40 , 14 ] ) and dedicated Tensor Processing Units (e.g., Intel AMX [ 47 ] , Arm SME [ 51 ] , Apple AMX [ 52 ] ). To harness this heterogeneous compute power, industry optimization strategies generally fall into two paradigms: Kernel-Level Optimization and Graph-Level Layout Propagation. Kernel-Level Optimization. This paradigm treats layout optimization as a local problem isolated to individual operators. Representative frameworks, including vendor libraries (e.g., Intel MKL [ 19 ] , Arm Compute Library [ 3 ] ) and traditional compilers (e.g., Halide [ 36 ] , TVM [ 8 ] ), focus on aligning data with hardware intrinsics. For element-wise operations, loop vectorization is employed to match SIMD lane width; for compute-intensive operators like GEMM or Convolution, Just-In-Time (JIT) packing mechanisms (pioneered by GotoBLAS [ 15 ] ) are used to mitigate cache associativity conflicts. Critique: While locally optimal, this approach suffers from layout thrashing. By failing to exploit layout reusability across the computation graph, it incurs redundant packing and unpacking overheads at operator boundaries, significantly degrading end-to-end inference latency. Graph-Level Layout Propagation. To address the redundancy of local packing, recent deep learning compilers employ global analysis to coordinate data layouts across the graph. Frameworks like TPP-MLIR [ 13 ] and ALT [ 55 ] utilize term rewriting systems to propagate preferred layouts between upstream and downstream nodes, fusing layout transformations via constant folding or dead code elimination. Critique: Although these methods reduce transformation overheads, they lack the granularity to navigate the Vector-Tensor trade-off in modern hybrid architectures. When Vector Units (requiring 1D layouts) and Tensor Units (requiring blocked 2D layouts) coexist, the optimal strategy is not static. Existing approaches often fail to model the complex cost trade-off between the acceleration gains of 2D tensorization and the overhead of 1D-to-2D layout permutation, leading to sub-optimal execution plans in mixed-precision or mixed-unit scenarios. 2.2 Distributed Strategy Search The distributed strategy search