FairyFuse: Multiplication-Free LLM Inference on CPUs via Fused Ternary Kernels Conference: ; ; CCS: Computing methodologies Vector / streaming algorithms CCS: Computing methodologies Parallel algorithms CCS: Computing methodologies Natural language generation

Fei Zuo, Xiaoyan Xi, Quanyi Zeng, Feiyu Wang, Ho Fai Leung · 2026

Read FairyFuse: Multiplication-Free LLM Inference on CPUs via Fused Ternary Kernels Conference: ; ; CCS: Computing methodologies Vector / streaming algorithms CCS: Computing methodologies Parallel algorithms CCS: Computing methodologies Natural language generation by Fei Zuo, Xiaoyan Xi, Quanyi Zeng, Feiyu Wang, Ho Fai Leung (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-20913.txt.

Introduction

Zuo and colleagues present FairyFuse, a CPU inference system for ternary language models whose weights take only minus one, zero, or plus one. It fuses the subcomputations of each layer into one AVX-512 loop of masked additions and subtractions, using no floating-point multiplications at all. Kernel speed rises about 30 times on bandwidth-limited CPUs, and end-to-end it reaches 32.4 tokens each second on one Intel Xeon chip with near-lossless quality.

Catalog record

CREATORFei Zuo, Xiaoyan Xi, Quanyi Zeng, Feiyu Wang, Ho Fai Leung
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE60001
LICENSEhttp://creativecommons.org/publicdomain/zero/1.0/
RIGHTSCC0
SOURCEarXiv

Source: https://arxiv.org/abs/2604.20913

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

FairyFuse: Multiplication-Free LLM Inference on CPUs via Fused Ternary Kernels Conference: ; ; CCS: Computing methodologies Vector / streaming algorithms CCS: Computing methodologies Parallel algorithms CCS: Computing methodologies Natural language generation Fei Zuo, Xiaoyan Xi, Quanyi Zeng, Feiyu Wang, Ho Fai Leung Abstract . Large language models are increasingly deployed on CPU-only platforms, from on-device assistants to privacy-sensitive edge servers, where memory bandwidth is the dominant bottleneck for autoregressive token generation. Weight quantization to four bits or below reduces memory pressure, yet existing inference systems still dequantize the compressed weights and execute floating-point multiplications, leaving much of the potential bit-width savings unrealized. Ternary weights, which take values in { − 1 , 0 , + 1 } \{-1,0,+1\} , offer a more radical alternative: every weight-activation product can be replaced by a conditional addition, subtraction, or no-op, eliminating multiplications altogether. Fairy2i recently demonstrated that complex-valued quantization-aware training can produce ternary LLMs whose quality rivals FP16, yet its reference runtime still dispatches the quantized weights through standard floating-point routines, leaving the ternary structure unexploited at the hardware level. Here we present FairyFuse, a companion inference system for Fairy2i that realizes the multiplication-free promise on commodity CPUs for the first time. FairyFuse fuses the eight real-valued sub-GEMVs of each widely-linear layer into a single AVX-512 loop driven entirely by masked additions and subtractions; assembly inspection confirms that the inner loop contains zero floating-point multiplication instructions. Roofline analysis reveals a structural asymmetry: the 16 × 16\times data compression from ternary packing shifts deeply memory-bound GEMV operations toward the compute ridge on bandwidth-limited CPUs, yielding a 29.6 × 29.6\times kernel speedup, whereas the same compression provides negligible benefit on bandwidth-rich GPUs. End-to-end, FairyFuse sustains 32.4 tokens per second on a single Intel Xeon 8558P socket, outperforming llama.cpp Q4_K_M by 1.24 × 1.24\times while preserving near-lossless quality (WikiText-2 perplexity 5.52 versus 5.47 for FP16; average downstream accuracy 66.0%). All measurements are reproducible with a coefficient of variation below 2%. 1. Introduction Large language models (LLMs) ( vaswani2017attention ; touvron2023llama2 ) are increasingly deployed beyond datacenters: on-device assistants, desktop copilots, and embedded agents all demand low-latency, privacy-preserving inference without discrete GPUs. On such platforms the CPU is the only available compute engine, and memory bandwidth is the dominant bottleneck for autoregressive decoding ( pope2023efficiently ) . Weight quantization is the primary lever for fitting billion-parameter models into limited DRAM and for keeping the weight-streaming working set cache-feasible. Figure 1 . FairyFuse compared with existing approaches for ternary LLM inference. (a) Conventional systems dequantize ternary weights to FP16 and perform standard multiplication. (b) LUT-based systems (e.g., T-MAC) replace multiplications with table lookups but incur on-chip memory pressure. (c) FairyFuse directly applies masked addition and subtraction on packed 2-bit weights, with zero multiplications and zero lookup-table overhead. At the extreme end of this spectrum, ternary weights \{-1,0,+1\} achieve 16\times compression over FP32, and each weight-activation product reduces to a conditional addition, subtraction, or no-op, eliminating multiplications entirely. This idea has roots in early ternary ( li2016ternary ; zhu2017trained ) and binary ( rastegari2016xnor ; hubara2016binarized ) networks, and recent work confirms that ternary LLMs can match FP16 quality: BitNet b1.58 ( ma2024era ) trains models from scratch at ternary precision, while Fairy2i ( feng2025fairy2i ) post-trains existing checkpoints into complex-valued ternary models via quantization-aware training. Fairy2i solves the algorithmic side of the problem; the systems challenge is to translate ternary weights into multiplication-free execution on real hardware. Existing inference systems, however, do not yet exploit the ternary structure at the microarchitectural level. Fairy2i’s reference runtime, designed primarily as a research prototype for validating model quality, dispatches the quantized weights through standard PyTorch linear operators, leaving room for a dedicated inference engine that eliminates multiplications at the instruction level. Similarly, llama.cpp ( llamacpp2024 ) applies dequantization followed by floating-point FMA for its Q2_K and Q4_K_M formats, so the low-bit encoding serves only as a storage optimization while the compute path remains multiply-heavy. An alternative line of work replaces multiplication with bitwise lookup tables: T-MAC ( wei2024tmac ) and BitNet.cpp ( bitnetcpp2024 ) use VPSHUFB -based LUT lookups, but the resulting kernels incur indirect memory accesses and non-trivial on-chip LUT footprints that compete with activation data for L1/L2 cache bandwidth, limiting sustained throughput on wide SIMD units. Our starting observation is that ternary arithmetic on x86 CPUs admits a strictly simpler datapath than either dequantization or table lookup. We pack sixteen \{-1,0,+1\} weights into a single 32-bit word, decode positive and negative masks in a single-cycle _pext_u32 pair (BMI2), and accumulate via AVX-512 masked vaddps / vsubps . The resulting inner loop contains zero floating-point multiplications and zero table lookups, a property we verify through assembly inspection. Realizing this idea for complex-valued models introduces an additional complication. Fairy2i’s widely-linear representation decomposes each linear layer into eight coupled real-valued sub-GEMVs. A naive schedule launches eight separate parallel regions, repeatedly loads the same activation tiles, spills intermediate buffers to DRAM, and re-decodes related mask patterns, erasing the bandwidth gains from ternary packing. To overcome this overhead, we design a Fused Widely-Linear Kernel that consolidates all eight sub-GEMVs into one row-parallel AVX-512 loop. Four design decisions make fusion profitable: each decoded mask pair serves both real and imaginary accumulation paths, amortizing decode cost (O1); activation registers are loaded once and shared across all eight sub-GEMVs (O2); complex conjugation reduces to a precomputed sign-flipped alias, eliminating per-lane negation (O3); and all partial sums remain register-resident until the final horizontal reduction, removing intermediate DRAM traffic (O4). Roofline analysis confirms that the 16\times compression shifts GEMV from deeply memory-bound (AI {}=0.25 ) toward the CPU ridge point (AI {}=8 ), yielding a 29.6\times speedup over FP32, while the same compression provides negligible benefit on bandwidth-rich GPUs. End-to-end, FairyFuse sustains 32.4 tok/s on a single Intel Xeon 8558P socket, 1.24\times faster than llama.cpp Q4_K_M, while preserving near-lossless quality: WikiText-2 perplexity is 5.52 (vs. 5.47 FP16) and average downstream accuracy is 66.0%. Assembly analysis confirms zero multiplication instructions in the inner loop, and all measurements are reproducible with CV {}<{} 2%. In summary, this paper makes three contributions: • We implement the first ternary-weight GEMV kernel on x86 CPUs that uses only masked AVX-512 additions and subtractions, with no floating-point multiplications or table lookups in the inner loop. • We consolidate the eight sub-GEMVs of a complex-valued widely-linear layer into a single SIMD-friendly loop via mask reuse, input sharing, sign-swap aliasing, and register-resident accumulation, achieving 1.55\times speedup over unfused execution. • Through full-stack evaluation on a 7B-parameter model, we demonstrate that ternary packing makes CPUs, not GPUs, the natural target for extreme quantization, and deliver throughput competitive with 4-bit baselines at 2-bit storage. The rest of the paper is organized as follows. Section 2 provides background on CPU inference, ternary quantization, and the widely-linear GEMV structure. Section 3 presents the FairyFuse algorithm and fused kernel design. Section 4 describes the implementation. Section 5 evaluates kernel performance, end-to-end throughput and quality, Roofline analysis, and design ablations. Section 6 discusses related work, and Section 7 concludes. 2. Background and Motivation 2.1. LLM Inference on CPUs Autoregressive large language model inference ( vaswani2017attention ) is dominated by the decode phase, where each new token is produced by applying a stack of

Read the complete text