RSR-core: A High-Performance Engine for Low-Bit Matrix-Vector Multiplication
Mohsen Dehghankar, Abolfazl Asudeh · 2026
Read RSR-core: A High-Performance Engine for Low-Bit Matrix-Vector Multiplication by Mohsen Dehghankar, Abolfazl Asudeh (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-27462.txt.
Introduction
Dehghankar and Asudeh speed up low-bit matrix-vector products. Such products sit at the heart of neural network inference. Weights stored in one or two bits cut memory sharply. The RSR algorithm promises fast math for these cases. Prior code ran too high-level for real hardware kernels. Their RSR-core engine fills that gap. It ships tuned kernels for both CPU and CUDA. Binary and ternary weights are both supported. A HuggingFace link eases real deployment.
Catalog record
| CREATOR | Mohsen Dehghankar, Abolfazl Asudeh |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 16823 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.27462
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
RSR-core: A High-Performance Engine for Low-Bit Matrix-Vector Multiplication Mohsen Dehghankar, Abolfazl Asudeh Abstract . Matrix–vector multiplication is a fundamental building block in neural networks, vector databases, and large language models, particularly during inference. As a result, efficient matrix–vector multiplication engines directly translate into more efficient inference. Recent work has explored low-bit quantization of model weights, where matrices are represented using binary (1-bit) or ternary (1.58-bit) values while activation kept in higher precision. These representations enable efficient hardware-level computation. In parallel, algorithms such as Redundant Segment Reduction (RSR) provide theoretical guarantees for accelerating low-bit matrix–vector multiplication. However, existing implementations operate at the application level and cannot be efficiently integrated into hardware kernels, limiting practical performance. To bridge this gap, we present RSR-core, a high-performance engine that implements the RSR algorithm as optimized low-level kernels for both CPU and CUDA environments. RSR-core supports efficient matrix–vector multiplication for binary and ternary weight matrices and general vectors while enabling practical deployment of RSR algorithm in real inference pipelines. RSR-core is provided as a production-ready engine with HuggingFace integration for preprocessing low-bit models and running accelerated inference. Experimental results demonstrate significant performance improvements over baseline HuggingFace PyTorch multiplication, achieving up to 62× speedup on CPU and up to 1.9× speedup for token generation on CUDA for popular ternary LLMs. The source code is publicly available at RSR-core repository 1 1 1 https://github.com/UIC-InDeXLab/RSR-core . 1. Introduction Figure 1 . The layered architecture of RSR-core system. Inference is a critical stage in neural networks, particularly for large language models (LLMs), where it dominates both energy consumption and resource allocation ( 12 ; 7 ) . A key computational bottleneck during inference is matrix–vector multiplication Mv , where the matrix M represents fixed model weights after training and the vector v corresponds to input activations generated during execution. Improving the efficiency of matrix–vector multiplication therefore directly translates into faster and more resource-efficient neural model inference. Efficient matrix–vector multiplication is also central to many data management and retrieval workloads beyond neural inference ( 4 ; 1 ) . For example, vector database systems compute inner products between a query vector and indexed data points for nearest neighbor retrieval, where the indexed dataset can be viewed as a matrix and the query as the input vector ( 3 ) . Similarly, ranking over tabular datasets applies user-defined weight vectors to attribute columns to compute scores for sorting and filtering, where the table rows form the matrix and the weight vector defines the ranking function ( 9 ; 10 ) . One common approach to accelerating matrix–vector multiplication, specifically in model inference, is low-bit quantization of weight matrices ( 6 ; 11 ) . In low-bit quantized LLMs, matrices are represented using binary or ternary values while activation vectors remain in higher precision, enabling specialized hardware-efficient implementations that reduce computation cost during inference. Orthogonal to hardware-aware quantization approaches, algorithmic methods aim to further accelerate matrix–vector multiplication. In particular, Redundant Segment Reduction (RSR) ( 2 ) reduces the computational cost of low-bit matrix--vector multiplication by a logarithmic factor by exploiting redundancy in matrix structure. Specifically, identical column patterns 2 2 2 In the original paper, the multiplication v\cdot M is considered instead of Mv , and therefore rows (rather than columns) are grouped together. contribute identical partial results and can therefore be aggregated before multiplication, reducing the total number of required operations without affecting inference accuracy . However, existing implementations of RSR operate at the application level and cannot be efficiently integrated into hardware kernel-level execution, limiting their practical performance benefits in real inference systems. In this work, we address this gap by providing a kernel-level implementation of the RSR algorithm and enabling its deployment in practical inference pipelines. To the best of our knowledge, this is the first implementation of RSR optimized for both CPU and CUDA inference environments. We present RSR-core , a high-performance engine implementing RSR as optimized CPU and CUDA kernels for efficient low-bit matrix–vector multiplication. RSR-core provides a production-ready Python interface integrated with HuggingFace workflows ( 5 ) , enabling users to preprocess quantized models once and execute accelerated inference efficiently through an end-to-end workflow. We benchmark RSR-core against widely used hardware-optimized matrix multiplication implementations provided by the PyTorch ( 8 ) backend and used within the HuggingFace inference interface, as well as BitNet ( 11 ) . Experimental results show substantial performance improvements, achieving up to 62\times speedup on CPU and up to 1.9\times speedup on CUDA for popular ternary LLM inference workloads compared to PyTorch bfloat16 computation, while preserving exact inference accuracy. 2. System Overview We consider matrix–vector multiplication M\cdot v , where M is a binary matrix with entries in \{0,1\} or a ternary matrix with entries in \{-1,0,+1\} . The matrix corresponds to quantized model weights and remains fixed after preprocessing, while the vector v represents input activations arriving during inference, as discussed in the Introduction section. The goal is to accelerate this multiplication in practical inference pipelines while preserving exact computation. Algorithm overview. The Redundant Segment Reduction (RSR) algorithm ( 2 ) consists of two phases: an offline preprocessing applied once to the matrix M , followed by a fast online multiplication phase applied repeatedly during inference. During preprocessing , RSR splits the matrix into horizontal blocks of k rows and identifies identical column segments within each block, grouping columns that share the same bit pattern. The preprocessing stage produces auxiliary data structures—including column permutations, group boundaries, and scatter indices—that enable efficient reuse of intermediate results during inference. During inference , given an input vector v , RSR aggregates vector entries corresponding to identical column segments before multiplication and distributes contributions only once per unique segment. This reduces the number of arithmetic operations required for computing M\cdot v while preserving exact multiplication results without sacrificing the space usage. 2.1. System Architecture RSR-core is an optimized RSR execution engine using a layered architecture illustrated in Figure 1 . At its core, the system provides kernel-level implementations of RSR multiplication on both CPU (C kernels) and CUDA (GPU kernels). These kernels are exposed through intermediate multiplier abstractions and integrated into a Python interface that supports end-to-end preprocessing and inference workflows. The architecture enables users to preprocess quantized models once and reuse the generated artifacts across repeated inference calls. The full source code is publicly available at this repository . 2.2. Kernels Implementation While the RSR algorithm is conceptually straightforward, a direct Python implementation does not yield practical speedups due to interpreter overhead, general-purpose sorting, and inefficient memory access patterns during the gather, aggregate, and scatter phases. RSR-core therefore provides native kernel implementations in C (CPU) and CUDA (GPU) for both binary (1-bit) and ternary (1.58-bit) matrices. A key design choice spans all backends: preprocessing uses counting sort over the discrete pattern space ( 2^{k} buckets for binary, 4^{k} for ternary), achieving O(n+\text{buckets}) complexity per block instead of O(n\log n) from comparison-based sorting. Metadata is uniformly shrunk to compact types—permutation indices and group boundaries stored as 16-bit integers, and scatter targets encoded as fixed-size bitmasks rather than variable-length index arrays—reducing bandwidth pressure in the inference hot path. On the CPU, the C kernels fuse the gather and aggregate phases into a single pass over the