DiBA: Diagonal and Binary Matrix Approximation for Neural Network Weight Compression

Nobutaka Ono · 2026

Read DiBA: Diagonal and Binary Matrix Approximation for Neural Network Weight Compression by Nobutaka Ono (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-05994.txt.

Introduction

Nobutaka Ono proposes a compact way to store the weight matrices inside neural networks. Dense matrices appear in linear layers, attention projections, and embeddings, and they cost much memory. His method, DiBA, approximates a matrix with three diagonal matrices and two binary matrices of zeros and ones.

A middle dimension sets the trade off between stored size and accuracy of the approximation. Matrix products then need far fewer multiplications than the dense form. A greedy solver alternates exact updates for the diagonal parts with one bit improvement tests for the binary parts. A lighter variant retunes only the diagonal entries after swapping layers into DiBA form. On real model weights and two task studies, accuracy improved after replacement.

Catalog record

CREATORNobutaka Ono
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE40 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.05994

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

DiBA: Diagonal and Binary Matrix Approximation for Neural Network Weight Compression Nobutaka Ono Abstract In this paper, we propose DiBA (Diagonal and Binary Matrix Approximation), a compact matrix factorization for neural network weight compression. Many components of modern networks, including linear layers, 1 × 1 1\times 1 convolutions, attention projections, and embedding layers, have dense matrix weights. DiBA approximates A ∈ ℝ m × n A\in\mathbb{R}^{m\times n} by A ^ = D 1 ​ B 1 ​ D 2 ​ B 2 ​ D 3 \widehat{A}=D_{1}B_{1}D_{2}B_{2}D_{3} , where D 1 , D 2 , D 3 D_{1},D_{2},D_{3} are diagonal matrices and B 1 , B 2 B_{1},B_{2} are 0 / 1 0/1 binary matrices. The intermediate dimension k k controls the trade-off between theoretical storage and approximation accuracy. For matrix-vector products, DiBA decomposes dense multiplication into three element-wise scaling operations and two binary mixing operations, reducing the floating-point multiplication count from m ​ n mn to m + k + n m+k+n . For optimization, we introduce DiBA-Greedy, an alternating solver that combines closed-form least-squares updates for the diagonal factors with exact one-bit improvement tests for the binary factors. We also introduce DiBARD (DiBA with Retuning only Diagonal factors), which replaces dense-matrix layers by DiBA factors, freezes the binary matrices, and retunes only the diagonal entries on downstream data. This preserves compact binary mixing without discrete search during adaptation. On 40 dense weight matrices extracted from public pretrained models, DiBA-Greedy yields consistent SNR improvements as the theoretical storage ratio increases. After DiBA replacement in two component-replacement studies, DiBARD improves DistilBERT/WikiText masked-token accuracy from 0.4447 to 0.5210 and Speech Commands test accuracy for an Audio Spectrogram Transformer from 0.7684 to 0.9781 without reoptimizing the binary factors. 1 Introduction Dense matrices remain one of the dominant storage and computation units in modern neural networks. Linear layers, token embeddings, attention projections, and 1\times 1 convolutions all store learned transformations as dense weight matrices. These matrices often account for a large fraction of model size, memory traffic, and matrix-vector or matrix-matrix arithmetic at inference time. Compressing pretrained dense weights while preserving downstream accuracy is therefore central to model distribution, on-device inference, low-memory deployment, and task-specific adaptation. A broad literature has studied compact neural representations. Pruning removes unimportant connections, filters, or channels, starting from classical second-order criteria such as Optimal Brain Damage and Optimal Brain Surgeon and extending to modern fine-tuning and one-shot pruning methods for pretrained models [ 1 , 2 , 3 , 4 , 5 , 6 , 7 , 8 , 9 ] . Quantization replaces real-valued weights by a small set of discrete codes, including vector quantization and weight sharing, binary and ternary networks, integer-arithmetic inference, post-training quantization, and weight-only quantization for large language models [ 10 , 11 , 12 , 13 , 14 , 15 , 16 , 17 , 18 , 19 , 20 , 22 , 23 , 24 , 25 , 26 , 27 ] . Low-rank and tensor decompositions exploit linear structure in matrices and convolution kernels [ 28 , 29 , 30 ] , while knowledge distillation transfers the behavior of a larger teacher model to a compact student [ 31 , 32 ] . Together, these approaches show that pretrained dense weights contain substantial redundancy, but they expose different trade-offs between storage, arithmetic structure, optimization difficulty, and downstream recovery. Among these families, quantization is especially practical on existing hardware, but its optimization problem is intrinsically discrete. In the simplest case, each scalar weight is rounded to the nearest quantization level. In realistic low-bit settings, however, one must simultaneously choose discrete assignments, scales and zero points, layer-, channel-, or block-wise bitwidths, and the reconstruction or task loss to optimize after quantization. As a result, low-bit quantization is often treated as a nonconvex problem with discrete constraints, mixed-integer structure, or binary assignment variables [ 34 , 22 ] . Existing methods use straight-through estimators [ 33 , 13 , 14 ] , learned step sizes or clipping parameters [ 20 , 21 ] , ADMM or proximal formulations [ 34 , 35 ] , loss-aware objectives [ 36 ] , Hessian- or second-order approximations for rounding and mixed precision [ 37 , 38 , 22 , 23 , 26 ] , and scale transformations that handle activation outliers or salient channels [ 25 , 27 ] . These techniques are powerful, but the underlying optimization remains hard in the very low-bit regime because it combines discrete search, continuous relaxation, local reconstruction, and calibration-data dependence. We take a different route. Rather than assign each individual weight to one of a few quantization levels, we approximate an entire dense matrix by interleaving diagonal scaling with shared 0/1 binary mixing. For a target matrix A\in\mathbb{R}^{m\times n} , DiBA represents \widehat{A}=D_{1}B_{1}D_{2}B_{2}D_{3}, (1) where D_{1},D_{2},D_{3} are real diagonal matrices, B_{1}\in\{0,1\}^{m\times k} and B_{2}\in\{0,1\}^{k\times n} are binary matrices, and the intermediate dimension k controls the storage-accuracy trade-off. The binary factors can be stored, in the theoretical accounting used in this paper, at one bit per entry, while the real-valued degrees of freedom are concentrated in the three diagonal factors. Figure 1: DiBA provides a structured approximation to a dense matrix using three real diagonal factors and two 0/1 binary factors; k controls the storage–accuracy trade-off. This representation is useful for three reasons. First, the 0/1 factors define shared selection-and-summation patterns rather than element-wise codebook entries; signs and magnitudes are carried by the diagonal scales. Second, although \widehat{A} has rank at most k , DiBA is not an unconstrained real-valued low-rank factorization: its large factors are one-bit mixing patterns, and its real-valued degrees of freedom are diagonal. Third, the continuous and discrete parts admit a natural alternating optimization strategy: the diagonal factors can be refit by closed-form least-squares updates, while the binary factors can be improved by exact one-bit objective-difference tests in the corresponding binary subproblems. We instantiate this idea with DiBA-Greedy, an alternating solver for matrix reconstruction. Given fixed binary factors, DiBA-Greedy updates D_{1} , D_{2} , and D_{3} by least squares. Given fixed diagonal factors and the other binary factor, it evaluates whether flipping an entry of B_{1} or B_{2} decreases the squared Frobenius reconstruction error, accepting only flips whose exact objective change satisfies \Delta<-\tau for a fixed flip tolerance \tau . The method is not claimed to find a global optimum, but each accepted flip is an exact improvement for its local binary subproblem, and the continuous factors are repeatedly refit to the current binary structure. We further introduce DiBARD, a diagonal-only retuning procedure for DiBA-replaced neural-network components. After replacing a dense neural-network weight matrix by DiBA factors, DiBARD freezes the binary matrices B_{1} and B_{2} and updates only the diagonal entries of D_{1} , D_{2} , and D_{3} on downstream data. This design avoids discrete search during task adaptation: the binary mixing structure remains fixed, and the recovery stage is a small continuous optimization over element-wise scales. We do not claim measured speedups from bit-packed kernels in this work. Our evaluation focuses on matrix approximation quality, theoretical storage ratio, and the extent to which diagonal-only retuning can recover downstream task performance. Our contributions are as follows. First, we formulate DiBA, a diagonal-and-binary factorization for approximating dense neural-network

Read the complete text