1 BIT IS ALL WE NEED: Binary Normalized Neural Networks
Eduardo L. L. Cabral, Paulo Pirozelli, Larissa Driemeier · 2025
Read 1 BIT IS ALL WE NEED: Binary Normalized Neural Networks by Eduardo L. L. Cabral, Paulo Pirozelli, Larissa Driemeier (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-07025.txt.
Introduction
Eduardo Lobo Lustoda Cabral, Paulo Pirozelli, and Larissa Driemeier introduce binary normalized neural networks in a 2025 paper whose title claims one bit is all we need. Modern neural networks keep growing, above all language models and large image models. Their size makes them hard to deploy, so researchers want models that use less memory and run more cheaply.
Their new layer type stores every parameter as a single bit. Weights in the kernels and the biases can each be only zero or one. Versions of the layer exist for the usual layer kinds, from fully connected and convolutional layers to attention, and each differs only slightly from its standard counterpart. To test the idea, the team built two models: one that sorts images into classes, and one language decoder. Their results suggest such models can work while using far less memory.
Catalog record
| CREATOR | Eduardo L. L. Cabral, Paulo Pirozelli, Larissa Driemeier |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 47 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-07025
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
1 BIT IS ALL WE NEED: Binary Normalized Neural Networks Eduardo L. L. Cabral, Paulo Pirozelli, Larissa Driemeier Abstract The increasing size of large neural network models, specifically language models and foundational image models, poses deployment challenges, prompting efforts to reduce memory requirements and enhance computational efficiency. These efforts are critical to ensure practical deployment and effective utilization of these models across various applications. In this work, a novel type of neural network layers and models is developed that uses only single-bit parameters. In this novel type of models all parameters of all layers, including kernel weights and biases, only have values equal to zero or one. This novel type of models uses layers named as binary normalized layer. These binary normalized layers can be of any type, such as fully connected, convolutional, attention, etc., and they consist of slight variations of the corresponding conventional layers. To show the effectiveness of the binary normalized layers, two different models are configured to solve a multiclass image classification problem and a language decoder to predict the next token of a sequence. The model to solve the image classification has convolutional and fully connected layers, and the language model is composed of transformer blocks with multi-head attention. The results show that models with binary normalized layers present almost the same results obtained by equivalent models with real 32-bit parameters. The binary normalized layers allow to develop models that use 32 times less memory than current models and have equivalent performance. Besides, the binary normalized layers can be easily implemented on current computers using 1-bit arrays, and do not require the development of dedicated electronic hardware. This novel type of layers opens a new era for large neural network models with reduced memory requirements that can be deployed using simple and cheap hardware, such as mobile devices or only cpus. Keywords Neural networks \cdot binary parameters \cdot binary normalized layers 1 Introduction Recent advances in machine learning techniques and hardware have enabled extraordinary performance in a wide range of applications — from traditional tasks such as pattern recognition and natural language processing to more complex domains, including autonomous control systems and the discovery of new materials. These developments have been made possible thanks to improved neural network architectures, increased computational power, and the availability of large datasets Goodfellow et al. 2016 . However, despite such impressive progress, current artificial intelligence models still face serious limitations when applied to embedded systems. Most state-of-the-art AI solutions rely heavily on cloud computing infrastructure and high-performance specialized hardware. Deep neural networks, such as those used for image classification, often require billions of floating point operations to process a single input sample Henzinger et al. 2021 . The increasing size of large-scale AI models, especially language models and foundational image models, introduces significant deployment challenges. These models require substantial computational resources, energy, and memory requirements that are difficult to satisfy outside data centers. This dependency makes it impractical to implement these models in systems with limited computational resources, especially in contexts where connectivity is restricted or nonexistent. Applications in isolated environments, such as underwater, underground, aerospace, or agricultural systems, face severe communication challenges, including physical interference, high latency, and limited bandwidth. As discussed in Plastiras et al. 2018 , local data processing becomes an evolutionary necessity to ensure the autonomy and responsiveness of such systems, particularly where rapid decision making is required under resource constraints. To address these constraints, quantization has emerged as a key technique for optimizing neural networks in resource-limited environments Henzinger et al. 2021 . Instead of relying on high-precision floating-point arithmetic (e.g., 32-bit), quantization uses lower-bit integer formats. It typically operates in the 2 to 8 bit range, which is widely adopted in industry with minimal impact on accuracy Henzinger et al. 2021 . This approach enables significantly reduced memory usage and bandwidth requirements, with compression ratios ranging from 35\times to 49\times , as shown by Han et al. 2016 . It also provides substantial speed-ups, achieving up to 3 times faster execution on standard CPUs and up to 10 times on specialized fixed point hardware such as Qualcomm DSPs with HVX support Jacob et al. 2018 . Furthermore, energy efficiency is greatly improved, which is especially critical for mobile and edge devices Krishnamoorthi 2018 ; Banner et al. 2019 . However, quantization presents challenges, particularly in preserving accuracy due to the non-differentiable nature of quantization functions and the sensitivity to ranges of varying values in weights and activations Hubara et al. 2017 . To address these issues, several techniques have been developed. Post-Training Quantization (PTQ) and Quantization-Aware Training (QAT) are two main strategies to reduce the precision of parameters of neural network models. PTQ applies quantization after a model has been fully trained in high precision (typically 32-bit floating point), aiming to reduce model size and inference latency without retraining. In contrast, QAT simulates quantization effects during training, allowing the model to learn and adapt to the information loss introduced by low-precision representations. As a result, QAT generally achieves significantly higher accuracy than PTQ, particularly at lower bit-widths. Several key studies have shaped the development of both PTQ and QAT. Jacob et al. 2018 presented a foundational quantization scheme that has been widely adopted in TensorFlow Lite, offering practical guidelines for deploying low-precision models in real-world applications. Krishnamoorthi 2018 provided a comprehensive white paper that systematically covers the principles and implementation details of both PTQ and QAT, serving as a central reference for researchers. For low-bit PTQ, Banner et al. 2019 proposed ACIQ (Analytical Clipping for Integer Quantization), a method for minimizing quantization error without retraining, particularly effective in the 4-bit setting. Regarding QAT specifically, Hubara et al. 2017 introduced a seminal approach to training Quantized Neural Networks (QNNs) with low-precision weights and activations using the Straight-Through Estimator (STE) to handle non-differentiability. As proposed in Jacob et al. 2018 , weights and biases are stored in 32-bit floating-point format to allow precise updates, but are quantized during the forward pass to simulate low-precision inference. Backpropagation still occurs using full-precision gradients, and the STE is employed to handle the non-differentiability of quantization. This dual approach ensures that small gradient updates are not lost, which would happen if parameters were permanently quantized during training. Once training is complete, only the low-precision parameters are retained for efficient inference. In van den Oord et al. 2018 the authors discussed the Vector Quantised-Variational AutoEncoder (VQ-VAE), a generative model that learns discrete latent representations. In Choi et al. 2019 the authors proposed PACT (Parameterized Clipping Activation), a method that improves activation quantization by learning optimal clipping thresholds during training. The field has also advanced with techniques such as SAWB (Statistical Aware Weight Binning) for weight quantization, which selects quantization ranges based on the weight distribution to reduce accuracy loss in low-precision networks. Zhuang et al. 2018 proposed training strategies for quantized networks, including two-stage optimization, progressive quantization, and the use of a full-precision teacher model to guide learning and improve final performance. Recently, Cabral and Driemeier 2025 explored the trade-offs involved in using low-bit representations for network weights. Their study demonstrates that 2.32-bit weights—corresponding to five discrete levels—can provide a favorable balance between memory efficiency and model accuracy. They also