BILLNET: A Binarized Conv3D-LSTM Network with Logic-gated residual architecture for hardware-efficient video inference
Van Thien Nguyen, William Guicquero and Gilles Sicard · 2025
Read BILLNET: A Binarized Conv3D-LSTM Network with Logic-gated residual architecture for hardware-efficient video inference by Van Thien Nguyen, William Guicquero and Gilles Sicard (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-14495.txt.
Introduction
Van Thien Nguyen, William Guicquero, and Gilles Sicard design a tiny video network that runs on strict hardware, in a paper from two thousand twenty-five. Video models that mix three-dimensional filters with memory cells score well but eat memory and compute. Their network shrinks both parts to on-off values.
The costly filter is split into cheaper steps, and a logic-gated path keeps accuracy up. A staged training plan lets the memory cells be fully shrunk as well. On a gesture data set the model reached high accuracy with very low memory and compute. The design suits chips with tight limits.
Catalog record
| CREATOR | Van Thien Nguyen, William Guicquero and Gilles Sicard |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 23 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-14495
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
BILLNET: A Binarized Conv3D-LSTM Network with Logic-gated residual architecture for hardware-efficient video inference Van Thien Nguyen, William Guicquero and Gilles Sicard Abstract Long Short-Term Memory (LSTM) and 3D convolution (Conv3D) show impressive results for many video-based applications but require large memory and intensive computing. Motivated by recent works on hardware-algorithmic co-design towards efficient inference, we propose a compact binarized Conv3D-LSTM model architecture called BILLNET, compatible with a highly resource-constrained hardware. Firstly, BILLNET proposes to factorize the costly standard Conv3D by two pointwise convolutions with a grouped convolution in-between. Secondly, BILLNET enables binarized weights and activations via a MUX-OR-gated residual architecture. Finally, to efficiently train BILLNET, we propose a multi-stage training strategy enabling to fully quantize LSTM layers. Results on Jester dataset show that our method can obtain high accuracy with extremely low memory and computational budgets compared to existing Conv3D resource-efficient models. I Introduction Video recognition has recently drawn substantial attention due to the success of several Deep Neural Networks (DNNs) [ 1 ] , [ 2 ] , [ 3 ] and the increasing number of large-scale video datasets [ 4 ] , [ 5 ] , [ 6 ] . Compared to other tasks like image classification that only relies on spatial data, video recognition is much more complex since it also requires extracting the underlying temporal features in the time direction. Among existing model architectures for spatio-temporal pattern recognition, Conv3D [ 7 ] and Recurrent Neural Networks (RNNs), e . g . LSTM [ 8 ] , have demonstrated to be relevant for learning latent spatio-temporal representations. However, these model components exhibit hardware-related drawbacks such as large memory requirements so as a high computational complexity. For instance, Conv3D expands the convolution kernel to the time direction for capturing local temporal features, therefore increasing both the local memory and computational needs by an order of magnitude compared to Conv2D. On the other hands, LSTM is computationally expensive because of its stateful nature, i . e ., computing the current features taking into account previous states. Consequently, designing a hardware-compliant Conv3D-LSTM model for embedded inference applications remains a significant challenge. Motivated by the need for efficient video inference, recent works have been focusing on the design of light-weight architectures [ 9 ] , [ 10 ] or hardware-aware network pruning [ 11 ] . Another approach to accelerate the computation during inference and further reduce the hardware-related costs consists in lowering the bitwidth of model’s weights and activations [ 12 ] . Even though significant works on Quantization Aware Training (QAT) [ 13 ] , [ 14 ] have been done in recent years, they are mostly focused on the quantization of feed-forward Convolutional Neural Networks (CNNs). On the contrary, training fully quantized models that embed recurrent layers like LSTMs remains an important issue. Indeed, quantizing the hidden states of LSTM involves a quantization error that is accumulated throughout the data sequence due to its recurrent nature, hence implying an overall accuracy degradation. It may explain why existing approaches [ 15 ] , [ 16 ] are limited to the use of quantization regarding the LSTM weights while keeping activations in a floating-point representation. This work thus aims at demonstrating that a fully-quantized model can also be deployed for video-based inference. To this end, we propose a hardware-compliant Conv3D-LSTM architecture called BILLNET on which binarization techniques are applied to further reduce the model size as well as the computational costs. Our main contributions are then: • A compact Binarized Conv3D-LSTM model architecture with a MUX-OR skip connection mechanism, • A multi-stage training procedure that provides a fully quantized BILLNET with bitshift normalization (removing additional biases related to batch normalization). II Related works II-A Residual connections for video inference Residual learning such as element-wise addition [ 17 ] and attention mechanism [ 18 ] , [ 19 ] is firstly introduced in 2D CNNs in order to increase network expressivity, favor feature reuse while easing the back-propagation for deeper models. Since Conv3D has become a more preferable option for video recognition than its 2D counterpart, it is straightforward that 3D CNNs should adopt residual architecture paradigms like element-wise addition [ 20 ] , [ 21 ] and attention mechanisms [ 22 ] to improve model performance. However, it is noteworthy mentioning that these skip connection operations are mostly performed using full-precision arithmetic, which results in additional hardware-related costs, especially in the context of fully-quantized models (targeting a dedicated hardware mapping). In our work, we thus introduce a 3D quantized MUX layer with an OR-gated connection that allows integrating both an element-wise additional connection and a channel-wise attention-like mechanism, while keeping a binarized data representation. II-B Efficient 3D CNN architecture. Several works have recently proposed alternative architectures to alleviate the parameter-heaviness of Conv3D. [ 23 ] , [ 24 ] partly replace Conv3D by 2D convolutions. [ 25 ] proposes different variants for the 3D residual block by separating 3\times 3\times 3 kernels with 1\times 3\times 3 spatial convolutions and 3\times 1\times 1 temporal convolutions. On the other hand, [ 26 ] processes the temporal features without parameters and multiply-accumulate (MAC) operations by shifting part of the channels along the time dimension. Finally, [ 9 ] converts various well-known resource-efficient 2D CNNs such as MobileNet [ 27 ] , ShuffleNet [ 28 ] to 3D CNNs. In our work, BILLNET uses a basic factorization closely related to the one of 3D ResNext [ 20 ] , including 2 pointwise layers and a grouped Conv3D, without nonlinearity ( i . e ., activation) inserted in-between. II-C Network quantization. Network quantization [ 12 ] reduces the bitwidth of weights and/or activations. In the most extreme case, Binarized Neural Networks (BNNs) [ 13 ] restrict both weights and activations to a 1-bit representation using Sign function, this ways reducing the costly full-precision MACs to bitwise operations ( i . e ., using XNOR gates). Despite tremendous progress during the last few years, there still exists a lack of efforts on model quantization for video inference. Recently, [ 29 ] adaptively selects the per-frame optimal bitwidth, conditioned on input data. [ 30 ] proposes a binary 3D CNN constraining weight and activation values to 0 or 1 . Besides, existing approaches [ 31 ] , [ 16 ] , [ 32 ] , [ 33 ] mostly focus on the compression of LSTM for language or speech models only. To the best of our knowledge, there is no prior works on fully-quantized LSTM in the context of video inference. Our work tries to fill in this gap by proposing a multi-stage training algorithm to provide a fully-quantized Conv3D-LSTM model. III BILLNET Figure 1 depicts the top-level view of BILLNET that involves integer-only MACs and bitwise operations such as 2-input multiplexers and OR gates. This model takes as input a sequence of 16 frames with a spatial resolution of 96\times 128 . BILLNET contains a spatio-temporal feature extractor with a Conv3D part to extract spatio-temporal features between adjacent frames, and a LSTM part to keep track longer-term temporal dependencies. In this section, we first focus on the 3D Convolution Factorization (CF), then on the custom 3D MUX-OR Residual (MOR) block, finally on the LSTM weights and