Trainable Bitwise Soft Quantization for Input Feature Compression

Karsten Schrödter, Jan Stenkamp, Nina Herrmann, Fabian Gieseke · 2026

Read Trainable Bitwise Soft Quantization for Input Feature Compression by Karsten Schrödter, Jan Stenkamp, Nina Herrmann, Fabian Gieseke (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-05172.txt.

Introduction

Karsten Schrödter and colleagues propose a trainable quantization layer that compresses neural-network input features on the device before transmission to remote servers. Each feature can be quantized to a user-chosen bit count, easing bandwidth, latency, and energy limits in Internet of Things settings. The layer approximates step functions with sigmoids, giving trainable thresholds, and concatenates sigmoid outputs into bitwise soft quantization. Tests beat standard quantization methods while staying near full-precision accuracy, reaching fivefold to sixteenfold compression against 32-bit inputs without major performance loss.

Catalog record

CREATORKarsten Schrödter, Jan Stenkamp, Nina Herrmann, Fabian Gieseke
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE55389
LICENSEhttp://creativecommons.org/licenses/by-sa/4.0/
RIGHTSCC BY SA
SOURCEarXiv

Source: https://arxiv.org/abs/2603.05172

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Trainable Bitwise Soft Quantization for Input Feature Compression Karsten Schrödter, Jan Stenkamp, Nina Herrmann, Fabian Gieseke Abstract The growing demand for machine learning applications in the context of the Internet of Things calls for new approaches to optimize the use of limited compute and memory resources. Despite significant progress that has been made w.r.t. reducing model sizes and improving efficiency, many applications still require remote servers to provide the required resources. However, such approaches rely on transmitting data from edge devices to remote servers, which may not always be feasible due to bandwidth, latency, or energy constraints. We propose a task-specific, trainable feature quantization layer that compresses the input features of a neural network. This can significantly reduce the amount of data that needs to be transferred from the device to a remote server. In particular, the layer allows each input feature to be quantized to a user-defined number of bits, enabling a simple on-device compression at the time of data collection. The layer is designed to approximate step functions with sigmoids, enabling trainable quantization thresholds. By concatenating outputs from multiple sigmoids, introduced as bitwise soft quantization, it achieves trainable quantized values when integrated with a neural network. We compare our method to full-precision inference as well as to several quantization baselines. Experiments show that our approach outperforms standard quantization methods, while maintaining accuracy levels close to those of full-precision models. In particular, depending on the dataset, compression factors of 5 × 5\times to 16 × 16\times can be achieved compared to 32 32 -bit input without significant performance loss. DNN Deep Neural Network IoT Internet of Things AI Artificial Intelligence 1 Introduction The number of IoT ( IoT ) applications has steadily grown in recent years, with use cases spanning a wide range of domains [ 34 , 38 , 36 , 17 ] . These applications are often based on microcontrollers that are equipped with sensors that measure parameters such as temperature, humidity, pressure, or vibrations. Typically, such microcontrollers have very limited compute and memory capabilities (e.g., only 2 KB of RAM), which severely restricts the types of algorithms that can be executed “locally” on the devices. This necessitates the development of efficient, resource-aware implementations tailored to the specific needs of the given application and hardware. Several approaches have been proposed for deploying machine learning models in resource-constrained settings. One common strategy is to reduce the model size to an extent that inference can be performed on the device. Nevertheless, this approach requires trade-offs between model complexity and the available resources [ 40 , 33 ] . Alternatively, the collected sensor data can also be transmitted to more powerful remote machines for further processing. However, this strategy also poses significant challenges. Reliable communication links and sufficient energy supplies might not be available. In extreme cases, devices are battery-powered and can only transmit very limited amounts of data—sometimes just a few bytes per hour. This is the case, for instance, with communication protocols such as LoRaWAN. Transmitting data via mobile networks (e.g., LTE) is in contrast often energy-intensive and might also not be possible due to a lack of cellular coverage [ 22 ] . We focus on scenarios where (a) on-device execution of a machine learning model is not feasible, and (b) only very limited amounts of data can be transmitted to a remote server. Such situations commonly arise when IoT devices are deployed in remote locations, e.g., for environmental monitoring (forestry, wildlife, agriculture, …). Hence, the devices mainly collect data, and strategies are required to minimize the size of data transmitted. Straightforward approaches are based on selecting important features (e.g., via forward feature selection [ 19 ] ). Another strategy involves compressing the collected (sensor) data by using lower precision for floating-point numbers. However, naive precision reduction is task-agnostic and often degrades downstream model performance. We propose an end-to-end trainable feature quantization layer that can be integrated into a (deep) neural network to learn task-specific compressions for each input feature. The resulting encoding scheme is based on simple intervals and can be efficiently implemented using standard if-then-else rules. As shown in Figure 1 , the quantization layer is jointly trained with a given neural network on a remote server to determine optimal thresholds and quantized values for efficient deployment. During inference, quantization is implemented on a resource-constrained device using lightweight logic, while decoding is performed on a more capable server, where the remaining neural network processes the compressed input. By integrating quantization into training, the method adapts to the underlying data distribution, enabling flexible and effective splitting of input features. (a) Training (b) Inference Figure 1: The trainable feature quantization layer is integrated into a neural network to learn task-specific compressions for each input feature: (a) During training, the quantization layer (green rectangle) and the neural network are trained jointly on a remote server. The layer gives rise to an encoder-decoder composition \quant_{i}=\dec_{i}\circ\enc_{i} for the i -th input feature. (b) During inference, the encoder \enc_{i} is used to encode the i -th feature on the resource-constrained device using lightweight coding logic. The compressed features are then sent to the server, where the decoder \dec_{i} is used to decode the i -th feature. All decoded features are then used as input for the remaining neural network, which is executed on the remote server. 2 Background We start by summarizing related work and our contributions. 2.1 Related Work Several approaches have been proposed to execute machine learning methods on IoT nodes. The main challenge in this context is to reduce both computational and memory demands to enable execution on highly resource-constrained hardware—without significant loss in model accuracy. This has led to the tiny machine learning paradigm [ 16 , 33 , 23 , 40 ] . When sufficiently optimized, such “tiny” models can run directly on microcontrollers, enabling compact and intelligent devices. However, reducing the model size generally leads to inferior results, especially when the IoT devices are extremely resource-constrained. As a result, offloading the collected data to more powerful remote machines still remains a common approach for many applications. Data transfer during the inference phase is also an active area of research. Broadly, existing approaches fall into two categories: (a) feature extraction, which transforms the original features into a new representation, and (b) feature selection, which identifies a relevant subset of the original features (e.g., [ 3 , 30 , 31 , 37 , 41 , 25 ] ). The research area of collaborative intelligence [ 2 ] explores the possibility of distributing inference tasks across multiple devices, with each device being responsible for computing a portion of the entire model. 21 first proposed a collaborative intelligence approach, called Neurosurgeon , which automatically determines split points for deep neural networks in a layer-wise manner. Works based on this approach also consider reducing the size of the data to be transmitted by focusing on image compression [ 27 , 28 , 35 , 43 ] . Another topic of interest is to actually find suitable split points for neural networks [ 4 ] and to identify so-called early-exit points [ 24 ] . Quantization is used to reduce the bit width of activations and weights of a neural network,

Read the complete text