Ternarization of Vision Language Models for use on edge devices
Ben Crulis, Cyril De Runz, Barthelemy Serres and Gilles Venturini · 2025
Read Ternarization of Vision Language Models for use on edge devices by Ben Crulis, Cyril De Runz, Barthelemy Serres and Gilles Venturini (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-06298.txt.
Introduction
Ben Crulis and co-authors wrote a 2025 paper on shrinking vision language models for small devices. Such models read both images and text, but they are too large for edge hardware. Rather than train a small model from scratch, the team converts a trained model so each weight takes one of three values.
A clustering step sets the start weights for the shrunk model, which cuts conversion time. Custom math routines run the three value model on a common lite engine. The team compared the full, three value, and two value forms on memory use, speed, and text quality. The three value form gave the best trade, with low memory and the fastest text output.
Catalog record
| CREATOR | Ben Crulis, Cyril De Runz, Barthelemy Serres and Gilles Venturini |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 40 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-06298
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Ternarization of Vision Language Models for use on edge devices Ben Crulis, Cyril De Runz, Barthelemy Serres and Gilles Venturini Abstract We propose a process to compress a pre-trained Vision Language Model into a ternary version of itself instead of training a ternary model from scratch. A new initialization scheme from pre-trained weights based on the k-means algorithm is proposed to reduce the ternarization time. We implement different custom operators for executing the ternary model on the TensorFlow Lite Engine. We compare the original model with its ternary and binary versions in terms of memory consumption, inference speed and perplexity. We find that the ternary model using our custom ternary matrix multiplication operator provides a good compromise in term of memory usage and perplexity, while having the fastest token generation speed. AI Artificial Intelligence ANN Artificial Neural Network API Application Programming Interface AVX Advanced Vector Extensions BNN Binary Neural Network BP Backpropagation CNN Convolutional Neural Network CPU Central Processing Unit CV Computer Vision DFA Direct Feedback Alignment DL Deep Learning DNN Deep Neural Network DRTP Direct Random Target Propagation FPGA Field-Programmable Gate Array GAN Generative Adversarial Network GD Gradient Descent GPS Global Positioning System GPU Graphical Processing Unit HCI Human-Computer Interface HTML HyperText Markup Language JVM Java Virtual Machine LCE Larq Compute Engine LLM Large Language Model ML Machine Learning MLP Multi Layer Perceptron PTQ Post Training Quantization QAT Quantization Aware Training RAM Random Access Memory RNN Recurrent Neural Network SGD Stochastic Gradient Descent SIMD Single Instruction, Multiple Data SOTA State Of The Art SOTA State Of The Art SSE Streaming SIMD Extensions SSL Self-Supervised Learning STE Straight-Through Estimator SfM Structure from Motion TDID Target-Driven Instance Detection TNN Ternary Neural Network VLM Vision Language Model VQA Visual Question Answering ViT Vision Transformer QAT Quantization-Aware Training PTA Post-Training Quantization I Introduction VLM appear as the ideal type of DNN ( DNN ) to solve many vision tasks. Indeed a single VLM can be used for tasks such as captioning, object detection, grounding or segmentation [ 1 ] . However, as they are built on top of LLM , VLM are often very large, with a number of parameters often ranging from the billions [ 2 , 3 ] to the hundreds of billions [ 4 ] . The large size of these models makes them difficult if not impossible to use on edge devices such as smartphones. Even smaller VLM models such as the Moondream model are too large to run on smartphones despite only having about 2 billions parameters. Fortunately, several methods have been developed in the domain of model compression to reduce the size of models while preserving their task performance. For instance, model distillation consists in training a smaller model to reproduce the outputs of a larger teacher model, effectively compressing the knowledge of the teacher model [ 5 , 6 ] . On the other hand, pruning removes the less useful weights of the model by setting their value to 0 , reducing the parameter count and thus the size [ 7 ] . Finally, quantization reduces the number of bits per parameter, effectively reducing the model size [ 6 ] . The most extreme form of quantization is the binarization of the weights [ 8 ] , which makes each parameter store only a single bit of information. Unfortunately, as far as we know, no method for binarizing pre-trained models exists. This mean we have no choice but to train binary VLM from scratch, which would still be long, difficult and expensive. A similar problem arises in model distillation, which requires the training of a smaller model from scratch. We thus settle on a compromise: we propose to compress an already relatively small VLM into a ternary version of itself in a process called ternarization . In ternarization, the ternary weights of the compressed model are initialized from the original unquantized weights. The ternary weights allow the resulting model to retain slightly more information from the original model compared to binary weights, while still being extremely compressed and enabling potential for computational optimization. In ternarization, a large part of the original weights are set to 0 , which means ternarization can also be considered a form of pruning. Overall, our proposed procedure consists in ternarizing and fine-tuning the original model in the PyTorch ecosystem. To initialize the parameters of the ternary model, we propose an algorithm that uses vector quantization to reduce the initial quantization error to reduce the amount of fine-tuning required. The resulting model is converted to the Tensorflow Lite format to make it compatible with edge device execution. This model is then compared to the original model as well as other variants in term of execution speed, memory usage and task performance. These variants differ by their use of different implementations of the ternary matrix multiplication algorithm. We openly release the conversion code as well as optimized operators to unpack the ternary weights or directly compute the matrix multiplication with a packed ternary matrix. II Related Work QAT ( QAT ) and PTQ ( PTQ ) techniques are defined and surveyed in [ 9 ] . Our work inscribes itself in the QAT paradigm as we re-train the model using calibration dataset, whereas PTQ would completely avoid fine-tuning. Furthermore, 2 bits quantization should not be confused with ternarization, as 2 bits quantization make parameters use all possible 4 values that can be encoded using 2 bits. Such a quantization is also performed block-wise on the weight matrix, contrary to ternarization which transforms the entire weight matrix of a layer at once. The ternarization of DNN was explored before, but most research works train the ternary models form scratch instead of starting from pre-trained full-precision models. For instance, [ 10 ] uses a truncated gaussian approximation and [ 11 ] combine quantization and pruning techniques to train a ternary CNN ( CNN ) model (ResNet). [ 12 ] Initializes the weights of their ternary model from the weights of a pre-trained model, but still requires a long training process over tens of epochs to recover the accuracy of the original model. In contrast, we stop our ternarization process after 2 epochs of fine-tuning. The authors of [ 13 ] ternarize a diffusion model for image generation. Similar to us, they implement custom low-level kernels to accelerate computations. However, they do not describe the ternarization process or release their code, which makes it impossible to compare their approach with ours in details. Contrary to [ 14 ] which also ternarize a LLM , we ternarize an existing model from its pretrained weights using a novel initialization method instead of training the ternary LLM from scratch. III New Ternarization Process The ternarization process starts from a pre-trained DNN model. The goal of the ternarization process is to assign most of the original weights of the model into one of the values of the set \{-1,0,1\} with the smallest performance loss possible (measured in term of perplexity). III-A Initialization of the ternary weights In order to extract the largest amount of information from the original model and thus accelerate the ternary fine-tuning process, a transfer learning approach like ours should initialize the latent weights to the values