TernaryCLIP: Efficiently Compressing Vision-Language Models with Ternary Weights and Distilled Knowledge
Shu-Hao Zhang 1, Wei-Cheng Tang 1, Chen Wu 2, Peng Hu 2, Nan Li 2, Liang-Jie Zhang 2, Qi Zhang 2, Shao-Qun Zhang 1, 🖂 · 2025
Read TernaryCLIP: Efficiently Compressing Vision-Language Models with Ternary Weights and Distilled Knowledge by Shu-Hao Zhang 1, Wei-Cheng Tang 1, Chen Wu 2, Peng Hu 2, Nan Li 2, Liang-Jie Zhang 2, Qi Zhang 2, Shao-Qun Zhang 1, 🖂 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-21879.txt.
Introduction
Shu-Hao Zhang and seven co-authors, Wei-Cheng Tang, Chen Wu, Peng Hu, Nan Li, Liang-Jie Zhang, Qi Zhang. Shao-Qun Zhang, shrink the vision-language model CLIP to a fraction of its normal cost. Their framework, TernaryCLIP, converts nearly all weights in both the image and text encoders into a three-value format, so each weight needs barely more than a single bit.
Training that respects the quantization, plus guidance distilled from the full model, keeps the squeezed version accurate. The team reports compression of about seventeen times, storage cut sixteen-fold, and faster inference, while scores on image classification and image-text search across forty-one datasets stayed promising. The result shows that even very large multimodal models can survive extreme compression and run cheaply.
Catalog record
| CREATOR | Shu-Hao Zhang 1, Wei-Cheng Tang 1, Chen Wu 2, Peng Hu 2, Nan Li 2, Liang-Jie Zhang 2, Qi Zhang 2, Shao-Qun Zhang 1, 🖂 |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 53 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-21879
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
TernaryCLIP: Efficiently Compressing Vision-Language Models with Ternary Weights and Distilled Knowledge Shu-Hao Zhang 1, Wei-Cheng Tang 1, Chen Wu 2, Peng Hu 2, Nan Li 2, Liang-Jie Zhang 2, Qi Zhang 2, Shao-Qun Zhang 1, 🖂 Abstract Recent years have witnessed an increasing interest in image-text contrastive modeling, exemplified by models such as Contrastive Language-Image Pretraining (CLIP). In this paper, we propose the TernaryCLIP, a lightweight computational framework that converts connection weights of both vision and text encoders of CLIP into the ternary format, instead of full-precision or floating ones. TernaryCLIP incorporates quantization-aware training and distillation modules, preventing precision degradation and enabling low-cost and high-efficiency computations. Comprehensive experiments demonstrate that TernaryCLIP can achieve up to 99% ternarized weights with 1.58-bit representation, 16.98 × \times compression ratio, 2.3 × \times inference acceleration, 16 × \times storage reduction, 10 × \times memory optimization, and 60% sparsity while maintaining promising performance on zero-shot image classification and image-text retrieval tasks across 41 commonly used datasets. Our work highlights the feasibility of extreme quantization for large multimodal models, supporting effective and efficient deployment on resource-constrained devices. The model and code can be accessed from Hugging Face and GitHub . Key words: Vision-Language Models, CLIP, Ternary Quantization, Knowledge Distillation Figure 1 : The workflow of the TernaryCLIP framework, comprising two modules: ternarization and distillation. 1 Introduction Large-scale multimodal models, such as Contrastive Language-Image Pretraining (CLIP) radford2021learning that aligns images and texts in a shared embedding space through contrastive learning, have demonstrated exceptional performance across vision-language tasks, including image classification and image-text retrieval. However, the impressive capabilities of these models come at the cost of significant resource demands brown2020language ; strubell2019energy , contributing to three critical challenges for practical deployment. First, the tremendous parameters result in substantial storage and memory consumption han2016deepcompression . Second, the full-precision model introduces heavy computational burdens affecting inference latency sze2017efficientprocessingdeepneural . Third, the deployment of downstream tasks typically requires massive, domain-specific, labeled datasets devlin2019bert , such as LAION-400M schuhmann2021laion400m , which incur huge annotation costs and limit practical applicability changpinyo2021cc12m . These challenges hinder the widespread deployment of large-scale multimodal models in resource-constrained environments such as mobile devices howard2017mobilenets . Consequently, model lightweight techniques have emerged as a prominent research direction to bridge the gap between promising performance and feasible deployment zhu2024surveymodelcompressionlarge . Quantization has emerged as a promising lightweight technique that converts full-precision weights into lower-bit formats banner2019post ; jacob2018quantization ; wu2016quantized . Specifically, ternary quantization compresses floating-point or 32-bit weights into three discrete values \{-\Delta,0,+\Delta\} , reducing the storage and memory consumption via an ultra-low 1.58-bit format with each weight requiring \log_{2}(3)\approx 1.585 bits in the optimal case. This quantization enables the replacement of complex floating-point matrix multiplications with simple addition and shift operations, thereby improving inference efficiency and reducing latency alemdar2017ternaryneuralnetworks ; li2022ternaryweightnetworks ; zhu2017trained . However, existing quantization approaches suffer from two critical limitations for multimodal efficient deployment. On the one hand, current research primarily focuses on unimodal architectures such as BERT for natural language processing zafrir2019q8bert ; fan2020ternarybert and ViT for computer vision li2022qvit ; liu2021post , leaving multimodal models like CLIP largely unexplored. On the other hand, quantization usually causes severe performance degradation that compromises practical applicability banner2019post ; rastegari2016xnor . Knowledge distillation has emerged as another lightweight technique that employs the teacher-student framework without massive labeled data gou2021knowledge ; hinton2015distilling . Specifically, distillation resolves the annotation requirement challenge by utilizing soft labels from teacher models to guide student learning, transferring rich representational knowledge without relying on domain-specific labeled datasets romero2015fitnets ; zagoruyko2016paying . This preserves the zero-shot capabilities of pre-trained models radford2021learning , thereby avoiding dependence on extensive labeled datasets and significantly reducing annotation costs. Moreover, distillation addresses training resource consumption by leveraging pre-trained teacher knowledge to substantially reduce student model training time, avoiding the enormous computational overhead of training from scratch cho2019efficacy ; mirzadeh2020improved . Despite the promising progress, comprehensive distillation strategies for ultra-low-bit multimodal scenarios, such as ternary quantization, remain unexplored mishra2017apprentice ; polino2018model . In this paper, we propose the TernaryCLIP by integrating ternary quantization and distillation into large-scale vision-language models. The workflow of our TernaryCLIP is illustrated in Figure 1 . The experimental results indicate substantial improvements in reducing the resource demands. For computational efficiency, TernaryCLIP achieves 1.58-bit weight representation with 99% quantization proportion and 16.98× compression ratio, resulting in 2.3× inference speedup, 16× storage reduction, 10× memory optimization, and 60% sparsity. For annotation dependency elimination, TernaryCLIP maintains competitive zero-shot performance across 41 datasets, preserving the generalization capabilities of full-precision models without requiring extensive downstream task-specific annotations. These results demonstrate the effectiveness of our proposed TernaryCLIP, maintaining competitive performance with substantially reduced resource requirements. To the best of our knowledge, the proposed TernaryCLIP is the first advance that achieves extremely-low compression of weight precision of large-scale vision-language models. Empirical results support the practicability that ultra-low-bit multimodal models are not only feasible but also capable of bridging the gap between performance and practical deployment in resource-constrained environments such as edge computing platforms. 2 Preliminaries 2.1 Introduction to CLIP The CLIP architecture ( radford2021learning ) comprises an image encoder f_{i} (e.g., ViT) and a text encoder f_{t} (e.g., BERT), which are trained to maximize cosine similarities between matched image-text pairs while minimizing similarities for unmatched pairs. Given a training dataset D={(I_{i},T_{i})}_{i=1}^{|D|} , CLIP learns the cross-modal representations through an InfoNCE-based contrastive loss ( oord2019rlcpc ) . TernaryCLIP adopts this contrastive learning framework with the loss function formulated as \mathcal{L}_{\text{task}}=(\mathcal{L}_{\text{I}\rightarrow\text{T}}+\mathcal{L}_{\text{T}\rightarrow\text{I}})/2 , where \mathcal{L}_{\text{I}\rightarrow\text{T}}=\text{CrossEntropy}(\text{logits},\text{labels}) and \mathcal{L}_{\text{T}\rightarrow\text{I}}=\text{CrossEntropy}(\text{logits}^{T},\text{labels}) . The logits are computed as \text{logits}=I_{e}T_{e}^{\top}/\tau=f_{i}(I)f_{t}(T)^{\top}/\tau , representing the scaled cosine similarities between image and text embeddings with a temperature hyperparameter \tau . Inspired by previous distillation methods for CLIP, such as CLIP-KD ( yang2024clip ) and ComKD-CLIP ( chen2024comkdclip ) , TernaryCLIP integrates three complementary distillation strategies: Contrastive Relational Distillation (CRD), Interactive Contrastive Learning (ICL), and Feature Distillation (FD). CRD transfers contrastive knowledge from teacher to student by aligning their cross-modal similarity distributions through KL divergence: \mathcal{L}_{\text{crd}}=\text{KL}(p_{t}|p_{s})+\text{KL}(q_{t}|q_{s}) , where p_{s},p_{t} and q_{s},q_{t} denote the image-to-text and text-to-image distributions for the student and teacher models, respectively. ICL facilitates cross-modal knowledge transfer by establishing contrastive relationships between student and teacher embeddings across modalities: \mathcal{L}_{\text{icl}}=(\mathcal{L}_{I_{s}\rightarrow T_{t}}+\mathcal{L}_{T_{s}\rightarrow I_{t}})/2 , where \text{logits}_{I_{s}\rightarrow T_{t}}=I_{e,s}T_{e,t}^{\top}/\tau and \text{logits}_{T_{s}\rightarrow I_{t}}=T_{e,s}I_{e,t}^{\top}/\tau represent the scaled similarities between student image embeddings and teacher text embeddings, and between student text embeddings and teacher image embeddings, respectively. FD aligns the embedding spaces by minimizing the mean squared error between student and teacher representations: \mathcal{L}_{\text{fd}}=\text{MSE}(I_{e,s},I_{e,t})+\text{MSE}(T_{e,s},T_{e,t}) , where \text{MSE}(I_{e,s},I_{e,t})=\sum_{i=1}^{n}(I_{e,s}^{(i)}-I_{e,t}^{(i)})^{2}/n measures the distance between student and teacher image embeddings, and similarly for text embeddings. 2.2 Related Studies Model Quantization. Quantization has emerged as a fundamental strategy for compressing neural networks to enable efficient inference. Traditional approaches, such as integer quantization, reduce memory usage and computational costs while maintaining acceptable accuracy jacob2018quantization ; wu2016quantized . Recent advances have extended to ternary quantization, which represents weights using only three values \{-1,0,+1\} , achieving superior efficiency while preserving model performance zafrir2019q8bert ; zhu2017trained . Notable quantization-aware training (QAT) techniques, including TWN li2022ternaryweightnetworks , TTQ zhu2017trained , and FATNN chen2021fatnn incorporate scaling factors to enhance ternary model optimization. The Transformer architecture, which has become essential for modern NLP tasks, has also been successfully quantized through methods such as