NeuCLIP: Efficient Large-Scale CLIP Training with Neural Normalizer Optimization

Xiyuan Wei Chih-Jen Lin Tianbao Yang · 2025

Read NeuCLIP: Efficient Large-Scale CLIP Training with Neural Normalizer Optimization by Xiyuan Wei Chih-Jen Lin Tianbao Yang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-08417.txt.

Introduction

Xiyuan Wei, Chih-Jen Lin, and Tianbao Yang propose NeuCLIP, a more efficient way to train CLIP models that match images with text. Contrastive training needs an accurate normalization term for each sample, and standard methods approximate it with huge batches that consume heavy resources. Earlier per sample estimators update in blocks each epoch, but their error grows when data sets are large or batches small.

NeuCLIP reformulates each sample loss through convex analysis so its log normalizer becomes an auxiliary variable. Variational analysis then replaces millions of separate variables with one compact neural network that predicts them. An alternating algorithm trains CLIP and the helper network together, with tailored architecture and acceleration for accurate estimates.

Catalog record

CREATORXiyuan Wei Chih-Jen Lin Tianbao Yang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE54 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-08417

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

NeuCLIP: Efficient Large-Scale CLIP Training with Neural Normalizer Optimization Xiyuan Wei Chih-Jen Lin Tianbao Yang Abstract Accurately estimating the normalization term (also known as the partition function) in the contrastive loss is a central challenge for training Contrastive Language-Image Pre-training (CLIP) models. Conventional methods rely on large batches for approximation, demanding substantial computational resources. To mitigate this issue, prior works introduced per-sample normalizer estimators, which are updated at each epoch in a blockwise coordinate manner to keep track of updated encoders. However, this scheme incurs optimization error that scales with the ratio of dataset size to batch size, limiting effectiveness for large datasets or small batches. To overcome this limitation, we propose NeuCLIP, a novel and elegant optimization framework based on two key ideas: (i) reformulating the contrastive loss for each sample via convex analysis into a minimization problem with an auxiliary variable representing its log-normalizer; and (ii) transforming the resulting minimization over n n auxiliary variables (where n n is the dataset size) via variational analysis into the minimization over a compact neural network that predicts the log-normalizers. We design an alternating optimization algorithm that jointly trains the CLIP model and the auxiliary network. By employing a tailored architecture and acceleration techniques for the auxiliary network, NeuCLIP achieves more accurate normalizer estimation, leading to improved performance compared with previous methods. Extensive experiments on large-scale CLIP training, spanning datasets from millions to billions of samples, demonstrate that NeuCLIP outperforms previous methods. 1 Introduction Since its introduction, Contrastive Language-Image Pretraining (CLIP) ( 34 ) has emerged as the de facto standard for vision-language representation learning. The strong capability to align images with corresponding texts has made CLIP valuable for a wide range of real-world applications, including zero-shot classification ( 31 ) , cross-modal retrieval ( 55 ) , text-to-image generation ( 35 ) , and high-quality dataset selection ( 15 ) . With the rise of large language models (LLMs), CLIP has also been widely adopted to equip LLMs with the ability to interpret visual inputs ( 2 ) . A fundamental impediment to training CLIP models is their extensive dependence on huge datasets: attaining competitive performance typically mandates access to millions or even billions of image-text pairs ( 15 ; 48 ) . A mainstream approach for training CLIP models is to optimize a bimodal contrastive loss, which contrasts each positive image-text pair against numerous negative pairs. To enable training on billions of samples, two primary strategies have emerged to approximate the prohibitive normalization term required for contrastive loss gradient calculation. The first strategy , which relies on massive GPU resources, uses an extremely large batch size to construct a contrastive loss within each batch for backpropagation. This strategy is used by many works, including OpenAI CLIP ( 34 ) and OpenCLIP ( 10 ) . The second strategy addresses the high resource demand of the first by directly optimizing a global contrastive loss, which contrasts each positive pair with all negative pairs. To tackle the computational challenge, an estimator of the normalizer for each sample’s contrastive loss is maintained and updated using a moving average formula following a rigorous framework of finite-sum coupled compositional optimization. It was first proposed by 54 for unimodal contrastive self-supervised learning, and later adopted by 51 with significant improvements for CLIP training, yielding the FastCLIP method. While being less resource demanding, this strategy suffers an inherent limitation that the optimization error scales with the ratio of dataset size to batch size, constraining its effectiveness for large datasets or small batch sizes. Recently, there emerge some new ideas for CLIP training. For example, 56 proposed SigCLIP with a sigmoid-based contrastive loss by formulating the problem as a binary classification problem, which avoids the computation of the normalization term involving numerous negative pairs. However, SigCLIP still requires a large batch size to achieve competitive performance. 43 proposed AmorLIP that leverages a lightweight network to predict the normalizer of each contrastive loss. While conceptually appealing, AmorLIP’s objective for training the lightweight network still faces the challenge of estimating the log-partition function for the gradient calculation of the lightweight network, leading to a chicken-and-egg problem. This paper aims to address the limitations of FastCLIP and AmorLIP for CLIP training through a principled approach towards optimizing the global contrastive loss with a neural normalizer. Our method is built on two key ideas: (1) using convex analysis, we reformulate the contrastive loss of each anchor data to a minimization problem with an auxiliary variable, whose optimal solution corresponds to the log-normalizer; and (2) using variational analysis, we transform the minimization over n auxiliary variables (where n is dataset size) into minimization over a compact network that directly predicts the log-normalizers, referred to as the normalizer-prediction network (NPN) . Compared with FastCLIP and AmorLIP, our method offers several notable advantages. First, the objective for learning the encoders and the NPN is unified, and its gradient avoids any nonlinear dependence on the partition function. This allows traditional stochastic gradient methods to be employed for updating both the encoders and the NPN without incurring gradient estimation bias. Second, the unrestricted optimal solution of the auxiliary variable motivates us to inject inductive bias into the design of the NPN, resulting in a simple yet effective architecture: a feedforward layer on top of the encoders followed by a log-sum-exponential pooling layer. Moreover, this seamless optimization framework enables key acceleration techniques, including alternating optimization with multiple NPN updates and periodic re-initialization of the NPN’s parameters using sampled updated embeddings, which yields significantly better normalizer approximation. The contributions of this paper are threefold: • We reformulate the contrastive loss into an equivalent form in which the normalization terms are explicitly exposed as optimization variables. This reformulation provides a principled foundation for efficient neural normalizer approximation. • We introduce a joint optimization problem that learns the encoders and a compact normalizer-prediction network (NPN) with a unified objective, which is derived from variational analysis. We also develop an efficient algorithm for alternatively optimizing the NPN and the CLIP encoders. • We validate the effectiveness of our approach through extensive experiments on large-scale datasets, showing consistent improvement over existing methods. Comprehensive ablation studies are also conducted to highlight the contribution of different components in our framework. 2 Related Works Efficient Training of CLIP Models. Numerous approaches have been proposed to enhance the efficiency of CLIP model training. Prior works include curating high-quality datasets ( 39 ; 15 ; 52 ; 49 ) , designing more efficient vision encoder architectures ( 16 ; 1 ; 7 ) , and applying image-token masking to reduce computational cost ( 27 ; 26 ) . Additional strategies leverage knowledge distillation to train compact student models ( 44 ) or employ a pretrained reference model to steer and accelerate the training of a target model, thereby improving scaling laws ( 50 ) . In contrast, our work is orthogonal to these directions: we focus on improving the optimization process itself by providing a more efficient and stable method for minimizing the contrastive loss. Optimizing the Global Contrastive Loss. The global contrastive loss was first introduced by 54 to address the large batch-size requirement of SimCLR ( 8 ) . They proposed an efficient optimization algorithm,

Read the complete text