Variance-Aware Loss Scheduling for Multimodal Alignment in Low-Data Settings

Sneh Pillai · 2025

Read Variance-Aware Loss Scheduling for Multimodal Alignment in Low-Data Settings by Sneh Pillai (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-03202.txt.

Introduction

Sneh Pillai, a solo author, schedules training losses by uncertainty in a 2025 paper. Vision-language models usually train on huge image-caption sets, but small-data settings invite overfitting and instability. His method watches how variable the model's alignment guesses are, then reweights the contrastive loss accordingly during training. Tests used a slice of a standard caption dataset to fake scarcity.

The schedule beat a fixed-weight baseline on retrieval accuracy, and edged out rival adaptive schemes based on entropy or similarity spread. Under noise attacks on images and captions, the variance-guided version held its recall best. Visual maps showed cleaner, more distinct embeddings.

Catalog record

CREATORSneh Pillai
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-03202

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Variance-Aware Loss Scheduling for Multimodal Alignment in Low-Data Settings Sneh Pillai Abstract Training vision-language models for image-text alignment typically requires large datasets to achieve robust performance. In low-data scenarios, standard contrastive learning can struggle to align modalities effectively due to overfitting and unstable training dynamics. In this paper, we propose a variance-aware loss scheduling approach that dynamically adjusts the weighting of the contrastive loss based on the statistical variability (uncertainty) in the model’s alignment predictions. Using a subset of the Flickr8k image-caption dataset to simulate limited data conditions, we demonstrate that our approach improves image-text retrieval accuracy compared to a fixed-weight baseline. We also compare against other adaptive weighting strategies (using output entropy and cosine similarity spread) and find that variance-aware scheduling provides the best overall trade-off. Qualitatively, our method yields more distinct multimodal embeddings as shown by t-SNE visualizations. Moreover, in a stress test with noise-injected captions and images, the variance-guided loss proves more robust, maintaining higher recall when random perturbations are introduced. These results highlight the benefit of adaptive loss weighting for multimodal alignment in low-data regimes. K eywords Multimodal alignment \cdot Image-text retrieval \cdot Contrastive learning \cdot Adaptive loss weighting \cdot Low-data training 1 Introduction Learning to align images and textual descriptions is a central problem in multimodal machine learning, with applications in image-caption retrieval, description generation, and visual question answering. A core challenge in image-text alignment is bridging the semantic gap between visual and linguistic representations. Recently, large-scale contrastive learning approaches have achieved remarkable success by leveraging massive datasets of image-text pairs [ 1 , 2 ] . Notably, the CLIP model by Radford et al. (2021) [ 1 ] was trained on 400 million image-text examples, enabling robust alignment and zero-shot transfer. However, such data-hungry methods are impractical in domains where only limited annotated pairs are available. In low-data settings (e.g., only a few thousand pairs), models tend to overfit and often fail to learn a generalizable joint embedding space. Traditional image-caption embedding models were developed on smaller datasets like Flickr8k and Flickr30k [ 3 ] . For instance, early approaches projected images and sentences into a common space using neural networks [ 4 ] . Kiros et al. (2014) embedded images and text with a multimodal neural language model [ 4 ] , and Karpathy & Li (2015) employed a recurrent neural network to align image regions with words, optimizing a structured max-margin objective [ 5 ] . Subsequent improvements incorporated harder negative mining [ 6 ] and attention mechanisms. Faghri et al. (2018) introduced VSE++ which focuses training on hard negatives to significantly improve retrieval performance [ 6 ] . Lee et al. (2018) proposed a stacked cross-attention network to model fine-grained alignments between image regions and words, achieving state-of-the-art results on benchmarks [ 7 ] . Beyond architectural advances, another line of related work explores adaptive loss functions and training schedules. In multi-task learning, Kendall et al. (2018) used task uncertainty (modeled as output variance) to weight loss terms, automatically balancing objectives during training [ 8 ] [ 8 ] . For classification, the focal loss (Lin et al. 2017) dynamically down-weights easy examples, effectively focusing on hard instances to address class imbalance [ 9 ] . Curriculum learning strategies introduce tasks or samples in increasing order of difficulty [ 10 ] , showing that a guided scheduling of training can yield better generalization [ 10 ] . These approaches inspire our work: we hypothesize that adaptively scheduling the contribution of the contrastive loss—based on the model’s current uncertainty or alignment quality—can lead to more effective multimodal training, especially when data are scarce. In this paper, we introduce variance-aware loss scheduling for contrastive image-text alignment. Our method monitors the variance in the model’s similarity scores during training and adjusts the loss weight accordingly. Intuitively, when the model is uncertain (e.g., outputs show small variance or high entropy, indicating it has not yet distinguished matches from non-matches), the training signal is amplified; when the model grows confident (large variance in scores between matches and non-matches), the loss weight is tempered to avoid over-emphasizing already learned distinctions. We implement this idea in a simple yet effective way and conduct experiments on a low-data scenario using Flickr8k. The contributions of our work are as follows: (1) We propose a novel dynamic loss weighting scheme for multimodal contrastive learning that is responsive to the model’s own output variability. (2) We provide a thorough comparison of variance-based scheduling with alternative adaptive strategies (entropy-based and cosine spread-based weighting) and a standard fixed-loss baseline. (3) We demonstrate that our approach yields improved retrieval performance and more clearly separated embeddings, and we show it offers increased robustness to noisy input data. To our knowledge, this is one of the first studies to explore adaptive loss scheduling for image-text alignment in limited data regimes. 2 Related Work Image-Text Alignment and Retrieval. A rich body of work addresses the problem of embedding images and text into a joint space for retrieval tasks. Early methods such as DeViSE (Frome et al. 2013) projected image features and word embeddings into a common semantic space using a deep neural network, enabling zero-shot image classification by semantic matching. Kiros et al. (2014) expanded on this by training a multimodal neural language model that unifies visual and textual representations [ 4 ] . On the Flickr8k and 30k datasets, Karpathy & Li (2015) achieved notable results by learning region-phrase alignments with a structured objective [ 5 ] , demonstrating the importance of fine-grained correspondence for caption retrieval. Subsequent models improved global matching through better loss functions and attention. Faghri et al. (2018) (VSE++) showed that mining hard negatives in a hinge-based loss substantially boosts Recall@K, as the model learns from challenging examples [ 6 ] . Similarly, Lee et al. (2018) introduced an attention-based mechanism (SCAN) to infer latent alignments between image regions and words, yielding state-of-the-art retrieval accuracy on Flickr30k and MS-COCO [ 7 ] . These advances, however, typically assume moderate-sized datasets (tens of thousands of images). In contrast, our work focuses on scenarios with an order of magnitude less data, where even these methods may falter without additional training heuristics. More recently, large-scale approaches have changed the landscape of multimodal learning. Radford et al. (2021) demonstrated that contrastive learning on an extremely large corpus (400M image-text pairs) can produce general-purpose representations (the CLIP model) [ 1 ] . Jia et al. (2021) similarly introduced ALIGN, trained on 1B noisy image-text pairs. These models excel in zero-shot settings but rely on vast data and compute resources. Our approach, while inspired by contrastive objectives, targets the opposite end of the spectrum: improving training efficacy when data is limited. Rather than scaling data, we scale the information extracted from each available sample via adaptive loss weighting. Adaptive Loss Weighting. Adjusting loss contributions dynamically has shown promise in various contexts. In multi-task learning, balancing different task losses is crucial; methods like uncertainty weighting (Kendall et al. 2018) treat loss weights as learnable parameters derived from task output variances [ 8 ] . This causes tasks with higher uncertainty (and typically higher loss variance) to be weighted lower, preventing them from destabilizing training. Another approach, GradNorm (Chen et al.

Read the complete text