Zero-Shot, But at What Cost? Unveiling the Hidden Overhead of MILS’s LLM-CLIP Framework for Image Captioning
Yassir Benhammou NstarX [email protected] Alessandro Tiberio NstarX [email protected] · 2025
Read Zero-Shot, But at What Cost? Unveiling the Hidden Overhead of MILS’s LLM-CLIP Framework for Image Captioning by Yassir Benhammou NstarX [email protected] Alessandro Tiberio NstarX [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-15199.txt.
Introduction
Benhammou and co-authors audit a much-praised method for describing images without any training. The method, called MILS, pairs a large language model with an image model and refines its captions step by step. Its makers claim strong zero-shot results. The audit asks a blunt question: what does that looping cost?
Measured closely, the repeated refinement burns far more computing power than rivals that answer in a single pass. Those rivals, such as BLIP-2 and GPT-4V, reach similar caption quality without the loop. The authors argue the hidden cost undercuts the method's practical appeal. Their study is the first to price this quality-versus-cost trade-off, and it warns designers of image tools to count compute, not just accuracy.
Catalog record
| CREATOR | Yassir Benhammou NstarX [email protected] Alessandro Tiberio NstarX [email protected] |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 27 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-15199
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Zero-Shot, But at What Cost? Unveiling the Hidden Overhead of MILS’s LLM-CLIP Framework for Image Captioning Yassir Benhammou NstarX [email protected] Alessandro Tiberio NstarX [email protected] Abstract MILS (Multimodal Iterative LLM Solver) is a recently published framework that claims “LLMs can see and hear without any training” by leveraging an iterative, LLM-CLIP based approach for zero-shot image captioning. While this MILS approach demonstrates good performance, our investigation reveals that this success comes at a hidden, substantial computational cost due to its expensive multi-step refinement process. In contrast, alternative models such as BLIP - 2 and GPT - 4V achieve competitive results through a streamlined, single-pass approach. We hypothesize that the significant overhead inherent in MILS’s iterative process may undermine its practical benefits, thereby challenging the narrative that zero-shot performance can be attained without incurring heavy resource demands. This work is the first to expose and quantify the trade-offs between output quality and computational cost in MILS, providing critical insights for the design of more efficient multimodal models. 1 Introduction Recent advances in large-scale pre-training have fundamentally transformed the landscape of multimodal learning. Models such as CLIP [ 1 ] and the Vision Transformer (ViT) [ 2 ] have demonstrated that robust visual representations can be learned from vast amounts of image–text pairs. These breakthroughs, along with pioneering work on large-scale language models [ 3 , 4 ] , have spurred the development of systems that bridge the gap between vision and language, thereby enabling tasks like image captioning [ 5 ] , video description, and audio captioning in zero-shot or minimally supervised settings. A particularly intriguing line of research is the exploration of zero-shot methods that do not require task-specific fine-tuning. MILS (Multimodal Iterative LLM Solver) [ 6 ] epitomizes this approach by leveraging the innate reasoning capabilities of large language models (LLMs) together with off-the-shelf multimodal models. MILS employs an iterative, test-time optimization process in which a pre-trained LLM, acting as a Generator , produces an extremely diverse set of candidate outputs—on the order of tens of thousands per image—via varied prompts derived from class labels and other textual templates. These candidates are then evaluated by a Scorer that uses a CLIP-like model to compute cosine similarity scores between the image’s visual features and the candidate captions’ textual representations [ 1 ] . The top candidates are fed back into the LLM to refine the caption iteratively until convergence or until a preset number of iterations is reached. Although MILS achieves state-of-the-art performance in zero-shot settings for image captioning, its approach raises critical efficiency concerns. The claim that “LLMs can see and hear without any training” is compelling at first glance; however, a deeper examination reveals that MILS relies on a highly resource-intensive iterative process. Generating an initial candidate set of approximately 30K captions per image and refining them over multiple iterations incurs substantial computational overhead, which may negate the practical benefits of its training-free paradigm when compared to more efficient single-pass methods. In contrast, recent models such as BLIP-2 [ 7 ] and emerging evaluations of GPT-4V(e.g., [ 8 ] ) have demonstrated competitive zero-shot performance using a single forward pass. For example, BLIP-2 combines a frozen Vision Transformer (typically ViT-L/14 [ 2 ] ), a lightweight Query Transformer, and a pre-trained language model (e.g., FLAN-T5-XL [ 3 ] ) to directly generate captions, significantly reducing inference time and resource usage. Moreover, chain-of-thought prompting techniques [ 9 , 10 ] have further illustrated that large language models can perform complex reasoning without heavy fine-tuning, underscoring the potential for efficient, zero-shot approaches. This dichotomy prompts a fundamental question: Can the substantial computational overhead inherent in MILS’s iterative process be justified by its performance gains, or do more efficient single-pass models provide a better trade-off between caption quality and resource utilization? In this work, we critically evaluate these trade-offs through extensive experiments on the MSCOCO dataset, measuring caption quality via BLEU, ROUGE_L, CIDEr, and METEOR, while also profiling computational resources such as inference time and GPU utilization. The remainder of this paper is structured as follows: Section 2 reviews related work and situates our study in the context of recent advances in multimodal learning and zero-shot techniques; Section 3 details the technical architectures of MILS, BLIP-2, and GPT-4V; Section 4 describes our experimental setup and evaluation metrics; Section 5 presents and discusses our results; and finally, Section 6 concludes with insights and potential directions for future work. 2 Related Work The evolution of image captioning and multimodal learning can be traced through a series of breakthrough models and techniques over the past decade. Early works such as Show and Tell [ 11 ] and Show, Attend and Tell [ 12 ] , both published in 2015, pioneered the integration of convolutional neural networks and recurrent architectures to generate natural language descriptions for images. These foundational methods introduced the notion of learning visual-semantic alignments, paving the way for subsequent advancements. By 2017, researchers introduced more sophisticated training strategies, exemplified by Self-Critical Sequence Training [ 13 ] , which directly optimized captioning metrics through reinforcement learning, and by 2018, Bottom-Up and Top-Down Attention [ 14 ] further refined caption generation by leveraging object-level features. These innovations greatly improved the fluency and relevance of generated captions. The advent of transformer architectures marked a significant turning point in the field. In 2020, Dosovitskiy et al. [ 2 ] introduced the Vision Transformer (ViT), which demonstrated that self-attention can capture high-level semantic information in images without relying on convolutional operations. Shortly thereafter, OpenAI’s CLIP [ 1 ] leveraged massive-scale image–text pairs to learn joint embeddings, making it possible for models to bridge visual and linguistic modalities in a robust manner. Building on these transformer-based advances, models such as UNITER [ 15 ] and Oscar [ 16 ] (both 2020) further unified visual and textual representations. ALIGN [ 17 ] and ViLT [ 18 ] , published in 2021, continued this trend by simplifying architectures while scaling up the training process. These methods enabled significant improvements across a variety of vision and language tasks, from image captioning to visual question answering. Other approaches, such as ZeroCap [ 19 ] (2022) and MeaCap [ 20 ] (2024), have explored gradient-based optimization or memory-augmented techniques to generate captions without extensive task-specific training. However, these methods often introduce additional complexities or computational costs. A recent and particularly intriguing development is the shift towards zero-shot methods that eliminate the need for task-specific fine-tuning. MILS (Multimodal Iterative LLM Solver) [ 6 ] , released in 2025, leverages the inherent reasoning capabilities of large language models by employing an iterative, test-time optimization process. In MILS, a pre-trained LLM (the Generator ) produces an extensive candidate set—approximately 30K candidate captions per image—through diverse prompts. These candidates are then evaluated by a CLIP-like model (the Scorer ) that computes cosine similarity between the visual features extracted by a frozen encoder (typically ViT-L/14) and the text embeddings. The top candidates are fed back into the LLM for iterative refinement, ultimately converging to a final caption. Although MILS achieves state-of-the-art performance in a zero-shot setting, its iterative process imposes significant computational overhead. In contrast, other recent models such as BLIP-2 [ 7 ] (2023) and evaluations of GPT-4V