FREE : Fast and Robust Vision Language Models with Early Exits

Divya Jyoti Bajpai, Manjesh Kumar Hanawal · 2025

Read FREE : Fast and Robust Vision Language Models with Early Exits by Divya Jyoti Bajpai, Manjesh Kumar Hanawal (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-06884.txt.

Introduction

Divya Bajpai and Manjesh Hanawal present FREE in 2025, a way to make vision-language models faster. These models join images and text with strong results, but their size slows answers in real use. Letting a model exit early can speed it, yet training the exits is hard with little labeled data.

FREE trains each exit in a contest setup, so early layers learn to mimic the final layer while a judge tries to tell them apart. The model then adapts its depth to each input. Tests show faster answers with small loss, plus better handling of overthinking. The method suits apps where delay matters.

Catalog record

CREATORDivya Jyoti Bajpai, Manjesh Kumar Hanawal
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE47 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-06884

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

FREE : Fast and Robust Vision Language Models with Early Exits Divya Jyoti Bajpai, Manjesh Kumar Hanawal Abstract In recent years, Vision-Language Models (VLMs) have shown remarkable performance improvements in Vision-Language tasks. However, their large size poses challenges for real-world applications where inference latency is a concern. To tackle this issue, we propose employing Early Exit (EE) strategies in VLMs. However, training exit classifiers in VLMs is challenging, particularly with limited labeled training data. To address this, we introduce FREE , an adversarial training approach within a GAN-based framework. Here, each exit consists of a transformer layer and a classifier. The transformer layer is adversarially trained to produce feature representations similar to the final layer, while a feature classifier serves as the discriminator. Our method focuses on performing input-adaptive inference that increases inference speed with minimal drop in performance. Experimental results demonstrate the effectiveness of our approach in enhancing accuracy and model robustness by mitigating overthinking and the phenomenon of mid-crisis that we highlight. We experimentally validate that our method speeds up the inference process by more than 1.51 × 1.51\times while retaining comparable performance. The source code is available at . Figure 1: This figure provides some example outputs of FREE ViT-g OPT 2.7B . The different colours show the difficulty levels of tokens in captions. Red: Easy to predict and predicted at initial (1-12) layers. Green: Mediocre hard, predicted at intermediary (13-24) layers. Black: Hard to predict, predicted at deeper (25-32) layers. 1 Introduction Vision-language pre-training (VLP) has evolved significantly with the emergence of sophisticated pre-trained Vision Language Models (VLMs). These models have consistently pushed the performance boundaries across various vision-language tasks. However, their demanding computational requirements and inference latency pose challenges for real-time applications. Several models, such as BLIP-2 Li et al. 2023 , MiniGPT Zhu et al. 2023 etc., leverage off-the-shelf large-scale pre-trained models as building components with their parameters frozen. This reduces VLMs training parameters but makes them extremely slow during inference, as highlighted in Bajpai and Hanawal 2024b leading to higher inference time. The use of the Language Model (LM) component with frozen parameters not only makes VLM susceptible to overthinking but also to another phenomenon that we term mid-crisis Elhoushi et al. 2024 . This phenomenon occurs when intermediate layers suffer performance drops due to the search for irrelevant features. While initial layers capture shallow representations and syntactic information, and deep layers learn semantic-fusion relations Fei et al. 2022 , intermediary layers tend to capture dataset patterns that degrade their performance, even losing the information learned by initial layers, and the model has to regain the lost information again in deeper layers. We illustrate this phenomenon in the left figure of Fig. 3 and Sec. 3.1 , showcasing accuracy on the VQAv2 (visual question answering) Das et al. 2017 dataset across different exits when trained with dedicated classifiers as in vanilla early exit setup for the intermediate layers. This raises the question: how can we mitigate mid-crisis and overthinking to enhance the accuracy and efficiency of VLMs? We address this using Early Exit (EE) techniques Xin et al. 2020 ; Zhou et al. 2020 ; Zhu 2021 , an input-adaptive method that reduces computational costs by bypassing certain layers while preserving the knowledge encoded in large-scale models. Since real-world datasets contain both “easy” and “hard” samples, EE ensures adaptive computation per sample. However, applying EE to VLMs presents challenges: (1) Exit classifiers introduce significant overhead—e.g., a single exit for OPT 2.7B Zhang et al. 2022a decoder adds approximately 130M parameters; (2) Training these exits requires substantial labeled data, limiting EE adoption in zero-shot VLMs that require minimal fine-tuning. We propose FREE F ast and R obust Vision-Language Models with E arly E xits. This efficient EE training framework minimizes training costs while maintaining accuracy. Our method employs a Generative Adversarial Network (GAN)-based Creswell et al. 2018 framework, leveraging the pre-trained VLM outputs to align feature representations between intermediate exits and the final layer. Unlike cosine similarity, which hampers generalization, the adversarial setup improves feature consistency, ensuring robust exit predictions. Our method attaches exits and Feature Classifiers (FCs) to intermediary layers of the decoder of the VLM. Each exit consists of a single exit transformer (ET) and an exit classifier (EC). The exit transformer is a replica of the layers in the decoder LM of the VLM. They are used as generators and feature classifiers as discriminators in a GAN-based setup, as shown in Fig. 2 . The task of the feature classifier is to correctly classify if the input is from the exit or final layer, and the task of the exit transformer is to generate representations similar to the final layer to fool the feature classifier of that exit. Figure 2: This figure illustrates our model’s architecture. The Q-Former output and previous tokens pass through the backbone. The final layer and ET outputs go to feature classifiers, refining ET to match final representations. During inference, the final classifier is used, and confident exits generate the caption. As the exit transformer is trained to generate representations similar to that of the final layer, we can use the final layer classifier at all exits as an EC with frozen parameters, which are used to map the outputs of the exit transformer to class probabilities. As the size of ETs is significantly smaller than that of ECs, it substantially reduces the number of parameters. In this way, a single LM layer attached to the exits helps produce similar feature representations and reduces the training parameters by utilizing a frozen final layer classifier. By attaching EEs, our method reduces the chances of mid-crisis or overthinking and makes the inference process faster. Fig. 1 shows how our method can speed up inference while maintaining comparable performance by using the EE methods. Adversarial training methods are prone to catastrophic forgetting Kirkpatrick et al. 2017 ; Ryu et al. 2022 and mode collapse , which can hinder exit training. To address these challenges, we propose both supervised and unsupervised strategies. In the supervised setting, when a small labeled dataset is available, we use hard labels to stabilize training, preventing catastrophic forgetting and avoiding local optima. In the unsupervised case, when labeled data is unavailable, we utilize soft labels from vanilla VLM or generate high-quality synthetic labels using CapFilt Li et al. 2022b , as employed in BLIP and BLIP-2. Our method remains effective across different scenarios. When a labeled dataset is available, it is used to mitigate catastrophic forgetting and mode collapse. If no labeled data is present, we employ knowledge distillation to solve this issue. When both labeled data and annotations are unavailable but computational resources are accessible, we generate synthetic labels using CapFilt. By providing solutions for all data constraints, our approach ensures stable and effective exit training while enhancing efficiency and generalization. In summary, our contributions are as follows: • We introduce an EE strategy named FREE for VLMs to effectively mitigate inference latency by reducing unnecessary computations inherent in their large-scale architecture. • We propose an efficient training strategy to improve performance at EE classifiers. FREE emulates the behaviour of the final layer at the exits through adversarial learning. This reduces the need for labeled

Read the complete text