Towards Minimal Fine-Tuning of VLMs
Tiange Luo, Lajanugen Logeswaran, Jaekyeom Kim, Justin Johnson, Honglak Lee, University of Michigan, LG AI Research · 2025
Read Towards Minimal Fine-Tuning of VLMs by Tiange Luo, Lajanugen Logeswaran, Jaekyeom Kim, Justin Johnson, Honglak Lee, University of Michigan, LG AI Research (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-19219.txt.
Introduction
Tiange Luo, Lajanugen Logeswaran, Jaekyeom Kim, and Justin Johnson fine-tune vision-language models more cheaply by being choosy. Standard low-rank adaptation updates every token and every attention head alike. Image-LoRA instead applies its small update only to visual tokens, leaving pure text flow untouched when no image is present.
It narrows further to the value path of a small set of heads, picked by a one-pass influence probe. The result needs far fewer trainable parameters and less training compute. On localization tests it matched standard adaptation, and shone when images dominated the input. Checks on text and video question answering, and on math, confirmed the savings.
Catalog record
| CREATOR | Tiange Luo, Lajanugen Logeswaran, Jaekyeom Kim, Justin Johnson, Honglak Lee, University of Michigan, LG AI Research |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 63 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-19219
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Towards Minimal Fine-Tuning of VLMs Tiange Luo, Lajanugen Logeswaran, Jaekyeom Kim, Justin Johnson, Honglak Lee, University of Michigan, LG AI Research Abstract We introduce Image‑LoRA, a lightweight parameter efficient fine‑tuning (PEFT) recipe for transformer‑based vision–language models (VLMs). Image‑LoRA applies low‑rank adaptation only to the value path of attention layers within the visual‑token span, reducing adapter-only training FLOPs roughly in proportion to the visual‑token fraction. We further adapt only a subset of attention heads, selected using head influence scores estimated with a rank‑1 Image-LoRA, and stabilize per‑layer updates via selection‑size normalization. Across screen‑centric grounding and referring benchmarks spanning text-heavy to image-heavy regimes, Image‑LoRA matches or closely approaches standard LoRA accuracy while using fewer trainable parameters and lower adapter-only training FLOPs. The method also preserves the pure‑text reasoning performance of VLMs before and after fine‑tuning, as further shown on GSM8K. 1 Introduction Low‑rank adaptation (LoRA) ( 9 ) has become a strong baseline for post‑training of large language models because it exposes a small, trainable subspace while keeping the base network frozen 18 . However, parameter‑efficient tuning (PEFT) for transformer-based vision–language models (VLMs) presents different constraints than text‑only LLMs: prompts often contain long stretches of text mixed with images, and attention modules comprise many heads, even though only a subset appears to carry most of the routing signal for visual reasoning, such as grounding and pointing 11 ; 1 . Existing PEFT recipes typically treat all tokens and all heads uniformly, which leaves efficiency on the table. We revisit LoRA through the lens of VLM structure and ask: what if we only adapt the parts that directly affect grounded vision–text reasoning? Figure 1 : Image-LoRA vs. Standard LoRA. Comparison of accuracy and adapter-only training FLOPs on ScreenSpot-Pro. Points are connected across increasing input-text:image token ratios (1:2 \rightarrow 1:5), where we control the ratios by dynamically adjusting image sizes. FLOPs are token-limited and computed as forward + backward multiply–adds. Our first design choice is visual-token only fine-tuning. Restricting adaptation to the visual-token span reduces adapter-only training FLOPs roughly in proportion to the visual-token fraction, as it avoids computation through both the input prompt and the output text. A further benefit is that fine-tuning only on the visual-token span ensures that the adapted weights never affect the pure-text reasoning performance of the fine-tuned VLMs, as also demonstrated in our GSM8K 5 experiment. This design is further supported by recent evidence that textual information can be routed through visual-token channels 24 ; 4 , suggesting that adapting visual tokens alone could be powerful. To ensure that updating the attention paths on the visual-token span meaningfully influences the representations of text tokens, we compared adapting K and V and found that adapting only V performs best. Additionally, localization-style objectives (e.g., pointing) place supervision primarily in the pixels, while the long text prompt mainly serves as task scaffolding. Concentrating adapters on the visual span allocates capacity where the loss is informative and may discourage shortcutting through language priors, as shown in our ViLP 15 experiment. Figure 2 : Overview. Left: We evaluate Image-LoRA on grounded vision–language reasoning datasets, including ScreenSpot-Pro 14 and RefCOCO 12 , where the model takes a text query with its system prompt and outputs a point indicating the referred object. We further evaluate on the pure text reasoning dataset GSM8K 5 to confirm that Image-LoRA does not affect pure text reasoning, and on ViLP 15 containing VQAs both aligned with and against language priors to show that Image-LoRA may mitigate shortcutting through such priors. Right: Image-LoRA is applied only to selected heads along the value path of attention layers within the visual-token span (the pink area). Second, rather than treating all attention heads equally, we ask whether a small set suffices for our tasks. A naïve approach is a finite‑difference ablation: enable one head at a time, recompute the loss over the dataset, and keep the heads with the largest loss drops—but this is prohibitively expensive. To this end, we propose a head influence score estimated by attaching a rank‑1 Image‑LoRA to every head, running a single forward–backward pass with the task loss, and scoring each head by the squared gradient norm of its Image-LoRA parameters – a first-order empirical-Fisher proxy that identifies a high-impact subset without iterative search. After head selection, different layers may retain different numbers of heads—unlike standard LoRA, which updates all heads per layer. To maintain stable per-layer update magnitude, we apply a selection-size layer normalization as part of Image-LoRA. Putting these pieces together, we propose Image-LoRA . The adapter is injected pre-RoPE (rotary position embeddings) and pre-KV‑cache so that its effect persists into decoding. Proof-of-concept experiments on screen‑centric grounding and referring expression datasets (ScreenSpot‑Pro 14 and RefCOCO 12 ) span a wide range of input-text:image token ratios (1:1/2, 1:1, 1:2, 1:3, 1:4, 1:5) along with QWen2.5-VL-7B, QWen2.5-VL-72B, and LLaVA-NeXT-7B models. We enforce the chosen ratio during both training and evaluation to probe regimes from text‑heavy to image‑heavy. Our results, shown in Figure 1 , demonstrate that Image‑LoRA retains accuracy while reducing trainable parameters and adapater-only training FLOPs, especially in image‑heavy settings. Furthermore, experiments on GSM8K 5 show that Image‑LoRA preserves pure-text reasoning performance, whereas standard LoRA may not. We also evaluate on ViLP 15 , a VQA dataset deliberately constructed to follow or break language priors, where Image‑LoRA outperforms standard LoRA. In the ablation studies, we further evaluate the proposed head-selection method against correlation-based, global-random, and per-layer-random baselines, as well as examine various design choices for Image-LoRA. Although our proof-of-concept experiments above show some promising signals for Image-LoRA, our experimental scale is limited, and fundamental constraints inherent to the Image-LoRA formulation remain and require further investigation, as discussed in Section 5 . Collectively, we present a vision-orientated PEFT recipe for transformer-based VLMs. By restricting adaptation to the visual-token span, stabilizing updates when adapting a subset of attention heads, and preserving pure-text reasoning behavior, Image-LoRA offers a structured step toward enabling web-scale vision pre-training on top of existing reasoning LLMs ( 10 ) . 2 Related Works PEFT of VLMs. A common approach to efficient fine-tuning of VLMs is to reduce the number of parameters to update, which is often called parameter-efficient fine-tuning (PEFT). One important milestone in PEFT is leveraging the reparameterization of the parameters of models, often notably represented by low-rank adaptation (LoRA) ( 9 ) , which has also been shown to be effective for VLMs ( 20 ; 7 ; 32 ; 27 ) . Recently, Schulman et al. 18 examined the effects of applying LoRA either only to attention layers or to all layers, including MLP/MoE layers, a topic explored in prior works ( 26 ; 31 ; 21 ; 8 ; 6 ; 23 ) . In contrast, our work explores a more structured approach—applying LoRA selectively to attention heads and to the visual token span only. Meanwhile, 19 reported findings that contradict 18 , showing that LoRA can behave differently from full fine-tuning, with both studies focusing on pure text-based benchmarks. In contrast, we focus on visual reasoning tasks, including both grounding and VQA tasks. Token selection and reduction for VLMs. Token selection and reduction have been applied to improve VLM inference efficiency. Token pruning removes unimportant input tokens based on relevance