Visual Variational Autoencoder Prompt Tuning

Xi Xiao, Yunbei Zhang, Yanshu Li, Xingjian Li, Tianyang Wang, Jihun Hamm, Xiao Wang, Min Xu, [0.5em] University of Alabama at Birmingham · 2025

Read Visual Variational Autoencoder Prompt Tuning by Xi Xiao, Yunbei Zhang, Yanshu Li, Xingjian Li, Tianyang Wang, Jihun Hamm, Xiao Wang, Min Xu, [0.5em] University of Alabama at Birmingham (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-17650.txt.

Introduction

Xi Xiao and co-authors make vision model prompts that adapt to each image in a 2025 paper. Prompt tuning adapts a frozen vision transformer by learning a few prompt tokens, a cheap alternative to full fine-tuning. Standard prompts are static: one set serves every image, ignoring that each picture differs. Their method generates the prompts per image instead.

A variational autoencoder reads the image into a latent sketch, then decodes that sketch into prompts tuned to the picture's own features. Across three benchmark suites covering fine-grained categories and broad transfer tasks, the dynamic prompts beat the leading tuning methods, with the clearest gain on one suite at over three points. The average gain across suites was about two points. The authors pitch input-aware prompts as the next step for cheap adaptation.

Catalog record

CREATORXi Xiao, Yunbei Zhang, Yanshu Li, Xingjian Li, Tianyang Wang, Jihun Hamm, Xiao Wang, Min Xu, [0.5em] University of Alabama at Birmingham
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE17 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-17650

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Visual Variational Autoencoder Prompt Tuning Xi Xiao, Yunbei Zhang, Yanshu Li, Xingjian Li, Tianyang Wang, Jihun Hamm, Xiao Wang, Min Xu, [0.5em] University of Alabama at Birmingham Abstract Parameter-efficient fine-tuning (PEFT) has emerged as a crucial approach for adapting large vision transformers to downstream tasks without the prohibitive computational costs of full fine-tuning. While existing visual prompt tuning (VPT) methods have made significant strides, they predominantly rely on static, domain-specific prompts that fail to capture the rich visual diversity within individual instances. This paper introduces V 2 APT (Visual Variational Autoencoder Prompt Tuning), a novel framework that generates dynamic, input-dependent prompts using a variational autoencoder architecture. By learning a latent representation of image-specific features and decoding them into customized prompts, V 2 APT adapts to the unique visual characteristics of each input. Extensive experiments on FGVC, HTA, and VTAB-1k benchmarks demonstrate that our approach consistently outperforms state-of-the-art PEFT methods. Notably, V 2 APT achieves +3.2% improvement over VPT-Deep on HTA, with an average performance gain of +2.0% across all three datasets. 1 Introduction “Generative models, including VAEs, enable machines to understand the underlying data distribution, paving the way for creative and unsupervised AI.” — Yann LeCun Vision Transformers (ViTs) Dosovitskiy et al. 2021 ; Liu et al. 2021 have revolutionized computer vision, establishing new performance benchmarks across the visual recognition landscape. Despite their impressive capabilities, harnessing these models for novel applications presents a formidable challenge due to their vast parameter space—often reaching hundreds of millions of trainable weights. Parameter-Efficient Fine-Tuning (PEFT) strategies, particularly Visual Prompt Tuning (VPT) Jia et al. 2022 , offer an elegant solution to this computational bottleneck by introducing compact, learnable prompt vectors while preserving the integrity of the pre-trained architecture. This approach has catalyzed numerous innovations, from cross-task cross-domain adaptations Bahng et al. 2022 ; Han et al. 2023 ; Pei et al. 2024 ; Zhang et al. 2025 ; Cai et al. 2024 to multimodal applications Khattak et al. 2023 ; Zhou et al. 2022 . However, conventional VPT approaches rely on static, randomly initialized prompts—a fundamental limitation that restricts their adaptability Jia et al. 2022 . This one-size-fits-all strategy overlooks the rich visual diversity within semantic categories, where images sharing the same label (e.g., “cat”) may display vastly different visual characteristics—from close-up portraits to distant silhouettes, from various breeds to different postures. Domain-specific prompts, while efficient, remain blind to these crucial instance variations, resulting in adaptation strategies that cannot fully respond to the unique visual signature of each input. To address these limitations, we propose V 2 APT (Visual Variational Autoencoder Prompt Tuning), a novel framework that dynamically generates input-dependent prompts tailored to individual images. V 2 APT leverages a Variational Autoencoder (VAE) architecture to capture and encode image-specific features into a latent representation, which is then decoded into customized prompts that adapt to the unique visual properties of each input. This approach enhances both adaptation precision and generalization robustness, enabling more effective transfer learning across diverse visual tasks and domains. We validate V 2 APT on the FGVC Krause et al. 2013 , HTA Huang et al. 2023c , and VTAB-1k Zhai et al. 2019b benchmarks. Our experimental results consistently demonstrate that V 2 APT outperforms existing state-of-the-art PEFT methods, while maintaining a minimal computational footprint. Specifically, in comparison to VPT-Deep Jia et al. 2022 , V 2 APT achieves a substantial absolute improvement of +3.2% on the HTA benchmark, with even more significant gains of +2.27% on VTAB-1k. These results highlight generative latent modeling as a promising new paradigm for prompt-based adaptation in ViTs. 2 Related Works Figure 1 : Comparison of standard Visual Prompt Tuning (VPT) and our proposed V 2 APT framework. ( Left ) VPT employs static Domain-Specific Prompts , which remain fixed across different images. ( Middle ) Our V 2 APT framework introduces Instance-dependent Prompts , dynamically generated via a Variational Autoencoder (VAE) based on each input image. ( Right ) The VAE encodes image embeddings into a KL-divergence-regularized latent space, then decodes them into instance prompts that combine with domain-specific prompts before entering the transformer, maintaining the same token count as VPT. 2.1 Visual Prompt Tuning (VPT) Visual Prompt Tuning (VPT) enhances Vision Transformers (ViTs) by prepending learnable prompts to inputs Bahng et al. 2022 , extending to various feature spaces while preserving model architecture and significantly reducing computational overhead Jia et al. 2022 . Recent advancements have refined this approach through multi-level prompt injection Han et al. 2023 and spatially aware prompts for improved feature alignment Pei et al. 2024 . The versatility of VPT has further expanded into multi-modal domains such CLIP-based models Khattak et al. 2023 ; Zhou et al. 2022 . Despite these innovations, existing approaches rely exclusively on static soft prompts, overlooking the potential of structured prior knowledge. While numerous VPT variants Chen et al. 2022 ; Cheng et al. 2023 ; Zeng et al. 2024 ; He et al. 2023 ; Huang et al. 2023a ; Huang et al. 2023b ; Khattak et al. 2023 ; Zhang et al. 2024 have improved performance across diverse tasks, none have explored data-driven prompt generation from latent space. Our work addresses this fundamental gap by integrating VAEs to generate dynamic, structured prompts, effectively bridging the divide between hard and soft prompting paradigms to enhance generalization and robustness in ViTs. 2.2 VAEs for Representation Learning Variational Autoencoders (VAEs) Kingma and Welling 2013 have been extensively employed for learning structured latent representations by enforcing a prior distribution on the latent space. In vision tasks, VAEs effectively capture underlying data distributions, facilitating generative modeling, feature regularization, and domain adaptation Zhao et al. 2017 ; Tolstikhin et al. 2018 ; Ma et al. 2025 ; Higgins et al. 2017 . While recent research has begun exploring VAEs for adaptive representations in transformers Xu et al. 2022 , their potential for prompt learning remains largely unexplored. Our work bridges this gap by introducing VAE-based instance-specific image prompts, where the regularized latent space models semantic structure across diverse visual inputs. This approach ensures adaptive and task-specific tuning that responds to individual image characteristics while maintaining computational efficiency. 3 Methodology In this section, we present V 2 APT, a novel visual prompt tuning framework that integrates Variational Autoencoders (VAEs) to dynamically generate instance-specific prompts. By leveraging latent distribution modeling, V 2 APT adapts prompts to the underlying data structure, improving task relevance and adaptability. We first introduce the fundamentals of visual prompt tuning in § 3.1 , followed by the detailed description of V 2 APT in § 3.2 . The overall architecture is illustrated in Fig. 1 . 3.1 Preliminary Visual Prompt Tuning. Visual Prompt Tuning (VPT) introduces a set of learnable prompts \mathbf{P}=\{\mathbf{P}^{1},\mathbf{P}^{2},\ldots,\mathbf{P}^{N}\} into a pre-trained ViT model \mathcal{T} with N transformer layers. These d -dimensional prompts are prepended to the input embeddings and optimized during training while keeping the rest of the model frozen. For the first layer, the contextual embeddings are computed as \mathbf{Z}^{1}=L_{1}(\mathbf{P}^{1},\mathbf{E}) , where \mathbf{E} denotes the frozen patch embeddings of input image patches, and \mathbf{P}^{1} represents the trainable prompts. For subsequent layers, the embeddings are iteratively updated using \mathbf{Z}^{i}=L_{i}(\mathbf{P}^{i},\mathbf{Z}^{i-1}) . However, these prompts are domain-specific, meaning they remain fixed for a given dataset

Read the complete text