TokenFLEX: Unified VLM Training for Flexible Visual Tokens Inference

Junshan Hu, Jialiang Mao, Zhikang Liu, Zhongpu Xia, Peng Jia, Xianpeng Lang · 2025

Read TokenFLEX: Unified VLM Training for Flexible Visual Tokens Inference by Junshan Hu, Jialiang Mao, Zhikang Liu, Zhongpu Xia, Peng Jia, Xianpeng Lang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-03154.txt.

Introduction

Hu and co-authors present a training method, published in 2025, that lets vision-language AI models adjust how much of a picture they take in. Such models usually convert every image into the same fixed number of picture tokens. Simple images then waste computing power, while complex ones starve for detail. The team's framework encodes an image into a flexible number of tokens. The language model behind it learns to work well at any of those counts.

Two ideas drive the design. A new training scheme practices the model at many token counts at once, so skill does not collapse when the count drops. A flexible inference step then picks the count per image. Easy images get few tokens and fast answers. Hard images get the detail they need. The work targets the cost of running large vision models at scale.

Catalog record

CREATORJunshan Hu, Jialiang Mao, Zhikang Liu, Zhongpu Xia, Peng Jia, Xianpeng Lang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-03154

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

TokenFLEX: Unified VLM Training for Flexible Visual Tokens Inference Junshan Hu, Jialiang Mao, Zhikang Liu, Zhongpu Xia, Peng Jia, Xianpeng Lang Abstract Conventional Vision-Language Models (VLMs) typically utilize a fixed number of vision tokens, regardless of task complexity. This one-size-fits-all strategy introduces notable inefficiencies: using excessive tokens leads to unnecessary computational overhead in simpler tasks, whereas insufficient tokens compromise fine-grained visual comprehension in more complex contexts. To overcome these limitations, we present TokenFLEX, an innovative and adaptable vision-language framework that encodes images into a variable number of tokens for efficient integration with a Large Language Model (LLM). Our approach is underpinned by two pivotal innovations. Firstly, we present a novel training paradigm that enhances performance across varying numbers of vision tokens by stochastically modulating token counts during training. Secondly, we design a lightweight vision token projector incorporating an adaptive pooling layer and SwiGLU, allowing for flexible downsampling of vision tokens and adaptive selection of features tailored to specific token counts. Comprehensive experiments reveal that TokenFLEX consistently outperforms its fixed-token counterparts, achieving notable performance gains across various token counts—enhancements of 1.6 %, 1.0%, and 0.4% with 64, 144, and 256 tokens, respectively—averaged over eight vision-language benchmarks. These results underscore TokenFLEX’s remarkable flexibility while maintaining high-performance vision-language understanding. 1 Introduction Vision-language models (VLMs) [ 28 , 25 , 26 , 8 , 7 , 3 , 44 , 33 , 45 , 47 ] have become foundational in multi-modal artificial intelligence, facilitating advancements in tasks such as visual question answering, image captioning, and cross-modal retrieval. Commercial models like GPT-4o [ 15 ] and Claude [ 2 ] consistently extend the capabilities of VLMs. Presently, the open-source community is rapidly advancing, achieving performance on par with, or even exceeding, that of proprietary models by improving model architectures, data strategy, and training recipes [ 32 , 19 , 9 , 4 ] . However, a fundamental limitation persists: existing VLMs enforce a fixed vision token count for all tasks, creating a rigid trade-off between task-specific requirements and model performance. Simple tasks (e.g., coarse category classification) are burdened with redundant tokens, while complex tasks (e.g., fine-grained scene analysis) suffer from insufficient visual detail due to token scarcity. This one-size-fits-all design forces models to compromise accuracy for tasks requiring precise visual understanding or waste computational resources on simpler scenarios. Figure 1 : Comparison of TokenFLEX with methods using fixed vision token counts. TokenFLEX is trained with a stochastic dynamic image token count, while other methods are trained with fixed token counts of 64, 144, and 256, respectively. The performance is measured as the average score across 8 multi-modal benchmarks on OpenCompass [ 11 ] . TokenFLEX consistently outperforms the fixed-token methods, particularly when using fewer tokens, such as 64. Current frontier VLMs face significant limitations in adjusting the number of vision tokens during inference, primarily due to two key challenges. The first challenge stems from the fact that these models are trained with a fixed number of vision tokens. For instance, LLaVA [ 26 , 19 ] consistently uses 729 tokens per image. Altering this number during inference can lead to out-of-distribution (OOD) issues, resulting in a notable decline in performance. As illustrated in Figure 1 , a model trained with a fixed 256 tokens exhibits a 4.2% drop in the OpenCompass [ 11 ] metric when using 64 tokens during inference, compared to a model explicitly trained with 64 tokens. The second challenge lies in the design of the projector, which is typically built for a fixed number of vision tokens and does not support arbitrary modifications. For example, LLaVA-OneVision [ 19 ] employs an MLP to map vision features, while PixelShuffle in InternVL [ 8 , 9 ] only supports a fixed 4x downsampling. Similarly, MiniCPM-V [ 47 ] relies on 64 predefined learnable queries. Although some projectors [ 46 , 21 , 31 ] theoretically support changing the visual token numbers during inference, their effectiveness still requires further exploration. In this paper, we propose TokenFLEX, a novel vision-language framework that liberates VLMs from fixed visual token constraints. Our approach enables flexible adjustment of visual token quantities based on task requirements while enhancing the projector architecture to support dynamic token configurations. Our key contributions are as follows: 1) Dynamic Token Mechanism. To address the fixed token constraint in training, we develop a stochastic training method that randomly select the number of vision token from a predefined set for each sample. This forces the model to learn robust cross-model alignment consistency across varying token numbers, effectively mitigating the OOD performance degradation during inference. This training paradigm supports flexible inference with arbitrary token counts while maintaining accuracy comparable to fixed-token baselines. 2) Lightweight Token-Adaptive Projector. To address the weakness of previous projectors, we design a new lightweight token-adaptive projector that incorporates an adaptive average pooling layer and SwiGLU [ 39 ] . The adaptive average pooling layer enables flexible modification of the number of vision tokens, while SwiGLU leverages its gate mechanisms to dynamically assign weights to visual features, prioritizing salient information under varying token configurations while suppressing redundant details. This adaptive weighting ensures critical visual semantics are preserved even when token counts are reduced. Extensive experiments across a broad range of vision-language benchmarks demonstrate that TokenFLEX consistently outperforms various fixed-token baselines. Notably, TokenFLEX shows improvements of 1.6%, 1.0%, and 0.4% with 64, 144, 256 tokens, respectively, averaged over eight benchmarks when compared to fixed-token baselines. Additionally, the proposed flexible token-length encoding mechanism further reduces training costs, in our experiments, it decreased visual token usage by up to 28% and shortened training time by 13%, enabling more efficient model training without sacrificing performance. 2 Related Work Figure 2 : Overall architecture. TokenFLEX combines a Visiual Encoder and a Large Language Model connected by a dynamic token projector. (Left) During the training phase, each sample randomly selects an image token length; during inference, the image token length can be chosen on-demand based on task complexity and computational budget. (Right) Our light-weight projector acts as an adaptive filter, allowing it to selectively emphasize important image tokens. 2.1 Large Vision-Language Models (VLMs) Recent advances in Vision-Language Models (VLMs) [ 28 , 25 , 26 , 8 , 7 , 3 , 44 , 33 , 45 , 47 , 24 , 20 , 9 , 31 ] have demonstrated remarkable cross-modal understanding capabilities. Pioneering works like Flamingo [ 1 ] established the paradigm of fusing frozen vision encoders (e.g., CLIP [ 37 ] ) with large language models (LLMs) through cross-modal projectors. This architecture has become the standard for modern VLMs: visual features extracted by vision encoders are projected into vision tokens via lightweight adapters (e.g., linear layers [ 24 ] or Q-Former modules [ 20 ] ), which are then concatenated with text tokens as input to LLMs. Recent advancements, exemplified by LLaVA family [ 24 , 19 ] , Qwen2-VL [ 44 ] , and InternVL2.5 [ 9 ] , have significantly improved multimodal understanding capabilities through scaled-up training datasets and optimized training recipes. However, most existing methods suffer from a critical limitation: they adopt fixed vision token allocation (e.g. 256 tokens per image [ 9 ] ) regardless of input complexity or computational constraints. This rigid

Read the complete text