LiteVLM: A Low-Latency Vision-Language Model Inference Pipeline for Resource-Constrained Environments

Jin Huang, Yuchao Jin, Le An, Josh Park · 2025

Read LiteVLM: A Low-Latency Vision-Language Model Inference Pipeline for Resource-Constrained Environments by Jin Huang, Yuchao Jin, Le An, Josh Park (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-07416.txt.

Introduction

Jin Huang, Yuchao Jin, and Le An lead a team that trims vision and language models until they answer fast enough for robots and self-driving cars. Their paper came out in two thousand twenty-five. Big models think slowly and use more compute than small devices have. Lag, not accuracy, is the wall that blocks real use.

Their pipeline cuts cost at three points. Patch selection throws out camera views with no content that matters. Token selection cuts the length of the input the language model has to work through. Speculative decoding then drafts answers faster. On a chip made for cars, the pipeline cut end to end lag more than twofold. Accuracy held steady through the cuts. Extra quantization pushed the speed gain higher. Each trick works with the others, and the authors frame the stack as a recipe for real time work.

Catalog record

CREATORJin Huang, Yuchao Jin, Le An, Josh Park
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE19 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-07416

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

LiteVLM: A Low-Latency Vision-Language Model Inference Pipeline for Resource-Constrained Environments Jin Huang, Yuchao Jin, Le An, Josh Park Abstract This paper introduces an efficient Vision-Language Model (VLM) pipeline specifically optimized for deployment on embedded devices, such as those used in robotics and autonomous driving. The pipeline significantly reduces the computational overhead by jointly leveraging patch selection to filter irrelevant camera views, a token selection module to reduce input sequence length for the LLM, and speculative decoding to accelerate token generation. Evaluation on the NVIDIA DRIVE Thor platform for automonous driving application, our pipeline achieves 2.5 × 2.5\times end-to-end latency reduction without compromising task accuracy. The speed-up further increases to 3.2 × 3.2\times when applying FP8 post-training quantization. These results demonstrate our pipeline as a viable solution for enabling real-time VLM deployment in resource-constrained environments. 1 Introduction Multi-modal Large Language Models (MLLMs), particularly Vision-Language Models (VLMs), have demonstrated remarkable capabilities in visual understanding and reasoning Shi et al. 2024 ; Liu et al. 2023 ; Bai et al. 2023 ; Zhu et al. 2023 ; Chen et al. 2023 ; Yang et al. 2023 ; Wang et al. 2024a , . Despite their strength, VLMs are computationally intensive. A typical architecture includes a Vision Transformer (ViT) Dosovitskiy et al. 2020 as encoder, an alignment module (e.g., MLP or Q-Former Li et al. 2023 structure) to align visual and text tokens, and an LLM decoder. In addition, many state-of-the-art VLMs also preprocess images of dynamic resolutions into fine-grained fixed-size patches Wang et al. 2024a to boost performance but with increased computation. In turn, challenges remain for real-time deployment on resource-constrained embedded hardware such as in robotics and autonomous driving, where inference latency becomes a critical bottleneck, as rapid decision making is essential in those applications. To address VLM latency, visual token compression Chen et al. 2024a ; Shang et al. 2024 ; Li et al. 2024a ; Lan et al. 2024 ; Cai et al. 2024a ; Hu et al. 2025 reduces input sequence length by merging, pooling, or pruning less informative tokens, which can mitigate the quadratic computational cost of self-attention layers in LLMs. Since visual tokens often dominate the input sequence, such compression can lead to significant computational savings. Another widely used technique is speculative decoding Leviathan et al. 2023 ; Cai et al. 2024b ; Li et al. 2024c ; Li et al. 2024b , accelerating the decoding stage by using a smaller draft model to propose multiple candidate tokens, which are then verified by the primary LLMs. This method significantly reduces the generation latency by potentially accepting multi-tokens per iteration compared with auto-regressive manner. However, these methods by their own may not be sufficient to meet latency requirement for production environment. This paper introduces a VLM pipeline specifically engineered for efficient deployment on embedded systems. Our key contributions are: 1. We propose an efficient VLM architecture LiteVLM by jointly leveraging patch selection, token selection, and speculative decoding. 2. We benchmark our method for autonomous driving application on NVIDIA DRIVE Thor platform and demonstrate \mathbf{2.5\times} reduction in latency compared with baseline while maintaining accuracy and even further speed up with FP8 quantization. 2 Related Work Figure 1 : The proposed framework builds upon a Vision-Language Model by introducing two novel modules: the Patch Selection Module and the Token Selection Module (highlighted in Orange ). We also incorporate Speculative Decoding Head (highlighted in Blue ) to accelerate the decoding process. Together, these components enable efficient token generation for real-time applications. 2.1 Visual Token Compression In VLMs, visual tokens frequently constitute the majority of the input sequence due to the fine-grained encoding of input images. This results in substantial computational overhead, particularly during the prefill stage, where the model processes the entire sequence before producing any output. To mitigate this issue, a range of visual token compression techniques have been proposed Chen et al. 2024a ; Shang et al. 2024 ; Li et al. 2024a ; Lan et al. 2024 ; Cai et al. 2024a ; Hu et al. 2025 . Some methods are based on Token Merging and reduce the token count by combining adjacent tokens or pooling their features Bai et al. 2023 ; Chen et al. 2023 . More advanced methods use adaptive merging based on routing to achieve variable compression rates Cai et al. 2024a ; Lan et al. 2024 ; Hu et al. 2025 . However, these techniques often require finetuning the VLM to adapt to the merged features. Another strategy is Token Pruning which aims to identify and remove less informative tokens based on importance scores Shang et al. 2024 ; Singh et al. 2024 . For example, FastV Chen et al. 2024a and HiRED Arif et al. 2024 prunes tokens based on self-attention scores from LLM or ViT encoder, which can avoid extensive model fine-tuning. While achieving good performance, these methods pose a risk of potentially discarding tokens associated with critical objects (e.g., distant pedestrians, traffic signs) that may receive low attention scores yet remain semantically crucial. 2.2 Vision-Language Models on Edge Devices Recent progress in multi-modal learning has led to the development of compact, high-performing open-source Vision-Language Models (VLMs), making them increasingly viable for deployment on resource-constrained edge devices. Notable examples include QwenVL-1.5B/3B Bai et al. 2023 ; Wang et al. 2024a and InternVL-1B/2B Chen et al. 2023 . These compact VLMs are increasingly being adopted by researchers in robotics and autonomous driving, domains where on-device deployment is often essential. For instance, studies such as Ma et al. 2023 ; Xu et al. 2023 ; Tian et al. 2024 ; Wang et al. 2024b have finetuned these models on driving-specific datasets for tasks including scene understanding, perception, behavior prediction, and trajectory generation. However, despite their reduced size compared to larger counterparts, these models, while easily manageable on high-end server GPUs, often still exceed the computational and memory limits of typical embedded devices. Consequently, real-world deployment remains a significant challenge, primarily due to stringent latency requirements and limited on-device resources. 3 Design Table 1 : Inference latency and accuracy of different execution stages on NVIDIA DRIVE Thor platforms. All models are deploy in FP16 by default. We also benchmarked our model in FP8 precision. Model Vision LLM Prefill LLM Decode Selection Module Latency (ms) End-to-End Avg. #Image Patches ViT Latency (ms) Avg. #Input Tokens Prefill Latency (ms) Avg. Extend-One Latency (ms) Decoding Latency (ms) Total Latency (ms) Speed-Up Combined Accuracy VLM 2B 12 136.9 3214 163.2 16.9 229.6 - 529.7 1.0\times 0.6618 VLM 2B + FastV Chen et al. 2024a (r=0.7) 12 136.9 2292 117.9 16.4 222.8 - 477.6 1.1\times 0.6610 VLM 2B + FastV Chen et al. 2024a (r=0.3) 12 136.9 1063 59.5 15.9 216.1 - 412.5 1.3\times 0.6468 VLM 2B + Eagle Li et al. 2024c 12 136.9 3214 163.2 9.0 121.4 - 421.5 1.3\times 0.6618 LiteVLM 3.5 45.1 858 54.2 7.7 104.1 10.2 213.6 \mathbf{2.5\times} 0.6602 LiteVLM in FP8 3.5 29.1 948 39.8 6.2 84.0 10.2 163.1 \mathbf{3.2\times} 0.6450 Figure 1 shows the overview of LiteVLM pipeline for an autonomous driving use case with multi-view inputs. Building upon a standard VLM, we introduce three key enhancements: 1. Patch Selection Module to reduce ViT input and subsequent LLM prefill latency by filtering

Read the complete text