Efficient Inference for Large Vision-Language Models: Bottlenecks, Techniques, and Prospects
Jun Zhang, Yicheng Ji, Feiyang Ren, Yihang Li, Bowen Zeng, Zonghao Chen, Ke Chen, Lidan Shou, Gang Chen, Huan Li · 2026
Read Efficient Inference for Large Vision-Language Models: Bottlenecks, Techniques, and Prospects by Jun Zhang, Yicheng Ji, Feiyang Ren, Yihang Li, Bowen Zeng, Zonghao Chen, Ke Chen, Lidan Shou, Gang Chen, Huan Li (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-05546.txt.
Introduction
Zhang and nine colleagues review fast inference for large vision-language models, where image tokens swamp cost across encoding, prefilling, and decoding. They sort tricks by life-cycle stage and by three axes: information density, long-context attention, and memory limits. The review ends with four frontiers, from hybrid compression to split-stage serving, backed by pilot tests.
Catalog record
| CREATOR | Jun Zhang, Yicheng Ji, Feiyang Ren, Yihang Li, Bowen Zeng, Zonghao Chen, Ke Chen, Lidan Shou, Gang Chen, Huan Li |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 82343 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.05546
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Efficient Inference for Large Vision-Language Models: Bottlenecks, Techniques, and Prospects Jun Zhang, Yicheng Ji, Feiyang Ren, Yihang Li, Bowen Zeng, Zonghao Chen, Ke Chen, Lidan Shou, Gang Chen, Huan Li Abstract Large Vision-Language Models (LVLMs) enable sophisticated reasoning over images and videos, yet their inference is hindered by a systemic efficiency barrier known as visual token dominance . This overhead is driven by a multi-regime interplay between high-resolution feature extraction, quadratic attention scaling, and memory bandwidth constraints. We present a systematic taxonomy of efficiency techniques structured around the inference lifecycle, consisting of encoding , prefilling , and decoding . Unlike prior reviews focused on isolated optimizations, we analyze the end-to-end pipeline to reveal how upstream decisions dictate downstream bottlenecks, covering compute-bound visual encoding, the intensive prefilling of massive contexts, and the “visual memory wall” in bandwidth-bound decoding. By decoupling the efficiency landscape into the axes of shaping information density, managing long-context attention, and overcoming memory limits, this work provides a structured analysis of how isolated optimizations compose to navigate the trade-off between visual fidelity and system efficiency. The survey concludes by outlining four future frontiers supported by pilot empirical insights, including hybrid compression based on functional unit sensitivity, modality-aware decoding with relaxed verification, progressive state management for streaming continuity, and stage-disaggregated serving through hardware-algorithm co-design. Our literature repository is at https://github.com/SuDIS-ZJU/Efficien t-LVLMs-Inference . † † footnotemark: † † footnotetext: ✌ Equal contribution. ✉ Corresponding author. 1 Introduction Large Vision-Language Models (LVLMs) 160 ; 5 ; 164 have evolved from research artifacts into the infrastructure for complex multimodal reasoning. However, as these models scale to process fine-grained visual inputs and long-form video streams, they encounter a systemic efficiency barrier: visual token dominance 187 ; 143 ; 92 . Unlike text-only inputs, visual data yields orders of magnitude more tokens, pushing inference into a regime constrained not merely by compute cycles, but by the quadratic scaling of attention and the ‘‘visual memory wall’’ 1 1 1 For instance, a Qwen2.5-VL-72B processing 20 images already exceeds 40K tokens and 13 GB of cache, while a 5-second 720p video surpasses 50K tokens and 16 GB. ( 154 ; 78 ; 158 ) . The central thesis of this survey is that LVLM inference is not a monolithic workload, but a dynamic pipeline traversing three distinct hardware regimes: i) Encoding (specifically visual encoding) is compute-bound by high-resolution feature extraction; ii) Prefilling suffers from the quadratic complexity of massive visual contexts; and iii) Decoding hits the memory wall due to static, bandwidth-consuming Key-Value (KV) caches. Optimizing one stage in isolation often shifts the bottleneck elsewhere without improving end-to-end latency. Despite the surge in interest, the current literature remains fragmented. Prior reviews have predominantly focused on isolated verticals, such as token compression techniques ( 124 ) or efficient architectures for specific modalities ( 227 ; 202 ) 2 2 2 Section 8 provides a detailed related survey discussion. . These works, however, overlook the systemic interconnectivity of the inference pipeline. They lack a holistic view of how upstream decisions (e.g., encoder resolution) dictate downstream bottlenecks (e.g., decoding bandwidth), leaving a gap in understanding end-to-end efficiency. This survey bridges this gap by advancing a unified, stage-wise taxonomy of efficient LVLM inference. We decouple the efficiency landscape into three critical axes: shaping information density (encoding), managing long-context attention (prefilling), and overcoming memory bandwidth limits (decoding). This framework provides a structured lens to evaluate how isolated optimizations compose, helping researchers navigate the trade-off between visual fidelity and system efficiency. Figure 1: Three-stage pipeline for LVLM inference. Figure 2: Efficient encoding workflow: architectural optimization ( Section 3.1 , Section 3.2 ) and input reduction ( Section 3.3 , Section 3.4 , Section 3.5 ). 2 Preliminaries and Inference Dynamics LVLMs encounter unique efficiency bottlenecks compared to Large Language Models (LLMs), primarily due to the massive visual inputs. We formalize the canonical LVLM architecture ( Section 2.1 ) and analyze its inference dynamics through a “physics of computing” lens ( Section 2.2 ), mapping hardware bottlenecks to user-centric metrics to structure the survey ( Section 2.3 ). 2.1 The Canonical LVLM Architecture LVLMs typically adopt a three-stage pipeline ( Figure 2 ) that connects a vision encoder to an LLM. 3 3 3 Detailed component implementations and model taxonomy are provided in Appendix B . Given a multimodal tuple (\mathbf{V},\mathbf{T}) comprising raw visual input 4 4 4 F frames with (H\cdot W) resolution and RGB channels. \mathbf{V}\in\mathbb{R}^{F\times H\times W\times 3} and a text prompt \mathbf{T} of N_{t} tokens, the pipeline is formalized as: ① Visual Encoding. The encoder \mathcal{E}_{\phi} (with parameters \phi ) processes \mathbf{V} into patch embeddings \mathbf{X}_{v}\in\mathbb{R}^{N_{p}\times D_{v}} with N_{p} the output patch number 5 5 5 For single-frame inputs ( F=1 ), N_{p}=(H\cdot W)/P^{2} with patch size (P\times P) ; for videos with F\geq 2 , N_{p} depends on keyframe selection ( Section 3.3 ), adaptive resolution ( Section 3.4 ), and other compression strategies (e.g., frame pooling and Q-Former). and D_{v} the vision channel dimension. ② Modality Alignment. A modality adapter \mathcal{A}_{\theta} (with parameters \theta ) maps \mathbf{X}_{v} into the LLM’s latent space, yielding visual context \mathbf{H}_{v}=\mathcal{A}_{\theta}(\mathbf{X}_{v})\in\mathbb{R}^{N_{v}\times D_{\mathcal{L}}} with D_{\mathcal{L}} the LLM hidden dimension. The effective token count N_{v} varies by projection strategy (e.g., pooling), defined by the compression ratio r=N_{v}/N_{p} . ③ Autoregressive Generation. The LLM backbone \mathcal{L}_{\psi} (with parameters \psi ) concatenates visual and text embeddings into a joint context \mathbf{C}=[\mathbf{H}_{v};\mathbf{H}_{t}] (where \mathbf{H}_{t}\in\mathbb{R}^{N_{t}\times D_{\mathcal{L}}} represents the prompt of N_{t} textual tokens) to generate the output response \mathbf{Y}=(y_{1},\dots,y_{N_{o}}) of length N_{o} autoregressively: p(\mathbf{Y}\mid\mathbf{C})=\prod\nolimits_{k=1}^{N_{o}}p(y_{k}\mid\mathbf{C},y_{<k};\psi). (1) Here, a defining characteristic is the visual token dominance : the visual content ( N_{v}\approx 576 – 4,000+) significantly exceeds standard text prompts ( N_{v}\gg N_{t} ). This structural imbalance dictates the inference bottlenecks analyzed below. 2.2 The Physics of Inference Bottlenecks We model the end-to-end inference latency as: \tau_{\text{total}}=\tau_{\text{ENC}}+\tau_{\text{PFL}}+N_{o}\cdot\tau_{\text{DEC}}, (2) where the first two terms contribute to Time-to-First-Token (TTFT) at encoding and prefilling, respectively, and \tau_{\text{DEC}} determines Time-Per-Output-Token (TPOT) at decoding. To identify bottlenecks, we apply the Roofline model 6 6 6 A detailed Roofline analysis is provided in Appendix E . , which bounds performance based on the workload’s arithmetic intensity \mathcal{I} (FLOPs / Byte). A stage is compute-bound if \mathcal{I}\geq\pi_{\text{peak}}/\beta_{\text{mem}} , saturating the peak compute throughput \pi_{\text{peak}} ; otherwise, it is memory-bound , throttled by the memory bandwidth \beta_{\text{mem}} . As summarized in Table 1 , encoding is compute-bound with high arithmetic intensity from dense matrix operations; prefilling exhibits mixed behavior where both computation (quadratic attention) and memory I/O (KV cache materialization) can dominate; and decoding is strictly memory-bound due to the low arithmetic intensity of autoregressive token generation. Understanding these stage-specific bottlenecks is crucial for targeting optimization efforts. Table 1: Hardware bottlenecks, arithmetic intensity, and complexity dynamics across the three inference stages. Characteristic Encoding Stage Prefilling Stage Decoding Stage Primary Metric TTFT TTFT TPOT Bottleneck Compute-Bound Compute & Memory Bound Memory-Bound Arithmetic Intensity High ( \gg 1 ) Medium Low ( \ll 1 ) Complexity (FLOPs) \mathcal{O}(N_{p}\cdot D_{v}^{2}) \mathcal{O}((N_{v}+N_{t})^{2}\cdot D_{\mathcal{L}}) \mathcal{O}((N_{v}+N_{t})\cdot D_{\mathcal{L}}) LVLM Challenge High-res inputs ( N_{p}\uparrow ) surge FLOPs ( N_{v}\gg N_{t} ) causes quadratic spikes Static visual KV cache saturates VRAM Encoding Stage: Compute-Bound . This stage executes dense matrix multiplications over N_{p} patches, a high-intensity