Efficient Inference for Large Vision-Language Models: Bottlenecks, Techniques, and Prospects

Jun Zhang, Yicheng Ji, Feiyang Ren, Yihang Li, Bowen Zeng, Zonghao Chen, Ke Chen, Lidan Shou, Gang Chen, Huan Li · 2026

Read Efficient Inference for Large Vision-Language Models: Bottlenecks, Techniques, and Prospects by Jun Zhang, Yicheng Ji, Feiyang Ren, Yihang Li, Bowen Zeng, Zonghao Chen, Ke Chen, Lidan Shou, Gang Chen, Huan Li (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-05546.txt.

Introduction

Zhang and nine colleagues review fast inference for large vision-language models, where image tokens swamp cost across encoding, prefilling, and decoding. They sort tricks by life-cycle stage and by three axes: information density, long-context attention, and memory limits. The review ends with four frontiers, from hybrid compression to split-stage serving, backed by pilot tests.

Catalog record

CREATORJun Zhang, Yicheng Ji, Feiyang Ren, Yihang Li, Bowen Zeng, Zonghao Chen, Ke Chen, Lidan Shou, Gang Chen, Huan Li
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE82343
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.05546

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Efficient Inference for Large Vision-Language Models: Bottlenecks, Techniques, and Prospects Jun Zhang, Yicheng Ji, Feiyang Ren, Yihang Li, Bowen Zeng, Zonghao Chen, Ke Chen, Lidan Shou, Gang Chen, Huan Li Abstract Large Vision-Language Models (LVLMs) enable sophisticated reasoning over images and videos, yet their inference is hindered by a systemic efficiency barrier known as visual token dominance . This overhead is driven by a multi-regime interplay between high-resolution feature extraction, quadratic attention scaling, and memory bandwidth constraints. We present a systematic taxonomy of efficiency techniques structured around the inference lifecycle, consisting of encoding , prefilling , and decoding . Unlike prior reviews focused on isolated optimizations, we analyze the end-to-end pipeline to reveal how upstream decisions dictate downstream bottlenecks, covering compute-bound visual encoding, the intensive prefilling of massive contexts, and the “visual memory wall” in bandwidth-bound decoding. By decoupling the efficiency landscape into the axes of shaping information density, managing long-context attention, and overcoming memory limits, this work provides a structured analysis of how isolated optimizations compose to navigate the trade-off between visual fidelity and system efficiency. The survey concludes by outlining four future frontiers supported by pilot empirical insights, including hybrid compression based on functional unit sensitivity, modality-aware decoding with relaxed verification, progressive state management for streaming continuity, and stage-disaggregated serving through hardware-algorithm co-design. Our literature repository is at https://github.com/SuDIS-ZJU/Efficien t-LVLMs-Inference . † † footnotemark: † † footnotetext: ✌ Equal contribution. ✉ Corresponding author. 1 Introduction Large Vision-Language Models (LVLMs) 160 ; 5 ; 164 have evolved from research artifacts into the infrastructure for complex multimodal reasoning. However, as these models scale to process fine-grained visual inputs and long-form video streams, they encounter a systemic efficiency barrier: visual token dominance 187 ; 143 ; 92 . Unlike text-only inputs, visual data yields orders of magnitude more tokens, pushing inference into a regime constrained not merely by compute cycles, but by the quadratic scaling of attention and the ‘‘visual memory wall’’ 1 1 1 For instance, a Qwen2.5-VL-72B processing 20 images already exceeds 40K tokens and 13 GB of cache, while a 5-second 720p video surpasses 50K tokens and 16 GB. ( 154 ; 78 ; 158 ) . The central thesis of this survey is that LVLM inference is not a monolithic workload, but a dynamic pipeline traversing three distinct hardware regimes: i) Encoding (specifically visual encoding) is compute-bound by high-resolution feature extraction; ii) Prefilling suffers from the quadratic complexity of massive visual contexts; and iii) Decoding hits the memory wall due to static, bandwidth-consuming Key-Value (KV) caches. Optimizing one stage in isolation often shifts the bottleneck elsewhere without improving end-to-end latency. Despite the surge in interest, the current literature remains fragmented. Prior reviews have predominantly focused on isolated verticals, such as token compression techniques ( 124 ) or efficient architectures for specific modalities ( 227 ; 202 ) 2 2 2 Section 8 provides a detailed related survey discussion. . These works, however, overlook the systemic interconnectivity of the inference pipeline. They lack a holistic view of how upstream decisions (e.g., encoder resolution) dictate downstream bottlenecks (e.g., decoding bandwidth), leaving a gap in understanding end-to-end efficiency. This survey bridges this gap by advancing a unified, stage-wise taxonomy of efficient LVLM inference. We decouple the efficiency landscape into three critical axes: shaping information density (encoding), managing long-context attention (prefilling), and overcoming memory bandwidth limits (decoding). This framework provides a structured lens to evaluate how isolated optimizations compose, helping researchers navigate the trade-off between visual fidelity and system efficiency. Figure 1: Three-stage pipeline for LVLM inference. Figure 2: Efficient encoding workflow: architectural optimization ( Section 3.1 , Section 3.2 ) and input reduction ( Section 3.3 , Section 3.4 , Section 3.5 ). 2 Preliminaries and Inference Dynamics LVLMs encounter unique efficiency bottlenecks compared to Large Language Models (LLMs), primarily due to the massive visual inputs. We formalize the canonical LVLM architecture ( Section 2.1 ) and analyze its inference dynamics through a “physics of computing” lens ( Section 2.2 ), mapping hardware bottlenecks to user-centric metrics to structure the survey ( Section 2.3 ). 2.1 The Canonical LVLM Architecture LVLMs typically adopt a three-stage pipeline ( Figure 2 ) that connects a vision encoder to an LLM. 3 3 3 Detailed component implementations and model taxonomy are provided in Appendix B . Given a multimodal tuple (\mathbf{V},\mathbf{T}) comprising raw visual input 4 4 4 F frames with (H\cdot W) resolution and RGB channels. \mathbf{V}\in\mathbb{R}^{F\times H\times W\times 3} and a text prompt \mathbf{T} of N_{t} tokens, the pipeline is formalized as: ① Visual Encoding. The encoder \mathcal{E}_{\phi} (with parameters \phi ) processes \mathbf{V} into patch embeddings \mathbf{X}_{v}\in\mathbb{R}^{N_{p}\times D_{v}} with N_{p} the output patch number 5 5 5 For single-frame inputs ( F=1 ), N_{p}=(H\cdot W)/P^{2} with patch size (P\times P) ; for videos with F\geq 2 , N_{p} depends on keyframe selection ( Section 3.3 ), adaptive resolution ( Section 3.4 ), and other compression strategies (e.g., frame pooling and Q-Former). and D_{v} the vision channel dimension. ② Modality Alignment. A modality adapter \mathcal{A}_{\theta} (with parameters \theta ) maps \mathbf{X}_{v} into the LLM’s latent space, yielding visual context \mathbf{H}_{v}=\mathcal{A}_{\theta}(\mathbf{X}_{v})\in\mathbb{R}^{N_{v}\times D_{\mathcal{L}}} with D_{\mathcal{L}} the LLM hidden dimension. The effective token count N_{v} varies by projection strategy (e.g., pooling), defined by the compression ratio r=N_{v}/N_{p} . ③ Autoregressive Generation. The LLM backbone \mathcal{L}_{\psi} (with parameters \psi ) concatenates visual and text embeddings into a joint context \mathbf{C}=[\mathbf{H}_{v};\mathbf{H}_{t}] (where \mathbf{H}_{t}\in\mathbb{R}^{N_{t}\times D_{\mathcal{L}}} represents the prompt of N_{t} textual tokens) to generate the output response \mathbf{Y}=(y_{1},\dots,y_{N_{o}}) of length N_{o} autoregressively: p(\mathbf{Y}\mid\mathbf{C})=\prod\nolimits_{k=1}^{N_{o}}p(y_{k}\mid\mathbf{C},y_{<k};\psi). (1) Here, a defining characteristic is the visual token dominance : the visual content ( N_{v}\approx 576 – 4,000+) significantly exceeds standard text prompts ( N_{v}\gg N_{t} ). This structural imbalance dictates the inference bottlenecks analyzed below. 2.2 The Physics of Inference Bottlenecks We model the end-to-end inference latency as: \tau_{\text{total}}=\tau_{\text{ENC}}+\tau_{\text{PFL}}+N_{o}\cdot\tau_{\text{DEC}}, (2) where the first two terms contribute to Time-to-First-Token (TTFT) at encoding and prefilling, respectively, and \tau_{\text{DEC}} determines Time-Per-Output-Token (TPOT) at decoding. To identify bottlenecks, we apply the Roofline model 6 6 6 A detailed Roofline analysis is provided in Appendix E . , which bounds performance based on the workload’s arithmetic intensity \mathcal{I} (FLOPs / Byte). A stage is compute-bound if \mathcal{I}\geq\pi_{\text{peak}}/\beta_{\text{mem}} , saturating the peak compute throughput \pi_{\text{peak}} ; otherwise, it is memory-bound , throttled by the memory bandwidth \beta_{\text{mem}} . As summarized in Table 1 , encoding is compute-bound with high arithmetic intensity from dense matrix operations; prefilling exhibits mixed behavior where both computation (quadratic attention) and memory I/O (KV cache materialization) can dominate; and decoding is strictly memory-bound due to the low arithmetic intensity of autoregressive token generation. Understanding these stage-specific bottlenecks is crucial for targeting optimization efforts. Table 1: Hardware bottlenecks, arithmetic intensity, and complexity dynamics across the three inference stages. Characteristic Encoding Stage Prefilling Stage Decoding Stage Primary Metric TTFT TTFT TPOT Bottleneck Compute-Bound Compute & Memory Bound Memory-Bound Arithmetic Intensity High ( \gg 1 ) Medium Low ( \ll 1 ) Complexity (FLOPs) \mathcal{O}(N_{p}\cdot D_{v}^{2}) \mathcal{O}((N_{v}+N_{t})^{2}\cdot D_{\mathcal{L}}) \mathcal{O}((N_{v}+N_{t})\cdot D_{\mathcal{L}}) LVLM Challenge High-res inputs ( N_{p}\uparrow ) surge FLOPs ( N_{v}\gg N_{t} ) causes quadratic spikes Static visual KV cache saturates VRAM Encoding Stage: Compute-Bound . This stage executes dense matrix multiplications over N_{p} patches, a high-intensity

Read the complete text