Cost-Efficient Multimodal LLM Inference via Cross-Tier GPU Heterogeneity

Donglin Yu · 2026

Read Cost-Efficient Multimodal LLM Inference via Cross-Tier GPU Heterogeneity by Donglin Yu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-12707.txt.

Introduction

Donglin Yu studies how to serve multimodal language models cheaply by splitting work across GPU tiers. Vision encoding is compute-bound, while text generation is bound by memory bandwidth. He shows the split point between vision and language parts cuts data transfer the most. That split lets cheap PCIe links replace costly high-bandwidth ones. His HeteroServe runtime schedules the two phases across mixed hardware. Under a fixed budget, the mixed cluster gave far more tokens per dollar than a uniform one.

Catalog record

CREATORDonglin Yu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE45486
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.12707

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Cost-Efficient Multimodal LLM Inference via Cross-Tier GPU Heterogeneity Donglin Yu Abstract Multimodal large language model (MLLM) inference splits into two phases with opposing hardware demands: vision encoding is compute-bound, while language generation is memory-bandwidth-bound. We show that under standard transformer KV caching, the modality boundary (between vision encoder and language model) minimizes cross-device transfer among all partition points that preserve standard stage-based execution. Partitioning here reduces transfer complexity from O ⁡ ( L ⋅ s ctx ) O(L\cdot s_{\text{ctx}}) bytes (GB-scale KV caches under stage-level disaggregation) to O ⁡ ( N v ⋅ d ) O(N_{v}\cdot d) bytes (MB-scale embeddings), an O ⁡ ( L ) O(L) reduction where L L is the transformer depth. The result holds across attention mechanisms (MHA/GQA), dynamic vision resolutions, and model scales, and the advantage grows as models deepen. A direct implication is that existing stage-level disaggregation systems are constrained to high-bandwidth interconnects (e.g., NVLink), whereas modality-level disaggregation enables cross-tier heterogeneous serving over commodity PCIe. A closed-form cost model shows that heterogeneous deployment is cost-optimal under phase-separable workloads (predicts 31.4% savings; observed 40.6%). We build HeteroServe, a phase-aware runtime with modality-level partitioning and cross-tier scheduling, and evaluate it on LLaVA-1.5-7B and Qwen2.5-VL against vLLM v0.3.0. On identical 4 × \times A100 hardware, engine optimizations raise throughput by up to 54%. Under a fixed budget, a heterogeneous cluster ($38k) improves Tokens/$ by 37% over a homogeneous baseline ($64k) without degrading latency. 1 Introduction Multimodal large language model (MLLM) inference exhibits an architectural mismatch: vision encoding and language decoding have different hardware optimality regimes. Vision encoding is compute-bound, saturating FP16 tensor cores with negligible memory bandwidth demand. Language decoding is memory-bandwidth-bound, streaming model weights and KV caches from HBM with minimal arithmetic intensity. These two phases stress completely different GPU resources, yet all deployed MLLM serving systems execute both on homogeneous datacenter hardware, paying a well-documented “HBM tax” ( Zhang et al. 2025 ; Mo et al. 2025 ) : the compute-bound phase wastes expensive high-bandwidth memory, while the bandwidth-bound phase underutilizes tensor cores. In practice, providers must overpay for HBM even when the current workload is encoder-heavy, because the entire pipeline is locked to a single GPU tier. Existing disaggregation systems attempt to address inference efficiency by partitioning at pipeline stage boundaries , i.e., separating prefill from decode (EPD ( Singh et al. 2025 ) , Cauchy ( Zhang et al. 2025 ) ). However, stage-level partition incurs KV cache transfer cost proportional to model depth: O(L\cdot s_{\text{ctx}}) bytes (hundreds of megabytes to gigabytes per request). This GB-scale transfer limits scalability across heterogeneous GPU tiers, because it demands NVLink or InfiniBand and confines all prior disaggregated serving to homogeneous datacenter environments. Intra-node scheduling (SpaceServe ( Li et al. 2025 ) , UnifiedServe ( Zhao et al. 2025 ) ) improves utilization through kernel co-location but does not change the cross-device communication structure and therefore cannot unlock cross-tier deployment. The bottleneck is not disaggregation itself but where the graph is cut . No existing system exploits the modality boundary for heterogeneous serving. Why has stage-level partitioning gone unquestioned? We argue that it reflects a language-only assumption : in text-only LLMs, the KV cache is the dominant intermediate state and stage boundaries are the only meaningful partition points. Multimodal inference violates this assumption: the vision encoder produces a compact embedding of size O(N_{v}\cdot d) (MB-scale), independent of transformer depth L . Yet no prior system has exploited this separation for cross-device serving. We formalize this observation as Theorem 1 : under standard transformer KV caching (i.e., without activation recomputation or KV offloading), the modality boundary (between vision encoder and language decoder) minimizes cross-device transfer complexity among partition points that preserve standard stage-based execution semantics. The transfer ratio between stage-level and modality-level disaggregation scales as \Theta(L) , corresponding to 12\times – 196\times across current MLLMs. The result holds independent of hidden dimension, attention mechanism (MHA or GQA), vision token count, and model scale, and the advantage grows with model depth. In practice, this reduces cross-device transfer from GB-scale to MB-scale, making cross-tier heterogeneous serving over commodity PCIe feasible. Consumer GPUs (RTX 4090, $3k, 330 TFLOPS) can handle vision encoding while datacenter GPUs (A100, $16k, 2 TB/s HBM) handle language decoding, a 3\times FLOPs-per-dollar advantage that is inaccessible under stage-level disaggregation. A closed-form cost model further shows that heterogeneous deployment is cost-optimal under phase-separable workloads (predicts 31.4% savings; we observe 40.6%). We build HeteroServe, a phase-aware runtime with embedding-only transfer, cross-type work stealing for idle-capacity recovery, and standard engine optimizations. We make the following contributions: • Transfer Optimality Analysis (§ 2 –§ 4.2 ). We characterize the phase separation in MLLM inference (§ 2 ) and prove (Theorem 1 ) that, under standard KV caching, the modality boundary minimizes cross-device transfer under standard stage-based execution, reducing communication by O(L) compared to stage-level disaggregation, with empirical ratios of 12\times – 196\times across five architectures. We further derive a closed-form cost model (§ 4.3 ) showing that heterogeneous deployment is cost-optimal under phase-separable workloads (predicts 31.4% savings; we observe 40.6%). • System Validation: HeteroServe (§ 5 ). We build HeteroServe, a phase-aware runtime that validates the analysis on real hardware, with embedding-only transfer over PCIe, cross-type work stealing for utilization recovery, and CUDA-Graph-accelerated decoding. • Empirical Validation (§ 6 ). On LLaVA-1.5-7B (MHA) and Qwen2.5-VL (GQA, tensor-parallel) with vLLM v0.3.0 baseline: on identical 4 \times A100 hardware, engine optimizations raise throughput by up to 54% (§ 6.6 ); under a fixed budget, modality-level heterogeneous deployment ($38k) improves Tokens/$ by 37% over a homogeneous baseline ($64k) without degrading latency. Engine optimizations (CUDA Graph, packed prefill, lazy KV allocation) serve as implementation hygiene to avoid confounding the architectural evaluation; they are not claimed as contributions. 2 Multimodal Inference Phase Characterization We begin by characterizing the computational phases of MLLM inference and their hardware implications. The end-to-end latency of a single multimodal request decomposes as T_{\text{e2e}}=T_{\text{vision}}+T_{\text{xfer}}+T_{\text{prefill}}+T_{\text{decode}} . Each phase has a distinct resource bottleneck. Vision encoding ( T_{\text{vision}} ) is compute-bound : on an RTX 4090, batched encoding achieves > 85% FP16 TFLOPS utilization ( {\sim}280 of 330 TFLOPS) while consuming < 5% of memory bandwidth. The compute cost is fixed per image and independent of language model depth L . Language decoding ( T_{\text{decode}} ) is memory-bandwidth-bound : autoregressive generation streams weights and KV caches from HBM with minimal arithmetic reuse, achieving > 80% HBM utilization on A100 ( {\sim}1.6 of 2 TB/s) while using < 10% of FP16 compute. Language prefill ( T_{\text{prefill}} ) is compute-bound but operates on language model weights. These bottlenecks map to orthogonal hardware optimality regimes . Vision encoding maximizes performance on high-TFLOPS, low-cost GPUs: an RTX 4090 ($3k, 330 TFLOPS) matches an A100 ($16k, 312 TFLOPS) at 3.3\times better FLOPs/$. Language decoding requires high-bandwidth GPUs: A100 provides 2 TB/s HBM with 80 GB; RTX 4090 has only 1 TB/s with 24 GB. Homogeneous deployments cannot resolve this mismatch, motivating a formal analysis of where to partition the pipeline. 3 Related Work We organize prior work along two axes that jointly define the design space of disaggregated multimodal serving: (1) partition granularity , i.e., where the inference graph is cut (stage boundary, modality boundary, or kernel co-location); and

Read the complete text