Phase Matters: Characterizing Heterogeneous Vision-Language Inference on a Mobile SoC CCS: Hardware Hardware accelerators CCS: Computer systems organization Embedded systems CCS: Computer systems organization Real-time systems CCS: Computing methodologies Computer vision Conference: arXiv preprint; June 22, 2026;

Aryama V Murthy, Yashas N Kotre, Prathmesh Sharma, Pragya Mishra, Sanjith Ganapathi, Priyesh Shukla · 2026

Read Phase Matters: Characterizing Heterogeneous Vision-Language Inference on a Mobile SoC CCS: Hardware Hardware accelerators CCS: Computer systems organization Embedded systems CCS: Computer systems organization Real-time systems CCS: Computing methodologies Computer vision Conference: arXiv preprint; June 22, 2026; by Aryama V Murthy, Yashas N Kotre, Prathmesh Sharma, Pragya Mishra, Sanjith Ganapathi, Priyesh Shukla (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-27906.txt.

Introduction

Phone-class chips can now run vision-language models on device. Developers lack phase-level guidance for mapping the pipeline across CPU and neural processing unit, though. Aryama Murthy and five colleagues characterise this inference in 2026. Their testbed is the Qualcomm Snapdragon 8 Elite chip. FastVLM-0.5B serves as the end-to-end case study. Encoder measurements span four architecture families.

Phase matters, the results show. Neural-unit execution speeds the prefill phase by about one point six times. The decode phase gains only about one point one eight times. Vision encoders gain far more, running twenty to forty-five times faster than on CPU. Steady-state temperature drops by about ten and a half degrees, and energy use falls by about two and a half times, avoiding throttling. A four-step graph rewrite brings previously unsupported encoders to the neural path. One reaches speedups of up to twenty-two times.

Catalog record

CREATORAryama V Murthy, Yashas N Kotre, Prathmesh Sharma, Pragya Mishra, Sanjith Ganapathi, Priyesh Shukla
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE23 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.27906

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Phase Matters: Characterizing Heterogeneous Vision-Language Inference on a Mobile SoC CCS: Hardware Hardware accelerators CCS: Computer systems organization Embedded systems CCS: Computer systems organization Real-time systems CCS: Computing methodologies Computer vision Conference: arXiv preprint; June 22, 2026; Aryama V Murthy, Yashas N Kotre, Prathmesh Sharma, Pragya Mishra, Sanjith Ganapathi, Priyesh Shukla Abstract . Recent phone-class mobile SoCs expose practical NPU execution paths for on-device vision-language model (VLM) inference, but developers still lack phase-level guidance for mapping VLM pipelines across heterogeneous backends. We present a hardware-in-the-loop characterization of VLM inference on the Qualcomm SM8750 (Snapdragon 8 Elite), covering phase throughput, cache-state effects, 100-run thermal stability, energy, heterogeneous CPU/NPU pipeline configurations, and visual-token-budget sensitivity. Using FastVLM-0.5B as an end-to-end case study, together with encoder-only measurements across four architecture families, we show that phase matters : NPU execution is highly phase-dependent, delivering 1.64 × \times speedup for prefill but only 1.18 × \times for decode, while vision encoders achieve 20–45 × \times speedups over CPU. These gains translate into 10.47 ∘ C lower steady-state temperature and 2.52 × \times lower energy, avoiding thermal throttling in always-on settings. Finally, we show that a four-step graph rewrite enables previously unsupported encoders, such as Phi-3.5-V, to reach the QNN path with up to 22 × \times speedup, providing a practical porting recipe for mobile VLM deployment. 1. Introduction Vision-language models (VLMs) are crossing a critical threshold: devices already in consumers’ hands—with billion-parameter multimodal models—can now run VLM inference entirely on-device, with no cloud round-trip. Wearables, AR overlays, robotics, and privacy-sensitive pipelines all demand exactly this capability ( fastvlm ; mobilevlm ; litevlm ) . The Qualcomm SM8750 (Snapdragon 8 Elite) exposes a stable Hexagon NPU path for full multimodal models via the QNN SDK ( qualcomm_qnn ; qualcomm_genai ) , making it the right deployment platform to ask: what does NPU acceleration concretely deliver for each phase of VLM inference, and how should practitioners configure heterogeneous pipelines? This question is non-trivial: VLM inference decomposes into three stages with fundamentally different hardware affinities ( llmnpu ) . Prefill saturates NPU tensor engines with batched GEMMs. Decode is bandwidth-bound and shape-dynamic, favouring CPU cache locality ( heteroinfer ; qualcomm_genai ) . Vision encoding is static-shape and compute-dense—the NPU’s clearest sweet spot, yet the least systematically studied. Treating these phases uniformly leads to measurable waste in latency, energy, and thermal budget. Prior systems work on heterogeneous mobile inference targets text-only LLMs ( llmnpu ; heteroinfer ) ; the vision encoding phase, its thermal behavior, and practical porting paths for unsupported encoders are not covered. We address this gap with a VLM-specific, hardware-in-the-loop study on SM8750, contributing phase-aware deployment guidance and a practical QNN porting workflow for production phone NPUs. Contributions: • Phase map: 1.64 \times NPU prefill, 1.18 \times decode, 6.41 \times cold-start overhead (Sec. 3 ). • Thermal suite: 10.47 ∘ C cooler, 2.52 \times lower energy, zero throttling on NPU over 100 runs (Sec. 4 ). • Pipeline map: Five CPU \leftrightarrow NPU configurations; W4A8 hybrid at 97.5 tok/s (Sec. 5 ). • Latency knob: Token-budget sweep 0.36–0.85 s TTFS, no recompilation (Sec. 5 ). • Encoder scaling: Four families \times CPU/GPU/NPU, 20–45 \times CPU-to-NPU speedups (Sec. 6 ). • Porting methodology: Four-step rewrite, 22 \times on Phi-3.5-V (Sec. 7 ). 2. Platform and Experimental Setup 2.1. Hardware: Qualcomm SM8750 Snapdragon 8 Elite Our target is the Qualcomm SM8750 ( snapdragon_brief ) , a 3 nm SoC with a 64-bit Oryon CPU (4.32 GHz), LPDDR5x at 5300 MHz, Adreno GPU, and a Hexagon NPU with scalar/vector/tensor engines and VTCM scratch ( qualcomm_genai ) . W4A8 INT4-weight packing halves VTCM bandwidth demand vs. INT8 ( w4a8 ; awq ) . Static-shape NPU paths use QNN SDK ( qualcomm_qnn ) via ONNX; dynamic delegation uses LiteRT ( litert ) . 2.2. VLM Phase Decomposition Following the roofline analysis framework ( williams2009roofline ) , VLM inference separates into three hardware-distinct phases (Table 1 ). Prefill has high operational intensity I=\text{FLOPs}/\text{Byte} due to batched GEMM and ViT attention over full image and prompt—NPU systolic engines achieve peak utilization here ( qualcomm_genai ; llmnpu ) . Decode processes one token per step while reading a growing KV cache: per-layer traffic scales as 2LDb bytes, making it bandwidth-bound. The NPU delivers 1.18 \times decode throughput; heterogeneous pipelines assign decode to CPU to maximize energy efficiency and preserve NPU capacity for the compute-intensive prefill phase ( heteroinfer ) . Vision encoding is entirely static-shape and compute-dense, yielding the highest NPU efficiency of any VLM stage ( fastvlm ; clip ; siglip ) . Table 1. VLM inference phase decomposition and backend affinity Phase Operators Intensity Backend Vision encode Conv, ViT-attn Very high (static) NPU Prefill GEMM, attn High (compute) NPU Decode 1-tok attn + KV Low (BW-bound) CPU † † NPU gains 1.18 \times on decode (Table 2 ); CPU preferred for energy efficiency in heterogeneous pipelines. 2.3. Models, Runtimes, and Protocol We use FastVLM-0.5B ( fastvlm ) as the primary model: a 0.5B-parameter VLM built on Qwen-0.5B ( qwen ) with a FastViT encoder that applies aggressive token compression for mobile execution. For encoder scaling, we additionally benchmark ViT-B/16 ( dosovitskiy2020vit ) , Phi-3.5-V ( phi3v ) , NanoVLM ( nanovlm ) , and MobileNetV5 Gemma3n ( gemma3n ) . The CPU path uses LiteRT ( litert ) with XNNPACK FP16. The NPU path uses LiteRT with Qualcomm AI Engine delegate ( qualcomm_qnn ) . The W4A8 quantized path uses QAIRT ( qualcomm_qnn ) INT4 weight, INT8 activation containerization for the LLM decode stage. All measurements use a fixed 64-tok text prompt, 336 \times 336 image, 128-tok output; 30 runs (100 for thermal); mean \pm std reported. Device: AC power, 23 ∘ C, CPU governor schedutil , NPU at Sustained Performance mode. Temperature: 1 s-sampled from NPU/CPU cluster thermal zones ( /sys/class/thermal/ ); energy: fuel-gauge I \times V at 100 ms granularity, capturing total SoC inference draw. Cache states ( nnv12 ; pask ) — S1 (cold-start): process killed, drop_caches=3 , QNN context cache deleted; first call reloads model and recompiles QNN graph (Table 4 cold). S2 (cold-cache): process restarted; model hot in page cache; QNN context cache intact; executor init on first call. S3 (warm): process alive, QNN context in VTCM, dispatch only. Table 2 reports S2 executor-init overhead; Table 4 reports full S1 end-to-end pipeline latency—each measuring a distinct, well-defined deployment scenario. 3. End-to-End NPU vs. CPU Throughput 3.1. Phase-Level Results Table 2 reports throughput under warm-cache conditions. NPU delivers 1.64 \times prefill throughput (197.43 vs. 120.64 tok/s) and 1.18 \times decode throughput (113.06 vs. 95.49 tok/s). The asymmetry reflects fundamentally different operational intensity: prefill’s large batched GEMMs saturate the Hexagon tensor engine, while decode’s per-step KV-cache reads are memory-bandwidth-limited, with VTCM saturation increasing at longer contexts ( qualcomm_genai ) . Table 2. FastVLM-0.5B phase throughput: CPU vs. NPU, SM8750, warm-cache, 64-tok prompt + 336 \times 336 image, 128-tok output; mean \pm std, 30 runs. † Executor init (S2): first-token overhead with model loaded and QNN context cache intact; full pipeline latencies across all cache states in Table 4 . Phase Metric CPU NPU Gain Prefill tok/s 120.6{\pm}4.2 197.4{\pm}1.3 1.64 \times Decode tok/s 95.5{\pm}0.2 113.1{\pm}0.4 1.18 \times Exec. init † ms

Read the complete text