VLM in a flash: I/O-Efficient Sparsification of Vision-Language Model via Neuron Chunking

Kichang Yang, Seonjun Kim, Minjae Kim, Nairan Zhang, Chi Zhang, Youngki Lee · 2025

Read VLM in a flash: I/O-Efficient Sparsification of Vision-Language Model via Neuron Chunking by Kichang Yang, Seonjun Kim, Minjae Kim, Nairan Zhang, Chi Zhang, Youngki Lee (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-18692.txt.

Introduction

Kichang Yang and five co-authors speed up vision language models running on edge devices that load weights from flash storage. Keeping only active neurons reduces storage traffic, but standard sparsification picks neurons by size of activation alone and ignores how storage access patterns affect speed.

Their neuron chunking method selects whole contiguous groups of neurons, scoring each group by importance divided by an estimated access latency based on contiguity. Aligning the choice with storage behavior improves input output efficiency several fold on two Jetson edge boards, making large multimodal models more practical off the cloud.

Catalog record

CREATORKichang Yang, Seonjun Kim, Minjae Kim, Nairan Zhang, Chi Zhang, Youngki Lee
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE63 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-18692

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

VLM in a flash: I/O-Efficient Sparsification of Vision-Language Model via Neuron Chunking Kichang Yang, Seonjun Kim, Minjae Kim, Nairan Zhang, Chi Zhang, Youngki Lee Abstract Edge deployment of large Vision-Language Models (VLMs) increasingly relies on flash-based weight offloading, where activation sparsification is used to reduce I/O overhead. However, conventional sparsification remains model-centric, selecting neurons solely by activation magnitude and neglecting how access patterns influence flash performance. We present Neuron Chunking , an I/O-efficient sparsification strategy that operates on chunks —groups of contiguous neurons in memory—and couples neuron importance with storage access cost. The method models I/O latency through a lightweight abstraction of access contiguity and selects chunks with high utility, defined as neuron importance normalized by estimated latency. By aligning sparsification decisions with the underlying storage behavior, Neuron Chunking improves I/O efficiency by up to 4.65× and 5.76× on Jetson Orin Nano and Jetson AGX Orin, respectively. 1 Introduction Recent vision–language models (VLMs) demonstrate strong multimodal reasoning and real-time language interaction with visual scenes. Deploying these models on edge devices is becoming essential for applications such as augmented reality (AR) and autonomous robotics that require on-device inference for robustness to limited connectivity and privacy [ 4 , 53 ] . These systems must process video frames continuously without frame drops while maintaining interactive latency. The scalability of on-device inference is fundamentally constrained by memory capacity. Edge platforms provide far less memory than what modern VLMs require. Jetson Orin Nano, for example, offers only 8 GB of memory, while LLaVA-OneVision-7B [ 18 ] requires 16 GB (fp16) for weights alone. Recent systems [ 1 , 49 , 9 , 36 , 2 ] and inference engines [ 29 , 8 ] address this mismatch through weight offloading, which stores model parameters in external flash memory and loads them on demand during inference. This method allows large models to execute on small devices but introduces substantial I/O latency, which often dominates total inference time. Activation sparsification has been widely explored to mitigate this latency [ 1 , 49 , 44 ] . The approach loads only the weights corresponding to neurons with high importance (e.g., magnitude of activation value), reducing total data transfer and improving input adaptivity. Despite its effectiveness, existing sparsification remains model-centric. It selects channels solely based on activation importance while assuming that I/O latency scales linearly with data size. Flash storage does not follow this assumption; its latency depends strongly on access contiguity, and scattered reads severely degrade throughput. Earlier methods were less affected because they targeted highly sparse LLMs or GPU memory transfers, where locality plays a smaller role. Modern VLMs exhibit smoother activation distributions and lower sparsity (Figure 3 ), leading to fragmented reads and high I/O overhead (Figure 4(b) ). Figure 1 : Illustration of conventional sparsification vs. our approach . Existing methods select neurons solely based on activation importance, which often leads to scattered, irregular access patterns with poor I/O efficiency. In contrast, our method explicitly accounts for actual I/O latency, favoring contiguous chunks that achieve better importance–latency trade-offs. We propose neuron chunking , a sparsification framework that improves the I/O efficiency of flash-offloaded inference by coupling activation sparsity with storage access behavior. The key idea is to jointly optimize neuron importance and flash I/O latency by selecting contiguous channel groups that provide a better trade-off between accuracy and latency. Contiguous reads provide higher flash throughput, allowing moderately important neighboring channels to be loaded more efficiently than distant but highly important ones (Figure 1 ). This design forms compact chunks that enhance access locality, leading to significant performance gains during VLM inference. Efficient realization of this strategy requires capturing hardware I/O behavior in a form that the runtime can readily exploit. neuron chunking achieves this by reducing complex access patterns into a compact structural representation, termed the contiguity distribution . It summarizes how memory accesses cluster into contiguous groups (i.e., chunks) while omitting their exact spatial layout. This abstraction underlies two key components of our system. A chunk-based latency model profiles load latency for each chunk size and efficiently estimates the I/O latency of arbitrary access patterns from their contiguity distributions. A utility-guided chunk selection algorithm formulates neuron selection as a constrained optimization problem and iteratively selects chunks that maximize the importance–latency utility. Evaluation on Jetson Orin AGX and Nano using open-source VLMs and standard benchmarks shows consistent improvement in the accuracy–latency trade-off. At comparable accuracy, neuron chunking reduces I/O latency by an average of 2.19× on Nano and 2.89× on AGX, with maximum gains of 4.65× and 5.76×, respectively. Our contributions are summarized as follows. • We identify and characterize the hardware inefficiencies that arise when conventional activation sparsification techniques are applied to flash-offloaded VLM inference, revealing their mismatch with underlying storage access patterns. • We propose neuron chunking , a sparsification framework that enhances flash I/O efficiency by coupling activation sparsity with storage access patterns. It jointly optimizes neuron importance and flash latency by selecting contiguous channel groups that balance accuracy and latency. • We introduce the contiguity distribution as a compact representation of flash access behavior. Building on this abstraction, we develop a chunk-based latency model and a utility-guided chunk selection algorithm that together enable latency-aware sparsification by balancing model quality and I/O efficiency. • We evaluate neuron chunking on Jetson Orin AGX and Nano with open-source VLMs and standard benchmarks. Results show consistent improvement in the accuracy–latency trade-off, reducing I/O latency by up to 5.76× on AGX and 4.65× on Nano while maintaining comparable accuracy. 2 Background and Motivation 2.1 Overview of LLM/VLM Inference Pipeline An LLM inference pipeline consists of two stages: (i) the prefill stage, where the input prompt, consisting of multiple tokens, is processed to generate key-value (KV) cache, and (ii) the decoding stage, where the model generates tokens one at a time autoregressively using KV cache. When processing an online video stream with VLMs, an additional frame-appending stage is introduced between the prefill and decoding stages. In this stage, incoming video frames are processed sequentially as they arrive. VLMs integrate a vision encoder alongside the backbone LLM. Each frame is divided into patches and passed through the vision encoder, which converts them into a sequence of visual tokens. These tokens are then fed into the language model, augmenting the existing KV cache generated from the language prompt (see Appendix B.1 for details). 2.2 Model-side Observation: Smooth Activation Profiles in VLMs Figure 2 : Activation-magnitude plot for two workloads: (teal) a ReLU-based LLM in the decode phase and (magenta) a gated-activation-based VLM in the frame appending phase. VLM exhibits a smoother distribution, with much less variation between high and low activation values. Figure 3 : Read throughput as a function of block size and number of requests, profiled on Jetson AGX Orin with a Samsung 990 Pro SSD. Throughput quickly saturates and remains stable once the request count exceeds minimal thresholds. Modern VLMs exhibit smooth activation distributions as shown in Figure 3 . This smoothness is a general property of the architecture rather than a model-specific artifact. It arises from two key factors: (i) gated activation functions such as SwiGLU [ 35 ] and GeLU [ 11 ] produce continuous rather than sparse activation values, and (ii) averaging

Read the complete text