Technology solutions targeting the performance of gen-AI inference in resource constrained platforms

Joyjit Kundu, Joshua Klein, Aakash Patel, Dwaipayan Biswas · 2026

Read Technology solutions targeting the performance of gen-AI inference in resource constrained platforms by Joyjit Kundu, Joshua Klein, Aakash Patel, Dwaipayan Biswas (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-11128.txt.

Introduction

Kundu and three colleagues evaluate tech fixes for generative AI inference in constrained settings. Generative AI workloads, notably language model inference, raise memory pressure. Multimodal inputs and large-document uses incur long contexts, requiring caching of key and value states. Even low concurrent serving on mobiles adds memory pressure and care load. The authors test two emerging solutions using a roofline-based performance model. For large models, they study high bandwidth storage and its bandwidth and latency needs. For small models, they test a bonded global buffer memory chiplet.

Catalog record

CREATORJoyjit Kundu, Joshua Klein, Aakash Patel, Dwaipayan Biswas
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE18444
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.11128

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Technology solutions targeting the performance of gen-AI inference in resource constrained platforms Joyjit Kundu, Joshua Klein, Aakash Patel, Dwaipayan Biswas Abstract The rise of generative AI workloads, particularly language model inference, is intensifying on/off-chip memory pressure. Multimodal inputs such as video streams or images and downstream applications like Question Answering (QA) and analysis over large documents incur long context lengths, requiring caching of massive Key and Value states of the previous tokens. Even a low degree of concurrent inference serving on resource-constrained devices, like mobiles, can further add to memory capacity pressure and runtime memory management complexity. In this paper, we evaluate the performance implications of two emerging technology solutions to alleviate the memory pressure in terms of both capacity and bandwidth using a hierarchical roofline-based analytical performance model. For large models (e.g., 13B parameters) and context lengths, we investigate the performance implications of High Bandwidth Storage (HBS) and outline bandwidth/latency requirements to achieve an acceptable throughput for interactivity. For small models (e.g., 1B parameters), we evaluate the merit of a bonded global buffer memory chiplet and propose how to best utilize it. DNN Deep Neural Network CNN Convolutional Neural Network AI Artificial Intelligence LLM Large Language Model GPU Graphics Processing Unit SM Streaming Multiprocessor HBM High-Bandwidth Memory ML Machine Learning MLP Multi-Layer Perceptron † † aftertitle: I Introduction The advent of Large Language Models (LLMs) has been nothing short of a disruption across all industries [ 1 ] . On-device (e.g., smartphone) deployment of such models, is especially interesting since it paves the path towards secure intelligent personal assistants [ 2 , 3 ] . However, typical language models with unprecedented performance often have billions, if not trillions of parameters [ 4 , 5 ] , demanding extremely high compute and memory requirements to run. For instance, a Llama-7B model with single precision (FP16) needs around 14 GB of memory just to hold the weights, exceeding the capacity of standard DDRs in a personal device, like a smartphone. During inference a sequence of input tokens or prefill is processed to generate the next tokens one by one. Context length is defined by the sum of prefill and generation length. Each token carries its associated Query, Key, and Value (QKV) vectors; the Keys and Values of all previous tokens are reused to compute attention for every token generation. To avoid recalculating them, models cache K and V matrices (KV cache). The KV cache takes up additional memory space and grows linearly with the context length [ 6 ] . Multimodal inputs and concurrent inference serving further worsen the memory management issue by extending context length and adding runtime complexity [ 7 ] . The on-chip memory or standard SRAM is often not sufficient to hold the model and the KV cache. Fetching them from off-chip memory and generating only one token at a time results in ops/bytes \sim\mathcal{O}(1) . Now, due to the memory wall problem [ 8 ] data movement becomes the major bottleneck in the generation phase of inference – this nature of the workload is termed as memory-boundedness. Various algorithmic solutions have been devised to tackle this problem, primarily capitalizing on the concept of dynamic memory management techniques that enable larger models and longer contexts with acceptable throughput. Some of the ideas include, attention score-based KV cache eviction with large context lengths on edge platforms yielding up to 2x speedup [ 7 ] , smart memory management techniques like block access from Flash and sparsity-aware loading achieving 3–20x speedup depending on the baseline and model type for iPhone SoCs [ 9 ] . Utilization of runtime optimizations such as concurrent inference serving case, workload-aware scheduling, disaggregated prefill & decode, and heterogeneous-Soc aware batching can deliver an additional 1.6–4.8x improvement [ 3 ] . Such algorithmic and middleware-level improvements can always be augmented with technology solutions. Large memory requirements of model weights and KV cache typically rely on SSD/Flash storage. However, they are orders of magnitude slower (in bandwidth) than DDR [ 9 , 10 ] . To fill this gap, high bandwidth storage (HBS) promises bandwidth comparable with DDR with orders of magnitude more capacity at the cost of latency (microseconds) [ 11 ] . An additional augmentation to storage (with or without HBS) can be a SRAM based hybrid-bonded buffer memory chiplet that uses a custom interface directly to the logic die-resident NPU. This proposal draws inspiration from compute-near/in-memory works that usually assume the logic die to be close to memory [ 12 ] . In this paper, we explore the performance implications of the above two technology propositions. Our main contributions are the following: • Within an analytical framework, we show a methodology of evaluating the performance of emerging technologies for gen-AI inference and further offer insights into the key factors behind their projected performance, thus, facilitating System-Technology co-optimization (STCO). • For large models, we explore bandwidth-latency requirements of High Bandwidth Storage (HBS) memory to achieve a throughput threshold for various context windows. • For small models, we investigate the performance of a SRAM based chiplet solution for Q, K, V matrices and weights. II Methodology We adapt an existing analytical performance modeling framework validated across a number of accelerators, LLM models, and mapping choices both for training and inference to incorporate the additional memory subsystem, namely, HBS and bonded memory chiplet [ 13 , 14 ] . Most of the other available performance models typically consider roofline analysis with only one memory level [ 15 ] and thus, often fail to capture intricacies of complex memory hierarchy. The core of the performance model used here relies on a hardware-aware hierarchical roofline analysis. It derives the compute/task-graph of the language models involving kernels like, GEMMs/GEMVs (General Matrix-Matrix or Matrix-Vector multiplications) and ones with element-wise operations. The performance is dominated by the following GEMM kernels: {X .{\rm W_{Q/K/V}}={\rm Q/K/V} }, {Q. {\rm K^{T}} = R}, {Softmax(R).V = Z}, {Z. {\rm W_{O}} = O} (projection), {O. {\rm W_{MLP1}}={\rm O_{MLP1}} }, { {\rm O_{MLP1}} . {\rm W_{MLP2}}={\rm O_{MLP2}} }, where W stands for the weight matrices and X for input. Respecting sequential dependencies, each kernel is mapped onto the given system architecture. GEMM/GEMV kernels are parallelized via tiling distributed across the processing elements, with tile sizes determined by cache and memory capacities. The memory access pattern of the above kernels are quite regular and thus, predictable analytically. Kernel latency is estimated by searching over candidate tiling strategies at each memory hierarchy. For a given level, the optimal tiling defines the memory access count, from which the arithmetic intensity (AI)—the ratio of FLOPs to bytes transferred—is computed. From the hardware perspective, each memory level has an inflection point defined by the ratio of the compute throughput to the memory bandwidth. If a kernel’s AI is below this point, it is memory-bound; otherwise, it is compute-bound. The kernel execution time is given by the critical path across all memory levels: {\rm time}=max\{\frac{{\rm total~FLOPs}}{{\rm compute~throughput}},\frac{{\rm data~traffic~volume}}{\rm memory~throughput}\} (1) We consider the following base memory hierarchy: NPU scratchpad – L2 – DDR where each memory level is characterized by its capacity, latency, and bandwidth [ 16 ] . III Experiment Design & Results Fig. 1 : TPS as a function of HBS

Read the complete text