LLMs on a Budget? Say HOLA
Zohaib Hasan Siddiqui, Jiechao Gao, Ebad Shabbir, Mohammad Anas Azeez, Rafiq Ali, Gautam Siddharth Kashyap, Usman Naseem · 2025
Read LLMs on a Budget? Say HOLA by Zohaib Hasan Siddiqui, Jiechao Gao, Ebad Shabbir, Mohammad Anas Azeez, Rafiq Ali, Gautam Siddharth Kashyap, Usman Naseem (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-18952.txt.
Introduction
Zohaib Hasan Siddiqui and co-authors present a framework for running large language models on small edge devices, in a 2025 paper. Edge hardware, such as a home or clinic device, lacks the memory and computing power that such models demand. Standard fixes like compression give only partial gains and often cost speed or accuracy.
Their framework, called HOLA, works on two levels. Inside, a layered speculative decoding method speeds generation without quality loss. Outside, a retrieval step adjusts its complexity to what the context needs. A third part blends pruning with weight compression. Scores on maths and reasoning benchmarks rose, while delay and memory fell on a Jetson Nano test device. The authors present the system as scalable and ready for production use in health, education, and embedded settings.
Catalog record
| CREATOR | Zohaib Hasan Siddiqui, Jiechao Gao, Ebad Shabbir, Mohammad Anas Azeez, Rafiq Ali, Gautam Siddharth Kashyap, Usman Naseem |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 27 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-18952
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
LLMs on a Budget? Say HOLA Zohaib Hasan Siddiqui, Jiechao Gao, Ebad Shabbir, Mohammad Anas Azeez, Rafiq Ali, Gautam Siddharth Kashyap, Usman Naseem Abstract Running Large Language Models (LLMs) on edge devices is constrained by high compute and memory demands—posing a barrier for real-time applications in sectors like healthcare, education, and embedded systems. Current solutions such as quantization, pruning, and retrieval-augmented generation (RAG) offer only partial optimizations and often compromise on speed or accuracy. We introduce HOLA , an end-to-end optimization framework for efficient LLM deployment. Internally, it leverages Hierarchical Speculative Decoding (HSD) for faster inference without quality loss. Externally, AdaComp-RAG adjusts retrieval complexity based on context needs. Together with Lo-Bi, which blends structured pruning (LoRA) and quantization, HOLA delivers significant gains: +17.6% EMA on GSM8K, +10.5% MCA on ARC, and reduced latency and memory on edge devices like Jetson Nano—proving both scalable and production-ready. 1 Introduction Large Language Models (LLMs) have transformed NLP applications—from question answering Ehtesham et al. 2025 and code generation Zhong and Wang 2024 to summarization Ghosh et al. 2024 and conversational agents Zenimoto 2024 . However, their deployment on edge and low-resource environments remains constrained by high compute and memory demands Dutta et al. 2025 , limiting real-world impact in domains like healthcare Lysandrou et al. 2023 , education Bayarri-Planas et al. 2025 , and personalized assistants Tack et al. 2024 . While techniques such as quantization Tack et al. 2024 , pruning Chimoto et al. 2024 , and retrieval-augmented generation (RAG) Parekh et al. 2024 offer partial relief, they often sacrifice accuracy, speed, or generality. Moreover, most approaches optimize isolated components—either internal inference Liu et al. 2023 or external retrieval Zheng et al. 2023 —without addressing the full pipeline. We introduce HOLA (Hierarchical Optimized Language Augmentation), a unified framework targeting both internal and external efficiency. It integrates (1) Hierarchical Speculative Decoding (HSD) for faster, accurate inference; (2) AdaComp-RAG, an adaptive retrieval system based on context relevance; and (3) Lo-Bi Optimization, combining LoRA-based structured pruning with mixed-precision quantization. 2 Related Work Deploying LLMs in constrained environments has driven advances in decoding acceleration, retrieval optimization, and model compression. Techniques like Speculative Decoding Sun et al. 2024 and Medusa Cai et al. 2024 use draft-and-verify schemes to speed up inference, though they often require extra verifier networks and struggle with long sequences. Adaptive retrieval methods Labruna et al. 2024 selectively route queries to improve efficiency but introduce added system complexity and retraining needs. Model compression remains central to edge deployment. Quantization approaches like GPTQ Bumgardner et al. 2025 and Bi-LLM Weng et al. 2025 reduce precision for memory savings, but aggressive quantization can impact accuracy. Structured pruning using LoRA Hu et al. and LoRA-Pruner Zhang et al. 2024b compress model weights with minimal retraining. While a few efforts combine pruning and quantization Ali et al. 2025 , most lack holistic, hardware-aware integration across the inference stack. 3 Methodology Our proposed framework, HOLA , is composed of three synergistic modules—HSD accelerates autoregressive generation via entropy-aware verification, AdaComp-RAG adaptively modulates retrieval granularity, and Lo-Bi Optimization ensures compact and resource-aware model deployment. The broader process of the HOLA is illustrated in Algorithm 1 . Algorithm 1 : HOLA 1: Input context \mathbf{x} 2: Generated output sequence \mathbf{y} 3: Step 1: HSD 4: Generate draft \hat{\mathbf{y}}=f_{\text{draft}}(\mathbf{x}) 5: for t=1 to T do 6: Compute token entropy H(\hat{y}_{t}) 7: if H(\hat{y}_{t})<\tau then \triangleright Fast-path 8: y_{t}\leftarrow\hat{y}_{t} 9: else \triangleright Fallback to verifier 10: y_{t}\leftarrow f_{\text{ver}}(\mathbf{x},y_{<t}) 11: end if 12: end for 13: Step 2: AdaComp-RAG 14: Measure query complexity C(\mathbf{q})=\|\nabla_{\mathbf{q}}\mathcal{L}\|_{2} 15: if C(\mathbf{q})\geq\delta then 16: Retrieve top- k documents and form \mathbf{x}^{\prime}=[\mathbf{x};\mathbf{d}_{1},\dots,\mathbf{d}_{k}] 17: else 18: \mathbf{x}^{\prime}\leftarrow\mathbf{x} 19: end if 20: Apply compositional attention on \mathbf{x}^{\prime} 21: Step 3: Lo-Bi Model Optimization 22: Compute low-rank update: \Delta\mathbf{W}=\mathbf{A}\mathbf{B} 23: Update weights: \mathbf{W}^{\prime}=\mathbf{W}+\Delta\mathbf{W} 24: for each subblock i in \mathbf{W}^{\prime} do 25: Select bit-precision p_{i}=\arg\min_{p\in\{4,8,16\}}\mathbb{E}[\|f_{\text{orig}}-f_{\text{quant}}^{(p)}\|_{2}] 26: Quantize subblock using p_{i} bits 27: end for 28: return Optimized output sequence \mathbf{y} 3.1 HSD Module To mitigate the inherent latency of sequential token-by-token generation in autoregressive transformers, we introduce a HSD mechanism. Let the target sequence be \mathbf{y}=(y_{1},y_{2},\ldots,y_{T}) conditioned on input context \mathbf{x} . Conventional decoding computes y_{t}\sim p(y_{t}\mid\mathbf{x},y_{<t}) in series, which becomes computationally expensive for long outputs. Instead, HSD generates a draft sequence \hat{\mathbf{y}} using a lightweight generator f_{\text{draft}} and verifies each token with a higher-fidelity verifier model f_{\text{ver}} . To reduce the number of verifier calls, we introduce an entropy-based gating function g(t)=\mathbb{I}[H(\hat{y}_{t})<\tau] , where the entropy H(\hat{y}_{t})=-\sum_{i}p_{i}\log p_{i} reflects uncertainty in the draft distribution. Tokens with entropy below a threshold \tau are directly accepted, and the final output sequence is constructed via Equation (1). y_{t}=\begin{cases}\hat{y}_{t}&\text{if }g(t)=1\\ f_{\text{ver}}(\mathbf{x},\hat{y}_{<t})&\text{otherwise}\end{cases} This reduces the number of verifier invocations from O(T) to O(k) , where k\ll T , enabling up to 2–3 \times decoding speedup without significant degradation in output quality. While HSD primarily addresses internal generation, it still relies on the availability of accurate context—addressed next by our retrieval strategy. 3.2 AdaComp-RAG Module To selectively augment queries with external knowledge, we propose AdaComp-RAG, an adaptive retrieval mechanism that modulates retrieval depth based on query uncertainty. Given a query embedding \mathbf{q}\in\mathbb{R}^{d} , we compute its retrieval complexity as the gradient norm of the loss function: C(\mathbf{q})=\left\|\nabla_{\mathbf{q}}\mathcal{L}\right\|_{2} , where \mathcal{L} is the autoregressive language modeling loss. If C(\mathbf{q})\geq\delta , we invoke a dense retrieval module over a document collection \{\mathbf{d}_{1},\dots,\mathbf{d}_{k}\} , resulting in an augmented input \mathbf{x}^{\prime}=[\mathbf{x};\mathbf{d}_{1};\dots;\mathbf{d}_{k}] . Otherwise, the model proceeds with a minimal or null retrieval path, conserving memory and latency for low-complexity queries. To integrate retrieved and original content, we apply a compositional attention mechanism over both contexts: \mathbf{A}=\text{softmax}\left(\frac{\mathbf{Q}(\mathbf{K}_{\text{doc}}\oplus\mathbf{K}_{\text{input}})^{\top}}{\sqrt{d}}\right) , where \mathbf{Q} , \mathbf{K}_{\text{doc}} , and \mathbf{K}_{\text{input}} are the query and key matrices for retrieved and input sequences respectively, and \oplus denotes concatenation. 3.3 Lo-Bi Optimization Module To ensure deployment feasibility on constrained hardware, we incorporate a dual-stage compression strategy termed Lo-Bi Optimization, combining LoRA with sensitivity-guided mixed-precision quantization. First, given a weight matrix \mathbf{W}\in\mathbb{R}^{d\times d} , LoRA decomposes the update: \Delta\mathbf{W}=\mathbf{A}\mathbf{B},\quad\mathbf{A}\in\mathbb{R}^{d\times r},\quad\mathbf{B}\in\mathbb{R}^{r\times d},\quad r\ll d and applies it as an additive delta to the pretrained weights: \mathbf{W}^{\prime}=\mathbf{W}+\Delta\mathbf{W} . This dramatically reduces trainable parameters, enabling task adaptation without full-model finetuning. Next, to reduce inference-time memory, we apply a mixed-precision quantizer Q(\cdot) , mapping subblocks of \mathbf{W}^{\prime} to 4-, 8-, or 16-bit precision. The optimal precision level p_{i} for each subblock: p_{i}=\arg\min_{p\in\mathcal{P}}\;\mathbb{E}_{x\sim\mathcal{D}}\left[\left\|f_{\text{orig}}(x)-f_{\text{quant}}^{(p)}(x)\right\|_{2}\right] , where \mathcal{P}=\{4,8,16\} . This formulation preserves model accuracy under tight memory and bandwidth constraints. Note: Together, these modules comprise a fully-integrated pipeline: HSD accelerates token generation, AdaComp-RAG modulates augmentation on a per-query basis, and Lo-Bi Optimization ensures that all components operate efficiently under limited computational budgets. 4 Experimental Setup We conducted experiments using a wide spectrum of LLMs—our selection includes compact, instruction-optimized, and general-purpose transformers. Among lightweight contenders, we evaluated Llama-3.2-3B 1 1 1 https://huggingface.co/meta-llama/Llama-3.2-3B-Instruct and TinyLlama 2 2 2 https://huggingface.co/TinyLlama/TinyLlama-1.1B-Chat-v1.0 , both designed for instruction following and low-resource deployment. As a classical baseline, we included gpt2 3 3 3 https://huggingface.co/openai-community/gpt2 , a 1.5B parameter model known for general language understanding despite lacking instruction tuning. From Mistral AI, we tested both Mistral-7B-v0.1 4 4 4 https://huggingface.co/mistralai/Mistral-7B-v0.1 and the instruction-tuned Mistral-7B-Instruct-v0.2 5 5 5 https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2 , which utilize grouped-query attention to balance performance and scalability. Additionally,