APreQEL: Adaptive Mixed Precision Quantization For Edge LLMs ISBN: 978-1-4503-XXXX-X/2018/06 Conference: ; ;

Meriem Bouzouad, Yuan-Hao Chang, Jalil Boukhobza · 2026

Read APreQEL: Adaptive Mixed Precision Quantization For Edge LLMs ISBN: 978-1-4503-XXXX-X/2018/06 Conference: ; ; by Meriem Bouzouad, Yuan-Hao Chang, Jalil Boukhobza (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-23575.txt.

Introduction

Bouzouad, Chang, and Boukhobza present APreQEL, a quantization scheme that tunes precision layer by layer for edge models. Language models strain small devices on memory, speed, and privacy. Plain quantization shrinks every layer the same way, yet layers react unlike each other. Memory use and speed pull apart, which muddies deployment choices. APreQEL scores what each layer adds and how each precision type runs on the target chip. It then assigns the type per layer under user priorities. The result opens designs that uniform shrinking cannot reach.

Catalog record

CREATORMeriem Bouzouad, Yuan-Hao Chang, Jalil Boukhobza
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33725
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.23575

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

APreQEL: Adaptive Mixed Precision Quantization For Edge LLMs ISBN: 978-1-4503-XXXX-X/2018/06 Conference: ; ; Meriem Bouzouad, Yuan-Hao Chang, Jalil Boukhobza Abstract . Today, large language models have demonstrated their strengths in various tasks ranging from reasoning, code generation, and complex problem solving. However, this advancement comes with a high computational cost and memory requirements, making it challenging to deploy these models on edge devices to ensure real-time responses and data privacy. Quantization is one common approach to reducing memory use, but most methods apply it uniformly across all layers. This does not account for the fact that different layers may respond differently to reduced precision. Importantly, memory consumption and computational throughput are not necessarily aligned, further complicating deployment decisions. This paper proposes an adaptive mixed precision quantization mechanism that balances memory, latency, and accuracy in edge deployment under user-defined priorities. This is achieved by analyzing the layer-wise contribution and by inferring how different quantization types behave across the target hardware platform in order to assign the most suitable quantization type to each layer. This integration ensures that layer importance and the overall performance trade-offs are jointly respected in this design. Our work unlocks new configuration designs that uniform quantization cannot achieve, expanding the solution space to efficiently deploy the LLMs on resource-constrained devices. 1. Introduction The enhancement of reasoning capabilities in large language models (LLMs) is mainly associated with the growth in their number of parameters. Thus, enabling those models to capture complex relationships and dependencies from data ( 12 ) . However, such scale-up necessitates substantial computational power and significant memory capacity, often requiring deployment on resource-rich cloud platforms for inference requests, especially to ensure scalability and assure quality of service for the users. This shift raises various security concerns, as well as a communication overhead, particularly due to internet connection and availability issues in certain regions. With the advent of edge intelligence and real-time processing of data, deploying LLM models on the edge has gained much interest in research areas ( 11 ) , especially with advancements in hardware accelerators in edge devices, such as integrated GPU, NPU, and powerful CPUs. However, the memory bandwidth does not scale with the compute speed, which hinders the overall performance of systems, particularly for LLMs, which require a significant amount of memory to load the weight matrices and context requests. Indeed, deploying LLMs on the edge requires having a reasonable Quality of Service metrics in terms of memory footprint while ensuring good accuracy with limited latency . One of the most adopted strategies to deploy those models efficiently is quantization, which reduces the precision of weights, generally from 16-bit to 8-bit or even 2-bit. This has allowed those large models to run on GPU consumer hardware or even on CPU; however, the hardware support for operations under 8-bits is limited to a few types of AI accelerators, which are rarely present in edge devices. As a result, the acceleration is not always reachable, since the de-quantization overhead can offset the theoretical speedup expected from memory footprint reduction. Many quantization schemes have appeared in recent years such as GPTQ ( 6 ) , AWQ ( 9 ) , SmoothQuant ( 13 ) , K-quant types from llama.cpp 1 1 1 https://github.com/ggml-org/llama.cpp library, which algorithm is formulated in ( 5 ) . Some recent work, such as ShortGPT ( 10 ) , have exploited the similarity between layers to prune the layers judged unimportant, while others ( 4 ) assign variable quantization depending on the importance of the layers; however, in addition to being limited to one QoS metric objective (memory footprint), their experiments are limited to 2-bit and 4-bit quantization. In addition, except for ( 4 ) , all the techniques cited above follow a uniform quantization scheme, which treats all layers in an LLM equally. However, not every layer affects accuracy in the same way. Some layers are critical and require higher precision, while others can tolerate lower precision without significant loss of accuracy ( 4 ) ; uniform quantization ignores this difference, often causing unnecessary reductions in accuracy or missing opportunities for more efficient compression ( 14 ) . Moreover, uniform quantization may lead to a degraded QoS, in terms of inference latency and memory if hardware-specific characteristics are ignored. This is because different quantization methods introduce varying, often uncorrelated impacts on latency and memory depending on the model architecture, as reported in the motivation Section ( 2.3 ), largely due to de-quantization overhead. As a result, model compression may fail to translate into actual efficiency gains on a given target hardware. To the best of our knowledge, previous studies did not explore the trade-offs achievable through mixed quantization schemes, particularly when latency is included. This limitation is particularly problematic in edge environments, where resource availability often fluctuates more than in the cloud, requiring designs that are both adaptable and efficient to meet shifting requirements. In this paper, we propose APreQEL, a flexible solution for layer-wise quantization in LLMs, which adapts to diverse deployment scenarios by incorporating user-driven QoS requirements. APreQEL makes it possible for users to make a trade-off between QoS metrics (memory footprint, model accuracy, and latency). To this end, our approach (1) ranks layers depending on their contribution to the change of information; (2) determines the best distribution of quantization types, accounting for their distinct QoS metrics; (3) combines the ranking from step (1) with the distribution from step (2) to assign the most suitable quantization type to each layer. Our findings demonstrate that APreQEL systematically guarantees that the generated solutions are spread over the Pareto-front, under any priority set by the user, enabling either focused optimization on a single metric, joint optimization of two metrics, or balanced trade-offs across memory footprint, accuracy, and latency. Notably, APreQEL achieves \sim 9% increase in the hypervolume indicator on average over the uniform baseline set, reflecting a broader Pareto front while considering only a small set of 28 configurations. 2. Background and Motivation 2.1. Background LLMs are built on top of a decoder-only transformer architecture ( 2 ) . The natural language prompt is converted to tokens, which are mapped to high-dimensional vectors called embeddings to capture their semantic content. Those embeddings are then passed through N decoding blocks, each containing a self-attention layer and a feedforward network (FNN) layer, to capture the contextual relationship between the different embeddings. Finally, LM_head module predicts the next token. 2.2. Motivation1: Layers contribute differently In this part, we examine how the intermediate representation (embeddings) evolves across transformer layers to assess whether layers contribute differently, suggesting that distinct layers may benefit from different quantization strategies. To conduct this observation, we used the Llama3.1 model, where token-level cosine similarity is computed using 70 prompts from the huggingface WebInstructSub-prometheus dataset 2 2 2 https://huggingface.co/datasets/chargoddard/WebInstructSub-prometheus and the transformers library of python, with 32bit precision. Figure 1. Cosine similarity distribution for Llama-3.1 4 4 4 https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct including Attention and FFN layers. Figure 4 illustrates the cosine similarity of hidden embeddings across consecutive layers, accounting for both the attention (blue) and the feed-forward network (grey). We observe different trends; sometimes attention layers are closer to 1 (highly similar), and sometimes the feed-forward network layer is, depending on their position; This

Read the complete text