Tiny-QMoE
Jack Cashman, Jiaqi Nie · 2025
Read Tiny-QMoE by Jack Cashman, Jiaqi Nie (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-22951.txt.
Introduction
Huge mixture-of-experts models can need terabytes of memory, far past what a phone holds. Jack Cashman and Jiaqi Nie discuss Tiny-QMoE, a push to squeeze such models onto small devices. A prior system, QMoE, compressed these models for large servers with high-end graphics cards. Phones, by contrast, share a few gigabytes of memory with the whole system.
The pair also weigh delay: sending each request to a remote server adds waiting time. Offline settings with no network access add a further motive. The paper frames the constraints and the goal, rather than reporting a finished system. It sets out why edge use needs its own compression work.
Catalog record
| CREATOR | Jack Cashman, Jiaqi Nie |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 31 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-22951
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Tiny-QMoE Jack Cashman, Jiaqi Nie Abstract The QMoE model [ 1 ] provides a practical approach for compression of massive Mixture-of-Experts (MoE) models. QMoE offers a solution geared towards memory limitations that often reach terabyte scales, and it has the advantage of working with high sparsity models which implicitly lend themselves to compression techniques. QMoE also has the advantage of only taking MoE models into account and does not evaluate its use with non mixture of expert systems. Although this prior attempt focuses on the limitations of large servers with the latest NVIDIA hardware which in the case of the H100 and V100 which have 80 GB of HBM (High Bandwidth Memory), what is not being considered is a significantly more constrained environment, such as in the case of mobile devices which may have in the case of the iPhone anywhere from 4 to 8 GB of unified memory which also needs to be shared with the operating system and additional processes. Although edge devices such as phones and laptops are becoming increasingly more computationally powerful, they are still not close to the level of advanced server machines such as NVIDIA. An additional constraint that we must consider is that of latency. The communication time of sending a request to an LLM server and then getting it back is an additional waiting time that can be removed. We may also want to use LLM technology in environments where there is no reliable network connection. As will be discussed later, while the latency of the internet is removed, the latency of the decompression is incurred. In this paper, we present a solution to this highly constrained memory problem. This takes the form of a re-imagined quantization compression schema and execution such that models which would have normally exceeded the memory requirements of say a 2060 with 6 GB of VRAM. Specifically, Tiny-QMoE works on a variety of LLAMA 3.2 models and does so by quantizing the models into 8-bit versions and then taking said models and storing them in a dictionary based compression format. 1 Introduction LLM’s or large language models have in the last few years experienced a meteoric rise in popularity. People have used these models for an unbelievably wide array of natural language and reasoning applications. While demand for these models is high, so to is the computational and memory cost of inference. If a user were to try and run a cutting edge large language model locally on device, they will not be able to run it due to this memory limitation. To get around this issue, the conventional knowledge has been that in order to run large language models locally, models with a lower memory threshold are needed. This production of smaller models comes at a noticeable cost in model performance. Significantly shrinking the number of parameters and aggressive quantization often make models so inaccurate that its output will be unusable by users. Compared to previous solutions Tiny-QMoE comes at the problem of memory constrain of mobile devices in a novel way. The system was required to make significant changes to prior approaches. The main contributions of this work are summarized as follows: • Novel Compression Technique: Developed a memory efficient approach to run quantized large language models (LLMs) locally without significant degradation in model performance or latency. This effort is accomplished through the storage of quantized LLAMA 3.2 models [ 2 ] into a dictionary based compression schema. • Preservation of Model Accuracy: Demonstrated the degree to which aggressive quantization can maintain usability and output quality when applied with our method. The issue of accuracy despite quantization must be accomplished through a data dependent approach rather than simply a naive reduction in value representation. The quantization of smaller models also requires a less aggressive degree of quantization for comparable reduction in performance compared to that of significantly larger and denser models. • Enhanced Accessibility of LLMs: Enabled the use of cutting edge LLMs on resource constrained devices, bridging the gap between high computational demands and device limitations. Large Language Models should be able to be ran on more constrained devices and in turn enable usage on device with the benefit of working without internet connectivity. An on device model would allow for more data privacy as user prompts will not have to be sent to the web. This privacy can minimize data tracking and the fears of possible data breaches. LLM’s on mobile devices may also bring the promise of better energy consumption as many cutting edge arm chips may outperform NVIDIA server scale AI systems in terms of performance per watt. This increase in accessibility requires the rejection of hardware specific code and in our current implementation focuses execution of inference on CPU execution. • Performance Evaluation: As we want to ensure that our models are in fact as reduced in size as we would hope for them to be and as performant both in terms of latency and quality of output we must run evaluations. Our evaluation of extensive experiments showcasing the trade off between model size and performance across various reasoning and natural language tasks. We expect there to be an increase in latency and a decrease in quality of output, but not too much. Our focus given these setbacks comes down to the compression which we aim to maximize. If the model size is less than the memory of a system then it would be able to operate within the respective system which would mean that compression can effectively increase the amount of hardware any model can be used on especially given our hardware agnostic approach. • Scalable Deployment of Variable Models: We design our framework with the intentionality that it can be applied to LLAMA 3.2 models of varying size and capabilities. The approach allows for the goals and limitations of the different implementations to be scalable allowing users to opt between the range of models. Our range of LLAMA 3.2 models consists of the 1B and 3B models but we plan to expand to include 11B and 90B where each model is named based on the number of parameters they have where LLAMA 3.2 1B has one billion parameters and the rest follow respectively. We have an expectation that as the number of parameters increases the compression rate will also increase. Our proposed current work sadly comes with several limitations however. This in turn makes it the case that simply taking the implementation directly from QMoE [ 1 ] would not be possible. QMoE simply cannot suit the altered context of these both smaller and non MoE models. While the QMoE approach demonstrates significant advancements in compression and efficiency, these gains face certain limitations associated with our approach: • Dependence on High Sparsity: QMoE relies heavily on high sparsity to achieve substantial compression rates. This dependency makes it less effective for smaller and in turn less sparse models to be used limiting its applicability in certain scenarios like that of LLAMA 3.2 and other smaller to medium sized models. The dependency on high sparsity is reasonable in the MoE context. The mixture of experts allows for an increased parameter distribution through a subnet of input tokens. Given that it