DILEMMA: Joint LLM Quantization and Distributed LLM Inference Over Edge Computing Systems
Minoo Hosseinzadeh, Hana Khamfroush · 2025
Read DILEMMA: Joint LLM Quantization and Distributed LLM Inference Over Edge Computing Systems by Minoo Hosseinzadeh, Hana Khamfroush (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-01704.txt.
Description
Large language models are too heavy for the small servers that sit near users at the network edge. Minoo Hosseinzadeh and Hana Khamfroush present DILEMMA, a framework that squeezes and splits a model across edge machines. It decides which layers go where, and how much each layer can be compressed.
The authors frame the choices as an optimization problem that minimizes answer delay while keeping model quality acceptable. Layer-wise compression is steered by knowledge distillation, which helps the small version mimic the full one. Tests on a mid-size model with a question-answering data set reached a compression ratio of about 12.75 percent without losing model quality. The results suggest edge deployment of language services is workable when placement and compression are planned together.
Catalog record
| CREATOR | Minoo Hosseinzadeh, Hana Khamfroush |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 25 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-01704