HACK: Homomorphic Acceleration via Compression of the Key-Value Cache for Disaggregated LLM Inference Conference: arXiv; 2025
Zeyu Zhang, Haiying Shen, Shay Vargaftik, Ran Ben Basat, Michael Mitzenmacher, Minlan Yu · 2025
Read HACK: Homomorphic Acceleration via Compression of the Key-Value Cache for Disaggregated LLM Inference Conference: arXiv; 2025 by Zeyu Zhang, Haiying Shen, Shay Vargaftik, Ran Ben Basat, Michael Mitzenmacher, Minlan Yu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-03589.txt.
Description
Zeyu Zhang and co-authors shrink the memory traffic that slows split large language model serving. Splitting a model run into a prefill stage and a decode stage uses hardware better. The snag is shipping key-value data between the two stages for long prompts. Shrinking that data by quantization helps, but restoring it costs heavy compute.
Their HACK system computes directly on the compressed data instead. The math mimics the full step closely enough to skip dequantization entirely. Trace-driven tests cut job completion time by up to about seventy-one percent. Gains over top quantization methods reached about fifty-two percent. Long prompts gain the most from the trick.
Catalog record
| CREATOR | Zeyu Zhang, Haiying Shen, Shay Vargaftik, Ran Ben Basat, Michael Mitzenmacher, Minlan Yu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 58 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-03589