FlexInfer: Breaking Memory Constraint via Flexible and Efficient Offloading for On-Device LLM Inference
Hongchao Du, Shangyu Wu, Arina Kharlamova, Nan Guan, Chun Jason Xue · 2025
Read FlexInfer: Breaking Memory Constraint via Flexible and Efficient Offloading for On-Device LLM Inference by Hongchao Du, Shangyu Wu, Arina Kharlamova, Nan Guan, Chun Jason Xue (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-03777.txt.
Description
Running a large language model on a small device runs into a wall: the model needs more memory than the device holds. Hongchao Du and co-authors present FlexInfer in this 2025 paper, a system that moves parts of the model in and out of memory as needed. Older offloading schemes often slowed the model or could not adapt to set limits.
FlexInfer prefetches parts ahead of time in the background, balances which memory stays locked, and keeps tensors in a flexible way. Together these steps ease the traffic jam between storage and memory. In tests under tight limits, the system ran up to twelve and a half times faster than rival methods. The authors aim to make big models usable on phones and other small hardware.
Catalog record
| CREATOR | Hongchao Du, Shangyu Wu, Arina Kharlamova, Nan Guan, Chun Jason Xue |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 26 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-03777