FlexInfer: Breaking Memory Constraint via Flexible and Efficient Offloading for On-Device LLM Inference

Hongchao Du, Shangyu Wu, Arina Kharlamova, Nan Guan, Chun Jason Xue · 2025

Read FlexInfer: Breaking Memory Constraint via Flexible and Efficient Offloading for On-Device LLM Inference by Hongchao Du, Shangyu Wu, Arina Kharlamova, Nan Guan, Chun Jason Xue (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-03777.txt.

Description

Running a large language model on a small device runs into a wall: the model needs more memory than the device holds. Hongchao Du and co-authors present FlexInfer in this 2025 paper, a system that moves parts of the model in and out of memory as needed. Older offloading schemes often slowed the model or could not adapt to set limits.

FlexInfer prefetches parts ahead of time in the background, balances which memory stays locked, and keeps tensors in a flexible way. Together these steps ease the traffic jam between storage and memory. In tests under tight limits, the system ran up to twelve and a half times faster than rival methods. The authors aim to make big models usable on phones and other small hardware.

Catalog record

CREATORHongchao Du, Shangyu Wu, Arina Kharlamova, Nan Guan, Chun Jason Xue
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE26 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-03777

Browse the full library · About the Tower of Babel