GUIDE: A G lobal U nified I nference Engine for D eploying Large Language Models in Heterogeneous E nvironments
Yanyu Chen, Ganhong Huang · 2024
Read GUIDE: A G lobal U nified I nference Engine for D eploying Large Language Models in Heterogeneous E nvironments by Yanyu Chen, Ganhong Huang (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-04788.txt.
Description
Yanyu Chen and Ganhong Huang present GUIDE, a 2024 framework for running large language models well on mixed hardware. Real deployments mix old and new chips, varied software, and shifting workloads, which wastes memory and makes reply speeds jump around. Non-experts struggle most with the tangle of settings.
The team first measure where the waste comes from, tracking memory drops, latency swings with batch size, and losses when several chips share the work. GUIDE then models a setup and simulates choices before money is spent on real runs. Its forecasts of key speeds land within about 10 to 42 percent of true values. The tool aims to give ordinary teams a cheap, data-based way to tune model serving on whatever hardware they have.
Catalog record
| CREATOR | Yanyu Chen, Ganhong Huang |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 62 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-04788