GUIDE: A G lobal U nified I nference Engine for D eploying Large Language Models in Heterogeneous E nvironments

Yanyu Chen, Ganhong Huang · 2024

Read GUIDE: A G lobal U nified I nference Engine for D eploying Large Language Models in Heterogeneous E nvironments by Yanyu Chen, Ganhong Huang (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-04788.txt.

Description

Yanyu Chen and Ganhong Huang present GUIDE, a 2024 framework for running large language models well on mixed hardware. Real deployments mix old and new chips, varied software, and shifting workloads, which wastes memory and makes reply speeds jump around. Non-experts struggle most with the tangle of settings.

The team first measure where the waste comes from, tracking memory drops, latency swings with batch size, and losses when several chips share the work. GUIDE then models a setup and simulates choices before money is spent on real runs. Its forecasts of key speeds land within about 10 to 42 percent of true values. The tool aims to give ordinary teams a cheap, data-based way to tune model serving on whatever hardware they have.

Catalog record

CREATORYanyu Chen, Ganhong Huang
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE62 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-04788

Browse the full library · About the Tower of Babel