Efficient LLM Inference: Bandwidth, Compute, Synchronization, and Capacity are all you need

Michael Davies, Neal Crago, Karthikeyan Sankaralingam, Christos Kozyrakis · 2025

Read Efficient LLM Inference: Bandwidth, Compute, Synchronization, and Capacity are all you need by Michael Davies, Neal Crago, Karthikeyan Sankaralingam, Christos Kozyrakis (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-14397.txt.

Description

Michael Davies and co-authors map the hard limits of language model serving in a 2025 paper. Their LIMINAL model predicts how fast a model can generate text on given hardware. It abstracts both the program's needs and the machine's traits. Against real runs, its calls landed within about eight percent on average.

The sweep ran from today's accelerator memory to future stacked designs. Five barriers proved non-negotiable, among them compute, memory size, bandwidth, and machine-to-machine talk. Passing ten thousand tokens per second, they find, will take more than better chips. Algorithmic advances must share the load. The authors offer the model as a planning lens for hardware buyers and designers. It shows which wall a proposed system hits first, before money is spent.

Catalog record

CREATORMichael Davies, Neal Crago, Karthikeyan Sankaralingam, Christos Kozyrakis
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE53 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-14397

Browse the full library · About the Tower of Babel