ADOR: A Design Exploration Framework for LLM Serving with Enhanced Latency and Throughput

Junsoo Kim, Hunjong Lee, Geonwoo Ko, Gyubin Choi, Seri Ham, Seongmin Hong, Joo-Young Kim · 2025

Read ADOR: A Design Exploration Framework for LLM Serving with Enhanced Latency and Throughput by Junsoo Kim, Hunjong Lee, Geonwoo Ko, Gyubin Choi, Seri Ham, Seongmin Hong, Joo-Young Kim (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-04253.txt.

Description

Seven researchers wrote a computer engineering paper in two thousand twenty-five on designing hardware to serve large language models. Serving is split into two stages with opposite needs. Reading the prompt is compute-heavy, while writing the answer leans on memory speed. Standard graphics chips balance the two poorly, wasting capacity.

Batching requests helps the hardware but slows each reply, which users feel. Their framework, ADOR, searches over ready-made architecture templates to recommend hardware tuned for model serving. It aims to reconcile the vendor's wish for full hardware use with the user's wish for fast answers, across varied serving setups.

Catalog record

CREATORJunsoo Kim, Hunjong Lee, Geonwoo Ko, Gyubin Choi, Seri Ham, Seongmin Hong, Joo-Young Kim
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE42 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-04253

Browse the full library · About the Tower of Babel