SageServe : Optimizing LLM Serving on Cloud Data Centers with Forecast Aware Auto-Scaling

Shashwat Jaiswal, Kunal Jain, Yogesh Simmhan, Anjaly Parayil, Ankur Mallick, Rujia Wang, Renee St. Amant, Chetan Bansal , Victor Ruhle, Anoop Kulkarni, Steve Kofsky, Saravan Rajmohan · 2025

Read SageServe : Optimizing LLM Serving on Cloud Data Centers with Forecast Aware Auto-Scaling by Shashwat Jaiswal, Kunal Jain, Yogesh Simmhan, Anjaly Parayil, Ankur Mallick, Rujia Wang, Renee St. Amant, Chetan Bansal , Victor Ruhle, Anoop Kulkarni, Steve Kofsky, Saravan Rajmohan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-14617.txt.

Description

Jaiswal and co-authors present SageServe, a framework for serving large language models in cloud data centers at lower cost. Cloud providers mix urgent jobs like chatbots with slower ones like report writing, each with its own service promises. Running them in separate resource pools wastes expensive graphics hardware whenever loads do not match. The team first studied real serving traffic from Microsoft Office 365, over ten million requests a day. They report patterns across regions and times of day.

SageServe reacts on two time scales. It routes requests between data centers quickly. It also scales hardware and places models ahead of time, using a traffic forecast. An optimization solver decides both parts jointly. In tests on real traffic, the system cut graphics-processor hours by up to 25 percent against the current setup. The possible saving reaches 2.5 million dollars a month, while latency promises still held.

Catalog record

CREATORShashwat Jaiswal, Kunal Jain, Yogesh Simmhan, Anjaly Parayil, Ankur Mallick, Rujia Wang, Renee St. Amant, Chetan Bansal , Victor Ruhle, Anoop Kulkarni, Steve Kofsky, Saravan Rajmohan
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE70 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-14617

Browse the full library · About the Tower of Babel