Splitwiser: Efficient LM inference with constrained resources

Asad Aali [email protected] Stanford University, Adney Cardoza [email protected] UT Austin, Melissa Capo [email protected] UT Austin · 2025

Read Splitwiser: Efficient LM inference with constrained resources by Asad Aali [email protected] Stanford University, Adney Cardoza [email protected] UT Austin, Melissa Capo [email protected] UT Austin (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-03763.txt.

Description

Asad Aali and co-authors share a report on running large language models on tight hardware. Serving a model has two phases. Reading the prompt is heavy on raw compute, while writing the reply leans on memory. The second phase often leaves the chip half idle, which wastes costly gear.

Their method, Splitwiser, splits the two phases of a request across the same chip. Sharing one chip cuts the need to ship data between devices. It also improves memory access and cache use. The team built the design on two well known stacks and saw gains in early tests. The code is open for others to try and extend.

Catalog record

CREATORAsad Aali [email protected] Stanford University, Adney Cardoza [email protected] UT Austin, Melissa Capo [email protected] UT Austin
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE25 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-03763

Browse the full library · About the Tower of Babel