LLM Router: Rethinking Routing with Prefill Activations

Tanay Varshney, Annie SurlaMichelle XuGomathy Venkata KrishnanMaximilian JeblickDavid AustinNeal VaidyaDavide Onofrio NVIDIA · 2026

Read LLM Router: Rethinking Routing with Prefill Activations by Tanay Varshney, Annie SurlaMichelle XuGomathy Venkata KrishnanMaximilian JeblickDavid AustinNeal VaidyaDavide Onofrio NVIDIA (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-20895.txt.

Description

Varshney and co-authors route queries between language models using internal activations instead of query text. Text based routers often miss model specific failures and true task difficulty. Their router reads the residual stream during prefill. A key idea splits the encoder that gives the signal from the target model whose success is predicted. Open weight encoders can thus predict closed model results. At its best, their SharedTrunkNet closes about 46 percent of the gap to an oracle router. It also saves about 74 percent of cost versus the priciest model.

Catalog record

CREATORTanay Varshney, Annie SurlaMichelle XuGomathy Venkata KrishnanMaximilian JeblickDavid AustinNeal VaidyaDavide Onofrio NVIDIA
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE47239
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.20895

Browse the full library · About the Tower of Babel