LLM Router: Rethinking Routing with Prefill Activations
Tanay Varshney, Annie SurlaMichelle XuGomathy Venkata KrishnanMaximilian JeblickDavid AustinNeal VaidyaDavide Onofrio NVIDIA · 2026
Read LLM Router: Rethinking Routing with Prefill Activations by Tanay Varshney, Annie SurlaMichelle XuGomathy Venkata KrishnanMaximilian JeblickDavid AustinNeal VaidyaDavide Onofrio NVIDIA (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-20895.txt.
Description
Varshney and co-authors route queries between language models using internal activations instead of query text. Text based routers often miss model specific failures and true task difficulty. Their router reads the residual stream during prefill. A key idea splits the encoder that gives the signal from the target model whose success is predicted. Open weight encoders can thus predict closed model results. At its best, their SharedTrunkNet closes about 46 percent of the gap to an oracle router. It also saves about 74 percent of cost versus the priciest model.
Catalog record
| CREATOR | Tanay Varshney, Annie SurlaMichelle XuGomathy Venkata KrishnanMaximilian JeblickDavid AustinNeal VaidyaDavide Onofrio NVIDIA |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 47239 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.20895