QuoKA : Query-oriented KV selection for Efficient LLM Prefill
Dalton Jones, Junyoung Park, Matthew Morse, Mingu Lee, Chris Lott, M. Harper Langston · 2026
Read QuoKA : Query-oriented KV selection for Efficient LLM Prefill by Dalton Jones, Junyoung Park, Matthew Morse, Mingu Lee, Chris Lott, M. Harper Langston (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-08722.txt.
Description
Chunked prefill in transformers wastes compute on keys that barely matter, but most sparse attention schemes pick keys blindly. Dalton Jones and five co-authors present QUOKA, a training-free, hardware-agnostic method that starts from queries. Their arXiv paper observes that queries dissimilar to the mean query interact with more keys and drive final logits most.
Prioritizing those queries, QUOKA first retains a small set of representative queries, then subselects the keys most aligned with them. Tests on needle-in-haystack, LongBench, RULER, and Math500 show time-to-first-token cut threefold and attention sped up fivefold on GPUs, nearly sevenfold on Intel CPUs, at near-baseline accuracy with eighty-eight percent fewer key-value pairs. Readers get a query-first shortcut for faster inference.
Catalog record
| CREATOR | Dalton Jones, Junyoung Park, Matthew Morse, Mingu Lee, Chris Lott, M. Harper Langston |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 62 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-08722