From Exact Hits to Close Enough: Semantic Caching for LLM Embeddings
Dvir David Biton, Roy Friedman · 2026
Read From Exact Hits to Close Enough: Semantic Caching for LLM Embeddings by Dvir David Biton, Roy Friedman (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-03301.txt.
Description
Dvir David Biton and Roy Friedman study semantic caching, a speed trick for large language model services. Classic caches reuse exact repeats; a semantic cache reuses answers for new requests that merely mean something close to an old one, judged by embedding vectors. Their 2026 paper asks which requests such a cache should keep when space runs out.
The authors prove that even with full hindsight, picking the perfect offline policy is computationally intractable, then offer fast heuristics that run in reasonable time. They also design live policies mixing recency, frequency, and semantic closeness. Tests on varied datasets show frequency-based rules are strong, while their new variant improves semantic accuracy further. All of the code is open source.
Catalog record
| CREATOR | Dvir David Biton, Roy Friedman |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 60342 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.03301