Advancing Semantic Caching for LLMs with Domain-Specific Embeddings and Synthetic Data
Waris Gill, Justin Cechmanek, Tyler Hutcherson, Srijith Rajamohan, Jen Agarwal, Muhammad Ali Gulzar, Manvinder Singh, Benoit Dion · 2025
Read Advancing Semantic Caching for LLMs with Domain-Specific Embeddings and Synthetic Data by Waris Gill, Justin Cechmanek, Tyler Hutcherson, Srijith Rajamohan, Jen Agarwal, Muhammad Ali Gulzar, Manvinder Singh, Benoit Dion (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-02268.txt.
Introduction
Chatbots answer the same questions again and again, and caching old answers can cut cost and delay. Waris Gill and co-authors improve that caching in this 2025 report. Their cache is semantic: it matches a new question to a stored one by meaning, not by exact words. Meaning is judged by embedding models, which turn text into number lists.
The team argues that small embedding models tuned for one field beat big general ones. Tuned for just one short training round on field data, the small models won on both accuracy and recall. Real labeled data for a field is scarce, so they also built a pipeline that makes fake training data. The fake data lifted scores further. The balance they strike suits live systems that need fast, cheap, and right answers.
Catalog record
| CREATOR | Waris Gill, Justin Cechmanek, Tyler Hutcherson, Srijith Rajamohan, Jen Agarwal, Muhammad Ali Gulzar, Manvinder Singh, Benoit Dion |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 31 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-02268
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Advancing Semantic Caching for LLMs with Domain-Specific Embeddings and Synthetic Data Waris Gill, Justin Cechmanek, Tyler Hutcherson, Srijith Rajamohan, Jen Agarwal, Muhammad Ali Gulzar, Manvinder Singh, Benoit Dion Abstract This report investigates enhancing semantic caching effectiveness by employing specialized, fine-tuned embedding models. Semantic caching relies on embedding similarity rather than exact key matching, presenting unique challenges in balancing precision, query latency, and computational efficiency. We propose leveraging smaller, domain-specific embedding models, fine-tuned with targeted real-world and synthetically generated datasets. Our empirical evaluations demonstrate that compact embedding models fine-tuned for just one epoch on specialized datasets significantly surpass both state-of-the-art open-source and proprietary alternatives in precision and recall. Moreover, we introduce a novel synthetic data generation pipeline for the semantic cache that mitigates the challenge of limited domain-specific annotated data, further boosting embedding performance. Our approach effectively balances computational overhead and accuracy, establishing a viable and efficient strategy for practical semantic caching implementations. 1 Introduction Large language models (LLMs) are rapidly becoming integral components of modern applications, significantly influencing everyday tasks. These models consist of billions of neurons and require substantial computational infrastructure to operate effectively. Each user query to an LLM involves billions of floating-point operations to generate an appropriate response Frantar et al. (2023) . In practice, users often issue repeated queries to online services. Prior research Lempel and Moran (2003) ; Xie and O’Hallaron (2002) ; Markatos (2001) indicates that approximately 33% of queries submitted to web search engines are repeated. A similar phenomenon occurs with LLM-based services Gill et al. (2025) . Recognizing this, researchers have proposed implementing a semantic cache to efficiently handle duplicate queries directed at LLMs Bang (2023) ; Gill et al. (2025) ; Zhu et al. (2023) . Unlike traditional key-value caches, a semantic cache does not rely on exact key matching. Instead, it declares a cache hit if the similarity between the embeddings generated from two textual inputs surpass a predefined cosine similarity threshold Bang (2023) ; Gill et al. (2025) . A semantic cache generally comprises two essential components: an embedding model, which computes query embeddings, and a vector database, which stores these embeddings and retrieves cached responses by matching embeddings of repeated queries. A variety of embedding models are available, ranging from open-source to proprietary closed-source models, both demonstrating state-of-the-art (SOTA) performance on benchmarks such as MTEB Muennighoff et al. (2023) . An important consideration thus arises: which embedding model is optimal for semantic caching? Both open-source and closed-source options have their respective advantages and disadvantages. Open-source models are freely accessible, allowing users to maintain data privacy by running models on their own infrastructure. However, these models often contain billions of parameters, as exemplified by Alibaba-NLP/gte-Qwen2-7B-instruct (a 7-billion-parameter embedding model Li et al. (2023) ; Ali () ), making them computationally intensive and less suitable for efficient semantic caching. Conversely, closed-source embedding models offered via managed services are costly, may raise data privacy concerns due to external data handling, and introduce network latency, potentially hindering the performance of semantic caches. Ideally, an embedding model used in semantic caching should be lightweight and computationally efficient. Unfortunately, smaller models typically lag behind their larger counterparts in performance. Nevertheless, recent studies have demonstrated that smaller, task-specific fine-tuned models can surpass the performance of significantly larger models Ouyang et al. (2022) ; Penedo et al. (2024) ; Du and Kaelbling (2024) . Motivated by this insight, we fine-tuned smaller embedding models on domain-specific datasets (medical and Quora kag (b) datasets). Remarkably, our experiments revealed that fine-tuning for just one epoch enabled these compact models to outperform both state-of-the-art closed-source and open-source models. However, fine-tuning these smaller models requires high-quality datasets, which may not always be readily available. To address this limitation, we developed a unique synthetic data generation pipeline for the semantic cache. This pipeline leverages existing datasets from diverse domains to produce targeted synthetic data tailored specifically for different applications and domains. Our evaluation demonstrates that synthetic data significantly enhances the smaller embedding model performance for semantic caching, achieving precision improvements of 9% compared to its non-finetuned base model. Moreover, when tested on real-world medical queries, the model fine-tuned on medical synthetic data achieves performance approaching or even matching that of leading open-source and closed-source models, surpassing OpenAI’s embedding model by 2% in precision. 2 Methodology Embedding models lie at the heart of semantic caching. Their role is to produce high-quality, high-dimensional representations that encapsulate the semantic information in text. The better these models are at distinguishing subtle nuances in meaning, the more effective the cache becomes at reusing relevant results. State-of-the-art embedding models, whether open-source (such as Alibaba-NLP/gte-Qwen2-7B-instruct) Li et al. (2023) ; Ali () ; Zhang et al. (2024) or closed-source Ama () ; Emb () ; Vec () ; Lee et al. (2024) ; Lee et al. (2025) ; Neelakantan et al. (2022) , have shown remarkable effectiveness in tasks like sentence similarity, text classification, and semantic retrieval. In semantic caching, however, the overarching challenge is balancing performance with computational and operational constraints. While larger models often excel in capturing fine-grained nuances, they are costly to run repeatedly at scale. Conversely, more efficient, smaller models may falter on challenging queries, leading to suboptimal retrieval and degraded cache performance. A key shortcoming of baseline (out-of-the-box) embedding models is that they are typically trained on broad, general-purpose corpora. When faced with domain-specific queries (e.g., finance, medical, or legal), these models may struggle to capture the nuanced relationships unique to that domain. This leads to lower performance, especially for duplicate queries that differ slightly in wording but share the same semantic content. For instance, in the medical domain, the queries “myocardial infarction treatment” and “how to treat a heart attack” may not be recognized as semantically equivalent by a general-purpose embedding model. Such limitations mean that, in practice, relying solely on these larger, general-purpose embedding models can either incur excessive compute overhead or fail to retrieve sufficiently accurate cached results for highly specialized queries. Consequently, there is a growing need for fine-tuned, domain-adapted embedding models, particularly compact ones, that can provide robust semantic representations while remaining efficient to deploy in production. To address the challenges outlined above, our methodology comprises three main components: (1) Model Selection and Training, (2) Domain-Specific Fine-Tuning, and (3) Synthetic Data Generation. We begin by selecting a compact, publicly available embedding model, ModernBERT Warner et al. (2024) , which contains approximately 149 million parameters. ModernBERT is a recent encoder-only transformer that outperforms comparable models such as BERT Devlin et al. (2019) , NomicBERT Nussbaum et al. (2025) , and RoBERTa Liu et al. (2019) . While our insights are broadly applicable to other smaller models like MiniLM Wang et al. (2020) , MPNet Song et al. (2020) , and ALBERT Lan et al. (2020) , we choose ModernBERT for its strong balance of efficiency and performance. Models like ModernBERT are particularly advantageous for semantic caching due to their reduced computational footprint and faster inference times. To fine-tune the ModernBERT embeddings for domain-specific queries, we use the online contrastive loss function Reimers and Gurevych (2019) . For simplicity and brevity, we refer to the fine-tuned ModernBERT model as LangCache-Embed in our evaluations, in contrast to the non-fine-tuned base ModernBERT hug