CustomIR: Unsupervised Fine-Tuning of Dense Embeddings for Known Document Corpora
Nathan Paull · 2025
Read CustomIR: Unsupervised Fine-Tuning of Dense Embeddings for Known Document Corpora by Nathan Paull (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-21729.txt.
Introduction
Nathan Paull adapts small search models to specialized document collections without any human labeling. General embedding models turn text into searchable vectors. They lose their edge on niche corpora far from their training data. Company email and chat archives are exactly such corpora. Retrieval-augmented chatbots need those models most there.
His framework, CustomIR, has a large language model invent diverse queries rooted in the target corpus itself. It then verifies hard negative examples, documents that look relevant but are not. Those negatives sharpen the training signal. Fine-tuned on the synthetic pairs, small models gain up to about two points of recall at ten results. The tests use enterprise email and messaging data. The gains let small models rival far larger ones, so cheaper deployments follow.
Catalog record
| CREATOR | Nathan Paull |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 23 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-21729
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
CustomIR: Unsupervised Fine-Tuning of Dense Embeddings for Known Document Corpora Nathan Paull Abstract Dense embedding models have become critical for modern information retrieval, particularly in RAG pipelines, but their performance often degrades when applied to specialized corpora outside their pre-training distribution. To address this we introduce CustomIR , a framework for unsupervised adaptation of pre-trained language embedding models to domain-specific corpora using synthetically generated query–document pairs. CustomIR leverages large language models (LLMs) to create diverse queries grounded in a known target corpus, paired with LLM-verified hard negatives, eliminating the need for costly human annotation. Experiments on enterprise email and messaging datasets show that CustomIR consistently improves retrieval effectiveness with small models gaining up to 2.3 points in Recall@10. This performance increase allows these small models to rival the performance of much larger alternatives, allowing for cheaper RAG deployments. These results highlight that targeted synthetic fine-tuning offers a scalable and cost-efficient strategy for increasing domain-specific performance. Introduction Dense embedding models have become the backbone of modern information retrieval, largely due to training on massive, heterogeneous datasets that capture broad linguistic patterns [ 27 , 21 , 8 ] . In practice, these models are designed to handle many domains simultaneously, leading to a strategy of scaling both model and dataset in order to improve benchmark performance [ 29 , 32 ] . While effective for broad coverage, this approach is computationally expensive and often intractable for real-world implementation of dense embeddings, even exceeding the domain needs of many applications where users operate within a single domain. We argue that the challenges faced by dense embeddings in specialized domains—such as legal, biomedical, or enterprise texts, stem less from insufficient coverage and more from an adaptation problem: inference data differs greatly from the training corpus, leading to data distribution imbalances or even out-of-distribution (OOD) effects. Rather than attempting to scale models and datasets to encompass all possible domains, it is more efficient to adapt and specialize smaller models to the target domain. To this end, we introduce CustomIR , a synthetic data generation framework that adapts pre-trained embeddings to a target corpus in an unsupervised manner. CustomIR generates domain-specific query–document pairs via large language models (LLMs), enabling fine-tuning without human annotation. This approach captures domain-specific terminology and end-user retrieval patterns, yielding substantial gains in specialized retrieval tasks while requiring only a fraction of the resources needed for scaling-based methods. Research Priors Natural Language Embedding Natural language embeddings are a central component of modern information retrieval and representation learning systems. Early approaches were dominated by predictive word embedding models such as Word2Vec [ 20 ] and GloVe [ 23 ] , which demonstrated the ability to capture semantic similarity in dense vector spaces. These static embeddings were later superseded by contextualized representations introduced by models such as ELMo [ 24 ] , which leveraged deep recurrent architectures to generate embeddings sensitive to linguistic context. The advent of transformer-based models, most prominently BERT [ 7 ] , marked a significant shift by enabling bidirectional contextual understanding at scale. Variants and extensions of BERT, such as RoBERTa [ 17 ] and ALBERT [ 15 ] , improved pretraining efficiency and embedding quality, making them widely adopted for downstream fine-tuning tasks. In parallel, sentence-level representation models such as Sentence-BERT (SBERT) [ 25 ] specialized BERT-style encoders for semantic similarity and retrieval tasks, producing embeddings optimized for pairwise comparison. Recent work has further advanced embedding models through large-scale contrastive learning [ 10 ] , domain-specific pretraining [ 13 ] , and multilingual extensions [ 9 ] , expanding their applicability to diverse tasks and languages. Moreover, models such as OpenAI’s text-embedding series and instruction-tuned encoders have demonstrated that fine-tuning embeddings with task-oriented supervision can yield substantial improvements in retrieval performance. LLM-Generated Synthetic Data Synthetic data generation with large language models has emerged as a promising alternative to traditional large-scale data collection. Rather than scaling datasets indiscriminately, this approach focuses on generating high-quality task-relevant examples to address model deficiencies. Prior work has shown that carefully constructed synthetic data can substantially improve performance in reasoning, language understanding, and retrieval tasks. For example, DeepSeek R1 demonstrates how LLM-generated synthetic reasoning chains and mathematical problems can improve logical inference and problem-solving abilities that are underrepresented in naturally occurring corpora [ 6 ] . The use of this synthetic reasoning data in the fine-tuning stage improves emergent thinking capabilities in the R1 model. Similarly, the Phi models achieve SOTA performance on small parameter language models through the use of high-quality synthetic, textbook-style data [ 12 , 16 , 1 , 2 ] . Synthetic data has emerged as a powerful tool for improving dense embeddings in information retrieval. For example, Qwen3-Embed leverages diverse query–document pairs generated from multiple query personas, yielding strong gains on MTEB benchmarks [ 35 ] . More broadly, recent studies show that synthetic data can rival or even surpass human-annotated pairs in training effectiveness, making it a scalable alternative for embedding model development [ 31 ] . One limitation of synthetic data is the propensity for LLMs to output stylistically uniform text, leading to low output diversity and, therefore, low quality synthetic data. A particularly important development to address this issue is the use of persona-driven generation. By varying the "voice" or perspective of the LLM, models can be trained on a richer distribution of linguistic intents [ 11 ] . This technique has been shown to reduce bias toward a single query style and to capture edge cases that otherwise remain unrepresented [ 18 ] . Such strategies are especially valuable for domain adaptation, where the goal is to capture the full spectrum of domain-specific patterns without requiring expensive human annotation or data capture. Methodology Dataset Selection and Motivation For our experiments, we use two enterprise communication datasets: the Enron email subset of the ConcurrentQA dataset [ 3 ] and internal Valkyrie Slack messages from Q1 2025. Both corpora closely resemble enterprise-style internal communications at different scales. We exclude the Wikipedia portion of ConcurrentQA because baseline models achieved near-ceiling performance on these documents, likely due to pre-training exposure (see Appendix B ). It is worth noting that the Enron dataset is already pre-processed to work directly with most off-the-shelf embedding models. For the Slack dataset we treat each message as its own document for simplicity. Dataset Documents Avg. Length (char) Enron Email 10,077 1069.7 Slack Messages 1,678 784.0 Table 1: Dataset statistics for the enterprise communication corpora used in our experiments. Using both email and Slack data allows us to study retrieval performance across different enterprise communication modalities without introducing significant pre-processing overhead, while demonstrating the generalizability of our approach across similar but distinct workplace domains. Model Selection Table 2 summarizes the embedding models evaluated in our study. We selected BGE-M3 [ 4 ] and Qwen3-Embed-Sm [ 35 ] as primary baselines due to their strong general-domain performance, architectural differences, and high throughput without quantization. These smaller models provide an effective testbed for zero-shot performance on enterprise-style corpora and serve as efficient training targets for CustomIR fine-tuning. Model Params Embed. Dim. Pooling Prompting BGE-M3 0.6B 1024 Mean None Qwen3-Embed-Sm 0.6B 1024 Last Token Query Only Qwen3-Embed-Md 4.0B 2560 Last Token Query Only Qwen3-Embed-Lg 8.0B 4096 Last Token Query