Less Finetuning, Better Retrieval: Rethinking LLM Adaptation for Biomedical Retrievers via Synthetic Data and Model Merging Sameh Khattab, Jean-Philippe Corbeil, Osman Alperen Koraş, Amin Dada, Julian Friedrich, François Beaulieu, Paul Vozila, Jens Kleesiek Abstract Retrieval-augmented generation (RAG) has become the backbone of grounding Large Language Models (LLMs), improving knowledge updates and reducing hallucinations. Recently, LLM-based retriever models have shown state-of-the-art performance for RAG applications. However, several technical aspects remain underexplored on how to adapt general-purpose LLMs into effective domain-specific retrievers, especially in specialized domains such as biomedicine. We present Synthesize-Train-Merge (STM), a modular framework that enhances decoder-only LLMs with synthetic hard negatives, retrieval prompt optimization, and model merging. Experiments on a subset of 12 medical and general tasks from the MTEB benchmark show STM boosts task-specific experts by up to 23.5% (average 7.5%) and produces merged models that outperform both single experts and strong baselines without extensive pretraining. Our results demonstrate a scalable, efficient path for turning general LLMs into high-performing, domain-specialized retrievers, preserving general-domain capabilities while excelling on specialized tasks. 1 Introduction Retrieval-augmented generation (RAG) 36 has become a standard approach for grounding Large Language Models (LLMs), leading to improved knowledge updating and reduced hallucinations 69 ; 21 ; 2 ; 8 ; 10 . RAG systems typically rely on lexical and semantic search methods 51 ; 65 , implemented via sparse or dense retrievers. Dense retrievers based on decoder-only LLMs achieve state-of-the-art performances on embedding-related tasks 63 ; 11 . These results suggest that general-purpose LLMs already provide a strong foundation for retrieval. Figure 1: Diagram of our recipe to obtain the STM retrievers: 1) synthetic data — including 1.1) hard negative generation and 1.2) retrieval prompt optimization —, 2) LoRA fine-tuning, and 3) model merging. We segment the BMRetriever dataset into four splits: Real Medical , Synthetic Medical , NLU , and Search . However, important questions remain on how such models should be adapted, especially for domain-specific retrieval such as in the biomedical field. While zero-shot approaches 39 ; 54 reported some successes, fine-tuned methods 11 ; 46 ; 62 are at the top of the MTEB leaderboard 44 . However, certain technical aspects remain underexplored about how to best convert general-purpose LLMs into domain-specific retrievers. Previous work 71 has shown that contrastive pre-training of LLMs on a large corpus, followed by instruction tuning, yields strong dense retrieval models for the medical domain. Hard negative mining has also been shown to substantially improve retriever performance 43 ; 35 ; 52 , and 66 highlight the importance of prompts for retriever models. Despite this progress, several questions remain open: is continual pre-training or all the finetuning data necessary to obtain strong retrievers? Which subsets of the data mix are the most effective for fine-tuning? Can prompt optimization lead to further gains? Can top-tier LLMs be used to synthesize effective hard negatives? In parallel, model merging 24 has emerged as techniques to compose robust models 67 ; 5 from expert models, enabling modular development 17 and efficient data ablation 45 . Basic model-merging techniques such as ModelSoup 67 have been incorporated into the training recipes of two recent models: EmbeddingGemma 60 , and Qwen3 Embedding 76 . Nonetheless, questions remain about using model merging to build retriever models: are there clear gains compared to full fine-tuning? Which data subsets are most effective? Which merging technique offers the best performance? In this work, we present Synthesize-Train-Merge (STM), a modular framework, for enhancing LLM-based dense retrievers along three axes: synthetic hard negatives, retriever prompt optimization, and model merging. We focus on biomedical retrieval while maintaining performance on general domains. Our contributions are as follows: • We present the first systematic evaluation of two model-merging techniques for LM-based retrievers , demonstrating significant gains over fine-tuning, see Figure 2 . • We conduct a systematic study on two underexplored axes for retriever models: synthetic hard negatives , and prompt optimization . • We achieve better results with less data : no pre-training (i.e.. 11.4M to 1.4M pairs), merging 3 experts out of 4 (i.e., -29% of pairs), and fine-tuning on less than 10% of the pairs. • We release three types of artifacts: source code , improved retriever fine-tuning dataset , and STM model(s) . 1 1 1 Code, data and model(s) will be released upon acceptance. Figure 2: Performance comparison of STM Merged Models versus models fine-tuned on the combined datasets of all merged experts, across three base models, using the average NDCG@10 metric across all datasets. 2 Related Works LLM2Vec BMRetriever STM (Ours) Attention Mask Bidir. Causal Bidir. Pooling Average EOS EOS Training Setup LoRA LoRA LoRA Training Recipe MNTP + SimSCE PT + FT FT + Merging Negatives SimSCE Sampled Top-k Synthetic Dataset size 1.5M 11.4M 1.4M Domain General Biomedical General & Biomedical Table 1: Comparison of attributes between previous methods (LLM2Vec, BMRetriever) and ours. LLM2Vec is a multi-task contrastive embedding model, and BMRetriever a biomedical dense retriever pretrained with pseudo-queries (PT) and finetuned on a mix of real and synthetic labeled data with mined hard negatives (FT). 2.1 Retrievers from Decoder-Only Models E5 64 first demonstrated state-of-the-art performances on the MTEB benchmark 44 from fine-tuning Mistral 7B 30 , a decoder-only model, on real and synthetic paired retrieval data. LLM2Vec 11 showed that using bidirectional attention masking with average pooling on LLMs, and training them with masked next token prediction and SimCSE 23 led to improvements on retrieval tasks. NV-Embed 35 introduced the latent attention pooling layer during training with positive-aware hard negatives and non-retrieval task data to reach stronger performances. BMRetriever 71 leveraged a vast contrastive pre-training, followed by instruction tuning on a mix of real and synthetic datasets, yielding strong dense retrieval models for the biomedical domain. 2.2 Hard Negative Mining Hard negative mining has become a crucial design choice for training modern dense retrievers. Classical negative mining schemes ( 70 ; 75 ) showed that retrieving top-ranked passages during training can accelerate contrastive learning. Despite its limitation to exact word overlap, BM25 ( 50 ) is still widely used to surface hard negatives ( 31 ; 75 ) . NV-Retriever ( 43 ) revisits this space with positive-aware hard negative mining, which boosts the performance on the MTEB benchmark. Beyond mined hard negatives, a small but growing line of work starts to explore generated hard negatives using LLMs. SyNeg ( 37 ) uses an LLM with a multi-attribute self-reflection prompting strategy to synthesize hard negatives and combines them with retrieved negatives in a hybrid sampling scheme. Their ablation study shows that gains only arise from the hybrid method. In contrast, we show that only prompting a top-tier LLM to generate hard negatives yields substantial gains. 2.3 Model Merging Linear-mode connectivity 22 ; 42 established that independently trained solutions can be connected by low-loss paths in parameter space, motivating model combination methods based on interpolation. Building on this insight, Model Soup 67 showed that averaging multiple training checkpoints can outperform selecting a single one. These ideas naturally extend from combining checkpoints of a single model to merging distinct expert models: task arithmetic 28 formulates merging as adding and subtracting task-specific deltas from a base model, while methods such as Ties-merging 72 and DARE 74 explicitly tackle parameter interference when merging multiple experts. Recent work further shows that such parameter-level merging can be competitive with data-mixing strategies 5 ; 45 . Authors 34 ; 17 employed merging to build robust medical LLMs. EmbeddingGemma 60 and Qwen3 Embedding 76 exploit merging in their recipe without studying its impact. 2.4 Prompt Optimization Prior work has extensively studied automatic prompt optimization for LLMs ( 48 ) . Automatic Prompt Engineer (APE) successfully performs black-box optimization over generated candidate prompts from a handwritten seed prompt ( 77 ) . PromptWizard ( 3 ) and GEPA ( 4 ) extend this line of work by coordinating multiple agents or applying reflective feedback, respectively. Although Promptriever ( 66 ) shows that prompting can substantially affect embedding quality, systematic studies of prompt optimization specifically for retrievers remain limited. 3 Methods 3.1 Synthetic Data Utilization We leverage LLMs to synthetically augment existing datasets along two complementary dimensions: (1) generating synthetic hard negatives, and (2) optimizing retrieval prompts. 3.1.1 Hard Negative Generation Training dense retrievers with contrastive objectives requires negatives that are both topically related and semantically distinct from the positives. Existing hard negative mining strategies frequently struggle to balance informativeness and correctness, often introducing either trivial negatives or false negatives that are actually relevant 43 . To alleviate this issue, we employ GPT-4.1 to generate synthetic hard negatives. Given a query q , a positive passage p^{+} , and an existing mined negative p^{-} , we prompt the LLM to generate a new negative passage \tilde{p}^{-} that remains lexically and topically aligned with q while being semantically irrelevant or contradictory. The prompt provides the full context (q,p^{+},p^{-}) and explicitly instructs the model to preserve surface-level similarity while altering semantic intent. The exact prompt template is provided in Appendix B.1 . 3.1.2 Prompt Optimization Prompting plays a critical role in decoder-only embedding models 43 , as the prompt directly conditions the resulting representation space. To investigate this effect systematically, we first use the DSpy framework 32 to apply GEPA for automatically optimizing retrieval prompts. Starting from an initial prompt \pi , GEPA iteratively proposes refined prompts \pi\textquoteright aimed at improving downstream retrieval performance on a held-out validation set. We employ two instances of LLaMA-70B for this: an fp8-quantized model for prompt generation and an fp16 model for reflective evaluation. The full GEPA configuration details are reported in Table 8 of Appendix A . Second, we examine the impact of randomly sampled retrieval prompts. Using an fp8-quantized LLaMA-70B, we generate sets of 10, 20, 50, and 100 generic retrieval prompts, which are randomly assigned to queries during fine-tuning. Representative examples of both optimized and randomly generated prompts are provided in Appendix B.2 . In all prompt-based settings, prompts are prepended to queries during fine-tuning and applied consistently at inference time. Full templates and example LLM-generated prompts are provided in Appendix B.2.1 . 3.2 Instruction Fine-Tuning We fine-tune decoder-only backbone models using a contrastive learning objective to obtain dense retrievers. We adopt the InfoNCE loss 26 , which encourages each query embedding to be closer to its corresponding positive passage than to all other passages in the batch and to any provided hard negative passage. Formally, given a batch of N triplets \{(q_{i},p_{i}^{+},p_{i}^{-})\}_{i=1}^{N} , the loss term for a given query q_{i} is defined as: \mathcal{L}_{i}=-\log\frac{e^{\mathrm{sim}(q_{i},p_{i}^{+})}}{e^{\mathrm{sim}(q_{i},p_{i}^{-})}+\sum_{j=1}^{N}e^{\mathrm{sim}(q_{i},p_{j}^{+})}}, where \mathrm{sim}(\cdot,\cdot) denotes cosine similarity between embeddings, and the denominator includes two terms: the first one uses a hard negative p_{i}^{-} , and the second one employs other passages p_{j}^{+} for j\neq i as in-batch negatives. We average over the loss terms in the batch to obtain the total loss. 3.2.1 Expert Model Fine-Tuning To enable modular specialization, we fine-tune multiple expert retrievers on different, coherent subsets of the BMRetriever fine-tuning dataset 71 : medical synthetic , medical real , natural language understanding (NLU), and search . The medical real subset includes sentence-level biomedical inference and similarity datasets such as MedNLI 53 and Medical Question Pairs 41 , as well as passage-level biomedical QA benchmarks including MEDIQA 12 , medical StackExchange QA 55 , and medical dialogue data 38 . The medical synthetic subset consists of LLM-generated biomedical retrieval pairs. To improve general-domain relevance modeling, the dataset further incorporates NLU benchmarks such as Natural Questions 33 , FEVER 59 , ELI5 20 , SNLI 14 , and the MS MARCO passage ranking dataset 9 . We train four experts per backbone model, each emphasizing a distinct data composition or training configuration (e.g., synthetic hard negatives, prompt optimization). 3.3 Model Merging Model merging aims to combine multiple expert retrievers into a single model that inherits complementary strengths without additional training. Given a set of expert models \{M_{k}\}_{k=1}^{K} with parameters \{\theta_{k}\} , merging methods compute a unified model \hat{M} by operating directly in parameter space. In the simplest case, linear merging 67 is defined as: \theta_{\hat{M}}=\sum_{k=1}^{K}\alpha_{k}\theta_{k}, (1) where \alpha_{k} are the weight coefficients with 0\leq\alpha_{k}\leq 1 . The task arithmetic approach 28 relies instead on a linear combination of task vectors \tau_{k}=\theta_{k}-\theta_{B} , which is the delta of parameters between the k^{th} expert and the parameters of the base model \theta_{B} . The merged model becomes \theta_{\hat{M}}=\theta_{B}+\sum_{k=1}^{K}\alpha_{k}\tau_{k}, (2) Ties merging 72 also leverages task vectors \tau_{k} with two strategies to mitigate the task-interference phenomenon: keeping only high-magnitude parameter changes by introducing a second parameter named density \delta_{k}\in[0,1] , and the sign agreement algorithm which is a majority vote on the signs across all \tau_{k} . 4 Experiments 4.1 Datasets For continual pre-training experiments, we follow the BMRetriever setup 71 in which they employ large-scale unlabeled biomedical and scientific corpora. For fine-tuning, we also use their fine-tuning data mixture. We separate it into four coherent subsets as shown in Table 2 . Medical General Split Med-Synth Med-Real Search NLU #Pairs 431,000 306,000 438,000 251,000 Table 2: Pair counts for our custom splits of the BMRetriever fine-tuning dataset, comprising four splits: two in the medical domain and two in the general domain. 4.2 Training Setup We experiment with three decoder-only backbone architectures of varying sizes; Qwen3 73 , Gemma 56 , and Phi-4 1 ; the configurations are summarized in Table 3 . Qwen3 Gemma Phi4 mini instruct Parameters 0.6B 2B 3.8B Dimensions 1,024 2,048 3,072 Table 3: Backbone model used for expert training. Parameters indicate the total model size, and Dimensions refers to the hidden size. Qwen3 0.6B Gemma 2B Phi4 3.8B Variant Med- Synth Med- Real Search NLU Avg Med- Synth Med- Real Search NLU Avg Med- Synth Med- Real Search NLU Avg FT 0.583 0.568 0.506 0.546 0.551 0.581 0.591 0.503 0.520 0.549 0.611 0.611 0.497 0.585 0.576 FT SHN 0.592 0.583 0.557 0.563 0.574 0.598 0.575 0.508 0.559 0.560 0.605 0.600 0.591 0.566 0.590 FT PO 0.594 0.568 0.566 0.583 0.578 0.599 0.569 0.567 0.613 0.587 0.622 0.574 0.614 0.619 0.607 FT SHN+PO 0.586 0.587 0.571 0.555 0.575 0.588 0.549 0.511 0.569 0.554 0.607 0.608 0.599 0.575 0.597 Table 4: NDCG@10 scores of experts averaged over 12 MTEB tasks. Each expert is trained on one of the four subsets of the BMRetriever fine-tuning dataset. FT refers to fine-tuning on the corresponding data subset. SHN denotes fine-tuning with synthetic hard negatives, PO applies the best-performing prompt optimization per model family, and SHN+PO combines both. Bold and underlined entries indicate the best and second-best performance within each expert column. Following prior work 11 ; 35 , we disable causal attention masking during fine-tuning to enable bidirectional attention. In preliminary experiments, we consider both EOS-token pooling and mean pooling strategies. However, we observe that the former yields slightly stronger performance. Therefore, we adopt EOS pooling for all reported results. During fine-tuning, retrieval prompts are prepended to each query, and models are trained using the InfoNCE loss. We fine-tune all models using LoRA adapters 27 applied to all linear layers following prior works 11 ; 35 ; 71 . Hyperparameters, including learning rate, batch size, number of steps, and LoRA configuration, are summarized in Table 9 of Appendix A . We conduct fine-tuning under several configurations. We begin with standard fine-tuning on the base dataset. We then fine-tune models on datasets augmented with synthetic hard negatives, followed by datasets generated using optimized prompts. Finally, we evaluate a configuration that combines synthetic hard negatives with the best optimized prompts. 4.3 Model Merging We perform model merging using MergeKit 24 , which supports parameter-space merging for HuggingFace-compatible models. We evaluate two merging strategies: linear interpolation 67 and Ties merging 72 . We select the best merged models following a grid search approach. For linear merging, we sweep weight coefficients \alpha\in\{0,0.1,\dots,0.9\} . For Ties merging, we sweep the weight coefficients over the same range, and vary the density parameter over \rho\in\{0.1,0.2,\dots,0.9\} . All merged models are evaluated without further training on development sets from four BEIR benchmark datasets 57 : NFCorpus 13 , FiQA-2018 57 , Quora 18 , and DBPedia 25 . We use the official dev splits and evaluate performance using NDCG@10 and Recall@10 metrics. To enable scalable evaluation across numerous merged models, we evaluate on a sampled subset of queries and documents for each dataset. 4.4 Baselines We compare our models against a diverse set of retrieval baselines, including BM25 40 , Contriever 29 , E5-v2 62 , GTR 47 , LLM2Vec 3B 11 , and BMRetriever 2B 71 . Baselines are selected to match our models in parameter scale or architectural family. 4.5 Evaluation We evaluate retrieval performance on the English medical subset of the Medical MTEB benchmark 44 ; 19 , which includes TREC-COVID 49 , SciFact 16 , NFCorpus 13 , Cure 7 , PublicHealthQA 68 , and MedicalQA 6 . To assess general-domain generalization, we additionally evaluate on five general-domain MTEB datasets, including FiQA 57 , ArguAna 61 , SciDocs 15 , and two NanoMTEB subsets (FEVER 58 and Quora 18 ). We use the official MTEB evaluation pipeline 44 , and report nDCG@10 as the evaluation metric. We average the scores across three possible splits: medical subset, general subset, and all datasets. At evaluation time, we use the same retrieval prompt for all the models finetuned with instructions. 5 Results and Analysis 5.1 Pre-training Figure 3: Performance averages of three base models pre-trained (PT) and/or fine-tuned (FT) on the BMRetriever datasets with 10M and 1.4M samples, respectively. We displayed in Figure 3 results comparing pretraining (PT) and fine-tuning (FT) setups. We observe that pretraining on 10M unlabeled pairs underperforms models fine-tuned only on 1.4M pairs, despite BMRetriever 71 previously benefitting from a pretraining phase. We therefore drop the pretraining step and use fine-tuning only in the following experiments. 5.2 Fine-Tuning Experts with Synthetic Data 5.2.1 Prompt Optimization Method Qwen3 0.6B Gemma 2B Phi4 3.8B FT-all 0.599 0.600 0.621 FT-all GP10 0.581 † 0.557 0.604 FT-all GP20 0.571 0.556 0.603 FT-all GP50 0.575 0.560 0.604 † FT-all GP100 0.580 0.555 0.604 FT-all GEPA-l 0.559 0.571 0.592 FT-all GEPA-m 0.563 0.577 † 0.595 FT-all GEPA-h 0.560 0.568 0.597 Med-Real 0.568 0.591 0.611 Med-Real PO 0.568 0.569 0.574 Med-Synth 0.583 0.581 0.611 Med-Synth PO 0.594 0.599 0.622 Search 0.506 0.503 0.497 Search PO 0.566 0.567 0.614 NLU 0.546 0.520 0.585 NLU PO 0.583 0.613 0.619 STM {}_{\text{Ties}} 0.615 0.619 0.643 STM {}_{\text{Linear}} 0.616 0.622 0.646 † Highest-performing prompt optimization for this base model. Table 5: Prompt Optimization results across models (average NDCG@10 over 12 MTEB tasks). Bold and underlined entries indicate the best and second-best performance within each backbone group. See full Table 10 in Appendix C . We displayed results for two prompt optimization techniques (generic prompts 10/20/50/100 and GEPA light/medium/heavy) in Table 5 . While fine-tuning on all the dataset is surpassing optimized FT-all variants, we observe substantial improvements when applying the top-performing technique per model architecture (noted by † in the first section of the table) at the expert level (noted by the PO in the second section) of the table. The gains are more considerable for the non-medical experts, while the Med-Real expert without prompt optimization remains superior. We carry over the best prompt optimization technique per model in the next section. 5.2.2 Expert Optimization In Table 4 , both SHN and PO consistently improve retrieval performance compared to standard fine-tuning when averaged over all experts. PO yields the strongest overall gains — average improvement of 5-7% over FT. This suggests that prompt-level adaptations are generally effective strategy across heterogeneous retrieval settings. However, we observe that combining SHN and PO does not reliably outperform SHN alone. In fact, except for the Med-Real and Search experts within the Qwen family, the SHN+PO variant consistently ranks below the PO-only approach, indicating that the benefits of SHN and PO are not additive and may partially interfere. Looking at individual experts, the largest relative gains are observed for the Search expert, particularly for Phi-4 Mini, where PO achieves a +23.5% relative improvement over standard fine-tuning. The second-largest improvement is seen for the NLU expert of the Gemma family, with a +17.9% relative gain. More generally, PO consistently delivers larger relative improvements for general domain experts (Search & NLU) than for medical domain experts. In contrast, SHN tends to degrade performance for medical experts as model size increases, most notably for Med-Real retrieval. This trend suggests that high-quality medical training data may already contain sufficiently challenging negatives, which larger models are better able to exploit. Overall, while the results are not consistent, prompt optimization emerges as the more robust and effective method compared to SHN. SHN can provide gains in selected settings, but its impact is model- and task-dependent, and its combination with PO rarely yields additional benefits. 5.3 Model Merging Model Avg Medical Avg General Avg All Qwen3 0.6B Med-Real SHN+PO 0.613 0.551 0.587 Med-Synth PO 0.623 0.553 0.594 Search SHN+PO 0.597 0.535 0.571 NLU PO 0.606 0.550 0.583 FT-all 0.633 0.551 0.599 FT-all SHN 0.632 0.555 0.600 STM_{Qwen3-Ties} 0.637 0.585 0.615 STM_{Qwen3-Linear} 0.638 0.585 0.616 Gemma 2B Med-Real 0.625 0.542 0.591 Med-Synth PO 0.625 0.564 0.599 Search PO 0.577 0.554 0.567 NLU PO 0.647 0.566 0.613 FT-all 0.638 0.548 0.600 FT-all SHN 0.637 0.561 0.605 STM_{Gemma-Ties} 0.651 0.576 0.619 STM_{Gemma-Linear} 0.654 0.577 0.622 Phi4 3.8B Med-Real † 0.643 0.567 0.611 Med-Synth PO 0.654 0.577 0.622 Search PO 0.636 0.583 0.614 NLU PO 0.647 0.580 0.619 FT-all 0.655 0.573 0.621 FT-all SHN 0.661 0.585 0.629 STM_{Phi4-Ties} 0.669 0.606 0.643 STM_{Phi4-Linear} 0.677 0.603 0.646 † For the STM Linear Merge, the PO variation was used. Table 6: Best-performing experts and their STM-merged results (average NDCG@10 across 12 MTEB tasks). \text{FT-all}_{SHN} indicates the model finetuned on the full dataset along with synthetic hard negatives. Bold indicates the best average, underline the second best within each backbone group. As summarized in Table 6 , linear interpolation consistently yields the strongest performance when combining the four individual experts across all model families. Linear merging slightly but consistently outperforms Ties merging, indicating that simple weighted interpolation is sufficient to effectively integrate complementary expert representations. Overall, merged models uniformly outperform fully fine-tuned counterparts across all backbones as shown in Figure 2 as well. We note that these gains are consistent across model sizes. When compared against the strongest individual expert, merged models also achieve superior performance. In all three model families, the linear-merged STM surpasses the best-performing single expert, confirming that merging captures complementary strengths across domain-specialized experts rather than amplifying a single dominant expert. 5.4 Comparison with Prior Retrievers Model Size Avg Med Avg General Avg All BM25 - 0.532 0.515 0.525 Contriever 150M 0.508 0.533 0.519 E5 Large V2 335M 0.654 0.576 0.622 GTR T5 XL 1.2B 0.581 0.586 0.583 BMRetriever 2B 0.645 0.560 0.609 LLM2Vec 3B 0.635 0.597 0.619 STM_{Qwen3} 0.6B 0.638 0.585 0.616 STM_{Gemma} 2B 0.654 0.577 0.622 STM_{Phi4} 3.8B 0.677 0.603 0.646 Table 7: Summary of retrieval performance (averages) against base lines across medical and general domains. Bold indicates the best average, underline the second best. Full results are reported in Appendix C . We further evaluate the best merged STMs along baselines from the literature on our targeted 12 datasets from MTEB. As shown in Table 7 , STM_{Phi4-Linear} achieves the strongest performance across both medical and general tasks, outperforming all baselines. In particular, it surpasses BMRetriever_{2B} and LLM2Vec , demonstrating that expert merging scales effectively to multi-domain retrieval and remains competitive with state-of-the-art retrievers trained on large and diverse corpora. Notably, STM_{Gemma-Linear} also delivers strong performance despite its smaller model size. It consistently outperforms BMRetriever_{2B} , which shares the same base model. These results highlight the efficiency of the proposed approach without relying on larger backbones or additional pre-training. 5.5 Merging Coefficients We provide the merging weight coefficients 2 2 2 We ignore the density coefficients for the Ties method in this analysis since the weight coefficients modulate directly the final amplitude of that expert in the merged models. in Figure 5 of Appendix C for each model. We notice similar coefficients for Qwen3 and Phi4 in contrast to the ones used for Gemma. Qwen3 and Phi4 did not use the Search expert at all to build their respective STM final models, and both utilize with higher amplitudes the medical experts along with the NLU expert at a weight of 0.5. For Gemma, the weight coefficients tend to be lower than 0.5 with no use of Med-Real or Med-Synth experts for linear merging or Ties, respectively. Overall, Ties-merged optimal models have lower coefficients compared to the linear merging ones, but coefficients of both methods are correlated. From analyzing the weight coefficients in terms of data ablation, we note that generally all optimal configurations for each model remove one expert. Thus, we infer that removing one of the expert could reduce the overall data budget from 18% for the NLU subset up to 29% for the Search subset out of the 1.4M available pairs. 5.6 Training Data Size Considerations Figure 4: Performance averages across 3 runs of three base models fine-tuned on three different sample sizes of the BMRetriever dataset. Standard deviations are not displayed since they are below 0.01. Ablations on dataset sizes visualized in Figure 4 reveal clear patterns. Models’ performances averaged across 3 runs saturate at around 100K samples outperforming those trained on the full 1.4M samples across all three base models. Therefore, curated high-quality data can be more effective than large-scale datasets; in line with the trends of the pretraining and merging coefficient results in sections 5.1 and 5.5 , respectively. While experiments of previous sections did not leverage this finding, future works could further explore this direction. 6 Conclusion We presented Synthesize-Train-Merge (STM), a modular framework for adapting decoder-only LLMs into effective dense retrievers for domain-specific tasks. By combining synthetic hard negatives, retrieval prompt optimization, and model merging, STM improves task-specific experts by up to 23.5% and produce unified models that outperform both individual experts and baselines fine-tuned on the experts datasets combined. Our results show that careful dataset selection and modular merging can yield strong retrieval performance without extensive pre-training or larger backbones. These findings suggest a scalable, efficient path for adapting LLMs to specialized retrieval tasks while maintaining general-domain generalization. Limitations Despite strong empirical results, our study has several limitations. First, we only explore two merging strategies (linear interpolation and Ties); more adaptive or task-aware merging approaches could provide further gains but are beyond the scope of this work. Second, our synthetic hard negative generation and prompt optimization rely on large LLMs, adding computational cost and potential sensitivity to the choice of generator model. We do not evaluate robustness across different LLMs or prompt variants. Appendix A Implementation Details Parameter Value LLM Configuration Prompt Generation Model LLaMA 70B (FP8) Reflection Model LLaMA 70B (FP16) Temperature 0.7 Max Tokens 1000 GEPA Hyperparameters Budget Level Auto (Light / Medium / Heavy) Training Examples 100 / 200 / 300 Evaluation Metric NDCG@10 Validation Examples 100 / 200 / 300 Reflection Minibatch Size 3 / 5 / 8 Number of Threads 1 Table 8: GEPA configuration details. Hyperparameter Maximum Tokens 512 Optimizer AdamW LR Scheduler Linear Warmup # Warmup Steps 100 bf16 True Learning Rate Phi4 mini instruct (3.8B) 2\ \times\ 10^{-5} Gemma (2B) 1\ \times\ 10^{-5} Qwen3 (0.6B) 5\ \times\ 10^{-5} # Epochs 1 LoRA Rank 16 LoRA \alpha 32 Total Batch Size Phi4 mini instruct (3.8B) 64 Gemma (2B) 128 Qwen3 (0.6B) 256 Table 9: Fine-tuning Hyperparameters. Appendix B Prompt Examples B.1 Hard Negative Generation Prompts The following prompt template was used for hard negative mining: Hard Negative Generation Prompt Given this query: "{query}" Original prompt: "{original prompt}" Original positive document: "{positive doc}" Original negative document: "{original negative}" Generate a HARD negative document that is: 1. Related to the same domain (extract domain from the query) 2. Contains similar terminology and concepts as the positive document 3. But is NOT relevant to answering the specific query 4. Should be moderately challenging to distinguish from the positive document 5. Should be a realistic document in the domain 6. Should be harder than the original negative document but not as hard as a super hard negative IMPORTANT: Return ONLY the document text. Do not include any introductory text, explanations, summaries, or meta-commentary. Just return the raw document content. B.2 Optimized Prompts B.2.1 Generic Prompts We generated random generic prompts using the following template. Generic Prompts Generation Prompt You are an expert in information retrieval prompt engineering. Generate a creative and diverse prompt specifically for DOCUMENT RETRIEVAL tasks. The prompt should help a retrieval model find relevant documents for any query. Use a different style/approach from these examples: - Direct retrieval instruction format - Document ranking format - Relevance scoring format - Query-document matching format - Information seeking format - Context-aware retrieval format - Domain-specific retrieval format - Simple document finding format Make it unique and varied. The prompt should be effective for DOCUMENT RETRIEVAL. Generate only the prompt text, no explanations, do not specify any domain or document type to avoid confusing the model: Generic Retrieval Prompts for General and Medical Domains Baseline (all variations) General Domain (MS MARCO): “Given a web search query, retrieve relevant passages that answer the query.” Medical Domain (Synthetic): “Given a query, find articles that discuss the correlation between a specific lifestyle factor and a disease.” 10 Generic prompts (all domains) “Imagine you’re a curator of a vast library, tasked with uncovering hidden gems that shed new light on a specific topic of interest. Given the query, navigate through the shelves of knowledge to gather a collection of documents that not only resonate with the inquiry but also offer diverse perspectives, insightful analysis, and thought-provoking discussions. The goal is to assemble a comprehensive anthology that enriches understanding, sparks curiosity, and fuels further exploration, ensuring that every included document contributes a unique voice to the chorus of knowledge on the subject at hand.” 20 Generic prompts (all domains) “Imagine you’re a librarian tasked with curating a personalized bookshelf for a curious reader. Given the topic of interest, navigate through a vast library and handpick a selection of texts that would spark fascinating discussions, provide insightful knowledge, and resonate deeply with the reader’s query, as if you were recommending books to a close friend.” 50 Generic prompts (all domains) “Imagine you’re a librarian tasked with curating a personalized anthology for a curious reader; given the threads of inquiry woven into the phrase, navigate the vast expanse of written works to unearth the most enlightening and informative texts that intricately weave together concepts, ideas, and narratives, and present a collection that not only resonates with the essence of the inquiry but also expands its boundaries, fostering a deeper understanding and sparking further exploration.” 100 Generic prompts (all domains) “Imagine you’re a librarian tasked with uncovering hidden gems in a vast archive, and someone has whispered a cryptic clue in your ear; given this whispered clue, what documents would you pull from the shelves to unravel the mystery, and what threads of connection would you follow to weave together a tapestry of insight and understanding?” B.2.2 GEPA-Optimized Retrieval Prompts GEPA-Optimized Retrieval Prompts for General and Medical Domains Baseline (all budgets) General Domain (MS MARCO): “Given a web search query, retrieve relevant passages that answer the query.” Medical Domain (Synthetic): “Given a query, find articles that discuss the correlation between a specific lifestyle factor and a disease.” GEPA-Light Budget General Domain (MS MARCO): “Design an efficient algorithm to retrieve relevant textual passages from the web that directly answer a given search query, considering factors such as query clarity, contextual relevance, and the specificity of the information sought, and optimize the search results based on the type of information required, such as factual, explanatory, or comparative analysis, to achieve high scores in evaluation metrics like NDCG@10, MAP@10, and Recall@10.” Medical Domain (Synthetic): “Retrieve recent, high-quality Pubmed passages that directly answer the given biomedical question, prioritizing peer-reviewed articles from reputable journals published within the last five years, and considering specific contexts such as demographics or health conditions if applicable.” GEPA-Medium Budget General Domain (MS MARCO): “Retrieve factual and explanatory passages from reputable online sources that directly answer the given web search query, ensuring the passages include specific keywords related to the query topic, are relevant to the context or domain of interest, and provide a clear, concise, and relevant response that matches the desired type of answer.” Medical Domain (Synthetic): “Retrieve recent, peer-reviewed Pubmed articles or passages that directly answer the given biomedical question, focusing on high-quality studies published within the last five years, and provide abstracts or summaries of these articles in the search results.” GEPA-Heavy Budget General Domain (MS MARCO): “Design an efficient algorithm to retrieve relevant textual passages from the web that directly answer a given search query, considering factors such as query clarity, contextual relevance, and the specificity of the information sought.” Medical Domain (Synthetic): “Retrieve recent, relevant Pubmed passages that directly answer the given biomedical question, prioritizing articles from reputable journals published within the last five years, and providing accurate and up-to-date information on the topic.” Appendix C Detailed Results Method Qwen3 0.6B Gemma 2B Phi4 3.8B Avg Med Avg Gen Avg All Avg Med Avg Gen Avg All Avg Med Avg Gen Avg All Med-Real 0.609 0.510 0.568 0.625 0.542 0.591 0.643 0.567 0.611 Med-Real PO 0.581 0.550 0.568 0.586 0.545 0.569 0.566 0.585 0.574 Med-Synth 0.614 0.540 0.583 0.606 0.546 0.581 0.642 0.567 0.611 Med-Synth PO 0.623 0.553 0.594 0.625 0.564 0.599 0.654 0.577 0.622 Search 0.527 0.477 0.506 0.523 0.475 0.503 0.515 0.472 0.497 Search PO 0.583 0.542 0.566 0.577 0.554 0.567 0.636 0.583 0.614 NLU 0.573 0.509 0.546 0.546 0.484 0.520 0.628 0.525 0.585 NLU PO 0.606 0.550 0.583 0.647 0.566 0.613 0.647 0.580 0.619 FT-all 0.633 0.551 0.599 0.638 0.548 0.600 0.655 0.573 0.621 FT-all GP10 0.609 0.543 0.581 † 0.579 0.526 0.557 0.641 0.552 0.604 FT-all GP20 0.595 0.537 0.571 0.576 0.530 0.556 0.643 0.547 0.603 FT-all GP50 0.603 0.535 0.575 0.586 0.524 0.560 0.642 0.552 0.604 † FT-all GP100 0.608 0.541 0.580 0.575 0.528 0.555 0.641 0.551 0.604 FT-all GEPA-l 0.592 0.513 0.559 0.607 0.521 0.571 0.632 0.537 0.592 FT-all GEPA-m 0.600 0.510 0.563 0.608 0.533 0.577 † 0.631 0.545 0.595 FT-all GEPA-h 0.594 0.513 0.560 0.600 0.523 0.568 0.636 0.544 0.597 STM_{\text{Ties}} 0.637 0.585 0.615 0.651 0.576 0.619 0.669 0.606 0.643 STM_{\text{Linear}} 0.638 0.585 0.616 0.654 0.577 0.622 0.677 0.603 0.646 † Highest-performing prompt optimization for this base model; used as the main prompt for experts. Table 10: Prompt Optimization results across the base models (average NDCG@10 over 12 MTEB tasks). Medical General Model Size Feedback QA Medical QA CUREv1 Public HealthQA NF Corpus TREC COVID Sci Fact SCI DOCS Nano FEVER Argu Ana Nano Quora FiQA 2018 Avg Med. Avg Gen. Avg All BM25 - 0.563 0.458 0.355 0.718 0.321 0.623 0.686 0.158 0.809 0.492 0.863 0.251 0.532 0.515 0.525 Contriever 150M 0.505 0.592 0.351 0.694 0.313 0.448 0.655 0.171 0.794 0.484 0.944 0.274 0.508 0.533 0.519 E5 Large V2 335M 0.704 0.699 0.562 0.856 0.372 0.666 0.722 0.205 0.889 0.464 0.912 0.411 0.654 0.576 0.622 GTR T5 XL 1.2B 0.577 0.692 0.507 0.713 0.333 0.601 0.642 0.157 0.846 0.528 0.957 0.442 0.581 0.586 0.583 BMRetriever 2B 0.587 0.727 0.471 0.812 0.347 0.839 0.729 0.186 0.940 0.356 0.960 0.357 0.645 0.560 0.609 LLM2Vec 3B 0.708 0.731 0.490 0.814 0.385 0.572 0.746 0.190 0.864 0.553 0.953 0.423 0.635 0.597 0.619 STM_{Qwen3} 0.6B 0.681 0.697 0.487 0.819 0.340 0.761 0.681 0.190 0.865 0.548 0.967 0.354 0.638 0.585 0.616 STM_{Gemma} 2B 0.621 0.717 0.503 0.864 0.368 0.793 0.715 0.201 0.898 0.479 0.969 0.338 0.654 0.577 0.622 STM_{Phi4} 3.8B 0.697 0.732 0.531 0.860 0.382 0.791 0.744 0.214 0.853 0.562 0.969 0.414 0.677 0.603 0.646 Table 11: Comparison of retrieval performance on English tasks from MTEB. Results are reported primarily on the medical subset, with additional evaluation on five general-domain subsets. Performance is measured using NDCG@10. Figure 5: Merging weight coefficients for each expert for Linear and TIES techniques for each model.