One prompt is not enough: Instruction Sensitivity Undermines Embedding Model Evaluation
Yevhen KostiukAarhus University [email protected], Kenneth EnevoldsenAarhus University [email protected] · 2026
Read One prompt is not enough: Instruction Sensitivity Undermines Embedding Model Evaluation by Yevhen KostiukAarhus University [email protected], Kenneth EnevoldsenAarhus University [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-22544.txt.
Introduction
Kostiuk and Enevoldsen study prompt effects in task vector models, each tested with a single prompt. Single-point testing misses task phrasing effects. Across 6 models and 11 datasets, shown scores skew score spreads over likely prompts. The default prompt can lowball or hype scores. Rankings are not robust to prompt picks: coders can lift rank by cherry-picking prompts, and under rigged picks any model can reach first place. Single-prompt testing is too thin; benchmarks should use many prompts or report effects with point scores.
Catalog record
| CREATOR | Yevhen KostiukAarhus University [email protected], Kenneth EnevoldsenAarhus University [email protected] |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 34164 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.22544
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
One prompt is not enough: Instruction Sensitivity Undermines Embedding Model Evaluation Yevhen KostiukAarhus University [email protected], Kenneth EnevoldsenAarhus University [email protected] Abstract 0 0 footnotetext: ∗ Equal contribution. Instruction embedding models have become common among state-of-the-art models, however are evaluated using a single prompt per task. The single-point evaluation ignores a main problem of the instruction-based approach namely: sensitivity to the phrasing of the instruction. We present an empirical study of prompt sensitivity across 6 embedding models, 11 datasets, and 15 task-specific prompts per dataset, a total of 990. We show that reported scores misrepresent the distribution of scores over plausible prompts. The default prompt can both systematically understate or overstate performance. Furthermore, we show that the leaderboard ranking is not robust to prompt selection: by choosing prompts favorably, any model in our study can be promoted to first place. Our findings suggest that single-prompt evaluation is insufficient for instruction-tuned embedding models and that benchmarks should incorporate prompt robustness, either by evaluating over multiple prompts or by reporting sensitivity alongside point estimates. The code can be found in our GitHub repository 1 1 1 https://github.com/centre-for-humanities-computing/instruction-sensitivity-evaluation . 1 Introduction and Background Embeddings are foundational to modern NLP applications, serving as the backbone for tasks such as Retrieval-Augmented Generation (RAG), semantic search, zero-shot clustering as well as newer tasks such as agentic memory 15 ; 33 . Recently, instruction-tuned embedding models initially popularized by 25 have become common among top-performing embedding models such as Qwen or the e5-series 35 ; 30 ; 7 . Instructions (or prompts) enables models to capture task-specific nuances of the text, offering higher flexibility for various domains and objectives. However, while instructions allow for a high degree of user-flexibility they also allow instructions to be tuned to obtain better benchmark performance, thus posing a fundamental challenge of how to best measure, report and compare such models that do not oversell nor undersell their capabilities. Figure 1: Overview of our approach , showing the currently unreported distribution of performances over reasonable prompts against the point estimate that is the reported results. Below that we show three possible option, either the reported result is representative of the distribution, it is lower than the (prompt deflation) or higher than (prompt inflation). Figure 2: Example of prompt deflation and inflation on a sample of evaluated tasks and models. For all the scores and distributions see Appendix A . The most common benchmarks for evaluating the quality of embedding are implemented using the beir for information retrieval 27 ; 12 and multitask mteb frameworks, which includes popular benchmarks such as MTEB, MMTEB 18 ; 7 . mteb evaluate instruction-based models using only a single prompt either supplied by the task or the model contributor. This approach ignores a defining characteristic of these models: that they are sensitive to phrasing of the instruction. Relying on a single prompt risks misrepresenting actual model performance. Furthermore, it is possible to overwrite the prompt within the models implementation – to allow customization to a specific prompt scheme – making it possible for the model developers to selectively choose the best performing prompts for a given task. Current research on prompt sensitivity is focused mostly on generative LLMs 29 ; 14 ; 22 ; 17 , while instruction-based embedding models remains largely underexplored. While recent efforts have addressed overfitting via paraphrases 8 , no work systematically examines the effect of prompt variation on embedding evaluation. Our work bridges this gap by isolating the prompt as the primary variable to quantify its impact on reported performance. In this paper, we question the validity of single-prompt evaluation for instruction-tuned embedding models. We investigate the following research questions: ( RQ1 ) To what extent are instruction-tuned embedding models sensitive to variations of prompts relevant to the task? ( RQ2 ) How do these variations influence the reported benchmark rankings? ( RQ3 ) To what degree is benchmark hacking facilitated by prompt hacking 21 ? To answer these questions, we conduct an empirical study across 11 diverse tasks types including retrieval, classification, clustering, and semantic textual similarity (STS). For each task, we synthesize a set of 15 task-specific prompts and evaluate a suite of different open-weight embedding models, which resulted in 165 prompts. Our findings reveal that reported benchmark scores frequently misrepresent expected performance: the default prompt can act as an outlier, artificially deflating or inflating a model’s perceived quality ( Figure 1 ). More critically, we show that by adversarially selecting prompts, any model in our study can be made to appear as the top-ranked model on a simulated leaderboard. We recommend that benchmarks transition from single-point estimates to multi-prompt evaluation — reporting score distributions or robustness metrics alongside point estimates — and that model developers either optimize for prompt robustness or adopt limited prompt schemes such as those used by the Jina v5 series 1 . 2 Methodology To test the influence of the prompts on the benchmark performance, we selected the following diverse list of tasks and datasets from mteb 2 2 2 We use the dataset name in mteb to ensure reproducibility. : retrieval (MIRACLRetrievalHardNegatives.v2, Touche2020Retrieval.v3, FEVERHardNegatives 34 ; 26 ; 28 ), classification (TweetSentimentClassification, ImdbClassification, AmazonCounterfactualClassification 4 ; 16 ; 20 ), clustering (MedrxivClusteringP2P.v2, StackExchangeClustering.v2 7 ; 9 ), and semantic similarity (STS15, STS14, STS22.v2 5 ; 3 ; 6 ). We chose tasks either derived from MTEB 18 or MMTEB 7 . To allow for comparison to the English only BGE models we only evaluate on the English subsets. For each task, we generated a list of 15 prompts with gpt-oss-120b 19 with vLLM 13 structured outputs with the default parameters. The model was instructed to generate prompts based on the combination of language, task description, and task name. The prompts were manually evaluated to ensure that they were coherent and relevant for task (see Appendix C , and Appendix D for default prompts). The prompts generation was set up to resemble a real use-case, where the practitioner will rely on a custom, task-oriented prompt. We use synthetic prompt generation to reflect real practitioner behaviour and to avoid author bias in the reported effects. The resulting prompts are short (mean 5.24 words, std 2.33), consistent with documented model usage examples. We validated that synthetic and human-written prompts produce largely comparable score distributions, and that where differences exist they do not affect our conclusions (see Appendix C ). We seek to evaluate a broad selection including both multilingual models: Qwen3-Embedding-0.6B 35 , multilingual-e5-large-instruct 30 , KaLM-embedding-multilingual-mini-instruct-v2.5 11 , as well as English models across a variety of sizes; bge-small-en-v1.5, bge-base-en-v1.5, and bge-large-en-v1.5 32 . We ran the models on each task for each of the task-specific prompts, and compared the results with the reported scores on mteb . Models revisions and versions are available in Appendix B . 3 Analysis and Discussion Figure 3: Prompt sensitivity : Coefficient of variance distributions across the models and prompts. We see that certain models have a high degree of variance across prompts. 3.1 Prompt Inflation and Deflation We examine whether the reported single-prompt results are representative of a model’s expected performance. In the Appendix A ( Figure 6 ), the score distributions across the prompts for each model/task combination are presented,