The Harder Text Embedding Benchmark (HTEB): Beyond One-dimensional Static Robustness
Manuel Frank, Haithem Afli · 2026
Read The Harder Text Embedding Benchmark (HTEB): Beyond One-dimensional Static Robustness by Manuel Frank, Haithem Afli (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-28190.txt.
Introduction
Manuel Frank and Haithem Afli argue that embedding robustness is multidimensional, not a single static score. Their Harder Text Embedding Benchmark perturbs inputs at test time along lexical, length, and language axes, validated by thousands of human ratings. Scale raised absolute scores without closing the original-transformed gap, and English sets proved more sensitive than multilingual ones.
Catalog record
| CREATOR | Manuel Frank, Haithem Afli |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 83008 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.28190
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
The Harder Text Embedding Benchmark (HTEB): Beyond One-dimensional Static Robustness Manuel Frank, Haithem Afli Abstract Embedding benchmarks like MTEB report a single score per model, implicitly treating robustness as a static, scalar property. We argue that embedding robustness is multidimensional, since models respond differently to different types of variation, and requires dynamic evaluation to expose failures hidden by static benchmarks. We introduce the Harder Text Embedding Benchmark (HTEB), a dynamic evaluation framework that challenges model robustness along three practically interpretable axes (Lexical/Stylistic, Length and Language) by stochastically transforming inputs at evaluation time with an LLM. Evaluating 16 open-weight embedding models on 32 datasets covering 42 languages under transformations validated by 4,800 human ratings on an English subsample, we find three patterns: (1) Models exhibit specific, partly decoupled robustness profiles across axes. (2) Across three model families, scale increases absolute scores but does not close the gap between original and transformed evaluations. Here, scaling tends to improve specifically the Language axis. (3) English datasets are more sensitive to HTEB transformations than multilingual datasets. This demonstrates that HTEB identifies strengths and weaknesses of models along deployment-relevant axes, challenging current embedding benchmarks and arguing for multidimensional, dynamic robustness evaluation. 1 Introduction Text embeddings are commonly evaluated with benchmarks such as MTEB ( Muennighoff et al. 2023 ) and MMTEB ( Enevoldsen et al. 2025 ) . While benchmark limitations have a long history in NLP ( Dhar and Søgaard 2026 ) , recent work raises specific concerns about the robustness and generalisability of static embedding benchmarks ( Assadi et al. 2026 ; Goel et al. 2025 ; Frank and Afli 2026 ) ; we argue that embedding robustness is multidimensional and requires dynamic evaluation. Figure 1: (M)MTEB datasets are transformed over n runs using eight transformations that generate lexical-/stylistic-, length- or language-related variations. Recent work has begun to address these limitations. SAGE ( Goel et al. 2025 ) tests robustness to controlled perturbations and noise injections on English datasets. PTEB ( Frank and Afli 2026 ) stochastically paraphrases 20 MTEB/MMTEB datasets including 25 languages at evaluation time, revealing degradation even in state-of-the-art embedding models. FLUKE ( Otmakhova et al. 2026 ) evaluates the downstream robustness through minimal linguistic modifications for the English language to find task-specific brittleness. Building on this line of work, we argue that PTEB’s dynamic, LLM-based evaluation-time paradigm is a promising direction, but that robustness evaluation must be broadened beyond paraphrasing and structured along interpretable axes: aggregate robustness scores can still obscure important model differences since robustness is multidimensional. We therefore introduce the Harder Text Embedding Benchmark (HTEB), which generates transformations along three deployment-oriented axes: Lexical/Stylistic, Length, and Language ( Figure 1 ). In summary, this work makes two main contributions: (1) We introduce HTEB, a dynamic benchmark that applies eight LLM-generated transformations along three axes, validated by 4,800 human annotations based on the English datasets. (2) Evaluating 16 embedding models on 32 datasets covering 42 languages, we show scalar rankings hide model-specific, partly decoupled robustness profiles, and scaling changes axis-specific robustness profiles but does not generally improve robustness in three model families. These findings challenge single-score embedding rankings and motivate per-axis robustness profiles for benchmarks. 2 Method 2.1 Models Figure 2: Four-step method to select the HTEB transformation model. Generative Models. The selection of generative models follows the process in Figure 2 . We select the LLM used to generate eight transformations by filtering open-weight vLLM-compatible candidates suitable for consumer-grade hardware (see hardware specification in Appendix B ), measuring transformation error rates on STS-B ( Cer et al. 2017 ) , and comparing the five lowest-error models with a crossover LLM-judge setup. The transformations are briefly described in Table 1 . Prompts are shown in Appendix D ; an example for each transformation in Table 12 of Appendix H . Embedding Models. We evaluate open-weight SentenceTransformers-compatible ( Reimers and Gurevych 2019 ; Reimers and Gurevych 2020 ) embedding models spanning model size, MTEB performance, and language coverage. All original and transformed scores are computed with our own evaluation pipeline, so scores may differ from public MTEB scores. Although we do not advocate limiting the evaluation to a single score, we still report total HTEB scores as an average over the axis scores. 2.2 Tasks, Datasets and Metrics We evaluate 11 models on 19 English datasets across eight tasks and 9 models on 13 multilingual datasets across seven tasks 1 1 1 Instead of a third Semantic Textual Similarity (STS) dataset, we include a Semantic Textual Relatedness (STR) dataset to capture broader relatedness ( Abdalla et al. 2023 ) . , with four models evaluated in both settings. The multilingual subset covers 42 languages, including low-resource African languages alongside mid- and high-resource languages. See Table 9 and Table 10 in Appendix E for details regarding the datasets and Appendix G for the metrics applied. 2.3 Statistical Analysis We apply non-parametric inferential statistics throughout this work, as normality cannot be assumed for NLP evaluations ( Søgaard 2013 ; Dror et al. 2018 ) . For LLM judge score comparison during transformation model selection, we use bootstrapping ( Efron and Tibshirani 1994 ) , with observations defined as sentence-pair scores averaged across judges and transformations. For the embedding model evaluation, we conduct three runs per model using different random seeds to mitigate single-run unreliability ( Reimers and Gurevych 2017 ; Reimers and Gurevych 2018 ; Madaan et al. 2024 ) . Each run is a seeded execution of the HTEB transformation pipeline including selection of target or intermediate languages where applicable. For all reported results and inferential tests, we average scores over runs because runs are stochastically dependent repeated evaluations of the same dataset-model-transformation combination and therefore should not be treated as independent observations. Using per-dataset averages as observations, we test per-transformation dataset-level paired score deltas using the Wilcoxon signed-rank tests ( Wilcoxon 1945 ) and Hodges–Lehmann shifts with 95% CIs. We use the Wilcoxon signed-rank test due to bootstrap instability at smaller sample sizes ( Søgaard et al. 2014 ; Hesterberg 2015 ) . The p-values for the selection of the transformation model and the evaluation of the embedding models are Holm-corrected ( Holm 1979 ) to control for multiple comparisons, an important problem when evaluating models on multiple datasets ( Dror et al. 2017 ) . Details for all statistical analyses are provided in Table 13 in Appendix I . Transformation Description Lexical/Stylistic Paraphrasing Rephrase preserving meaning Backtranslation Translate to random language and back Style Change Convert formal to informal language and vice versa Length Expansion Elaborate preserving core meaning Summarisation Compress preserving core meaning Summ. Expansion Expand then summarise Language Translation Translate each text to the same random target language Cross-Translation Translate each text to independent random languages Table 1: HTEB’s eight transformations grouped by robustness dimension. 2.4 Human Evaluation We only include transformations in HTEB that receive sufficiently high ratings from human annotators. Specifically, we set the acceptance threshold at a mean score of 3 on a 1-to-5 Likert scale, corresponding to "acceptable". This helps reduce the risk that observed performance drops are primarily driven by semantic distortion introduced by the transformations. All ratings are collected along two criteria: 1. Transformation quality: The degree to which the output