Maintaining MTEB: Towards Long Term Usability and Reproducibility of Embedding Benchmarks
Isaac Chung, Imene Kerboua, Márton Kardos, Roman Solomatin, Kenneth Enevoldsen · 2025
Read Maintaining MTEB: Towards Long Term Usability and Reproducibility of Embedding Benchmarks by Isaac Chung, Imene Kerboua, Márton Kardos, Roman Solomatin, Kenneth Enevoldsen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-21182.txt.
Introduction
Isaac Chung and co-authors describe in 2025 how a major text embedding benchmark is maintained for long-term use. The benchmark, MTEB, is a standard test for embedding models. Keeping it reproducible and extensible takes engineering work beyond the original methodology papers that defined its core tasks.
They detail continuous integration pipelines that check data integrity and run tests automatically. Design choices aim at reliable, comparable results over time. Community contributions and new tasks are handled through defined processes. Their experience offers lessons for other machine learning benchmark maintainers facing growth and quality demands.
Catalog record
| CREATOR | Isaac Chung, Imene Kerboua, Márton Kardos, Roman Solomatin, Kenneth Enevoldsen |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 25 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-21182
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Maintaining MTEB: Towards Long Term Usability and Reproducibility of Embedding Benchmarks Isaac Chung, Imene Kerboua, Márton Kardos, Roman Solomatin, Kenneth Enevoldsen Abstract The Massive Text Embedding Benchmark (MTEB) has become a standard evaluation platform for text embedding models. While previous work has established the core benchmark methodology, this paper focuses on the engineering aspects that ensure MTEB’s continued reproducibility and extensibility. We present our approach to maintaining robust continuous integration pipelines that validate dataset integrity, automate test execution, and assess benchmark results’ generalizability. We detail the design choices that collectively enhance reproducibility and usability. Furthermore, we discuss our strategies for handling community contributions and extending the benchmark with new tasks and datasets. These engineering practices have been instrumental in scaling MTEB to become more comprehensive while maintaining quality and, ultimately, relevance to the field. Our experiences offer valuable insights for benchmark maintainers facing similar challenges in ensuring reproducibility and usability in machine learning evaluation frameworks. The MTEB repository is available at: https://github.com/embeddings-benchmark/mteb † † affiliationnotice: Equal contribution 1 Introduction Benchmarks are critical in guiding machine learning progress, particularly in the rapidly evolving field of embeddings and representation learning. MTEB ( Muennighoff et al. 2022 ) has emerged as a comprehensive evaluation framework for embedding models across diverse tasks and languages. While initially designed as a standalone benchmark, MTEB has evolved into a versatile evaluation ecosystem. This natural evolution has been driven by sustained interest and engagement from the research community, integrating numerous extensions and specialized variants: language-specific benchmarks like C-MTEB ( Xiao et al. 2024 ) , MTEB-French ( Ciancone et al. 2024 ) , and German Text Embedding Clustering Benchmark ( Wehrli et al. 2024 ) ; regional benchmarks such as SEB ( Enevoldsen et al. 2024 ) ; multilingual expansions through MMTEB ( Enevoldsen et al. 2025 ) covering over 250 languages; domain-specific adaptations like ChemTEB ( Kasmaee et al. 2024 ) ; and cross-modal evaluations like MIEB ( Xiao et al. 2025 ) . However, as the benchmark grows in scope and adoption, maintaining its sustainability and effectiveness presents significant engineering challenges that have received limited attention in the literature. These challenges span several dimensions: reproducibility — ensuring results can be consistently replicated given a model; usability — evaluating and presenting comparisons effectively and efficiently; extensibility — accommodating new tasks, languages, and modalities without disrupting existing functionality; integrity — ensuring results reflect genuine generalization, free from training data contamination, and relevance — addressing current research challenges as the field evolves. The literature has emphasized benchmark relevance, but often neglects the practical infrastructure required to maintain a benchmark’s utility over time. This paper explores the technical design choices underpinning MTEB’s sustainability and extensibility. We focus on the software engineering practices implemented to 1) ensure dataset quality , 2) assess benchmark zero-shot levels , and 3) facilitate community engagement and expansion . Through concrete case studies, we demonstrate how our approach addresses real-world challenges in benchmark maintenance, from multilingual expansion to contamination detection and community contribution scaling. By sharing our experiences in maintaining MTEB, we aim to highlight often overlooked aspects of benchmark development that are essential for long-term usability. The engineering approaches described here address common challenges in benchmark maintenance and can inform similar efforts across the machine learning community, ultimately contributing to more reproducible and trustworthy evaluation standards. 2 The MTEB Framework Maintaining a large-scale benchmark repository like MTEB requires robust infrastructure that balances reproducibility and flexibility to accommodate diverse and novel models and evaluation scenarios. Rather than focusing solely on implementation details, we highlight the design principles that enable MTEB’s sustainable growth and innovation potential. 2.1 System Architecture Figure 1: MTEB is designed with a modular architecture that separates concerns into distinct components. Users can select which model, task, or benchmark to run via the Python API or command line interface. The produced results can then be submitted to the results repository, which is read by the leaderboard. MTEB consists of a set of code repositories and a leaderboard. Figure 1 shows at a high level the relationship between models, tasks, and result aggregation: Standardized Model interfaces for embedding generation, allowing evaluation of diverse architectures from sentence transformers to instruction-tuned LLMs with minimal adaptation; Task definitions encapsulate evaluation logic for different paradigms (classification, clustering, retrieval, etc.) while remaining agnostic to specific datasets or models; Dataset handlers manage data loading, preprocessing, and validation while maintaining consistent interfaces across languages and domains; Result processors standardize output formats and compute metrics uniformly across tasks. To automate validations and releases, we adopt standard CI/CD practice in the MTEB repository, which is detailed in Appendix A . 2.2 Leaderboard In addition to the evaluation package, MTEB features an open leaderboard that displays model performance across the benchmark, enabling businesses or users to select suitable models for their needs and supporting researchers in driving progress in the field. For a visualization of the leaderboard, see Figure 3 in Appendix B . Submissions of results to the leaderboard are often made by the broader community. These can be made by submitting a pull request to the results repository. 3 Ensuring Benchmark Reproducibility While MTEB’s modular architecture provides the foundation for its functionality, specific reproducibility measures are essential to ensure the benchmark’s validity over time. The primary challenges we identified revolve around ambiguity in the evaluation workflow: which version of a dataset or model is being used, which version of the code is executing the evaluation, and whether saved results can be reliably loaded and trusted. 3.1 Multi-level Versioning System A key challenge in benchmark maintenance is managing changes to various components while preserving result comparability. Our versioning system addresses this through: Task versioning : Tasks undergo updates when evaluation protocols are changed, e.g. using stratified sampling. Each version maintains its own evaluation protocol. Dataset versioning : Each dataset references a specific revision from its source (typically Hugging Face), ensuring identical data across evaluation runs. Versions are updated when issues like mislabeled examples, or duplicates are discovered. Model versioning : Models reference specific checkpoints or API versions to maintain consistency, as model behavior can change significantly between releases. Code versioning : The MTEB package itself follows semantic versioning, with clear compatibility boundaries. 3.2 Reproducible Evaluation Environments Certain evaluation methods introduce potential variability that must be controlled. For example, clustering tasks using K-means and classification tasks using linear probing require hyperparameter search procedures that can produce different results based on initialization. To address this, we provide consistent seeds for deterministic execution of evaluation tasks. Additionally, recognizing the substantial computational resources required for benchmark evaluation, we added CO_{2} emissions logging with the codecarbon library 1 1 1 https://codecarbon.io/ . This helps users quantify and report the environmental impact of their evaluation runs, promoting transparency in the computational costs of benchmarking. 3.3 Community Verifiable Results The MTEB leaderboard is designed to maximize transparency and verifiability. Submissions require: Reference implementation : Each model must have a well-documented reference implementation. For open-source models, this includes direct links to code repositories and model weights. For proprietary models, implementations may rely on API calls, though this introduces a trust assumption that the API consistently serves the same model version. Peer review process : Submitted results undergo community review via pull requests