Towards Domain Specification of Embedding Models in Medicine
Mohammad Khodadad 1,∗,†, Ali Shiraee 1,∗, Mahdi Astaraki 1, Hamidreza Mahyar 1 · 2025
Read Towards Domain Specification of Embedding Models in Medicine by Mohammad Khodadad 1,∗,†, Ali Shiraee 1,∗, Mahdi Astaraki 1, Hamidreza Mahyar 1 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-19407.txt.
Introduction
Mohammad Khodadad and co-authors present a medical text embedding model and a broad test suite for it in a 2025 paper. Embedding models turn medical text into number vectors that power search, question answering, and decision support. Most models train on a narrow slice of medical writing and are tested on benchmarks that miss many real tasks.
Their model was fine-tuned on varied medical sources with a self-teaching contrast method. The companion benchmark holds fifty-one tasks across classification, grouping, paired text judgments, and retrieval, modeled on a general benchmark but tuned to medical text. The combined approach beat leading alternatives across task types. The authors offer both the model and the suite as a firmer base for medical language tools.
Catalog record
| CREATOR | Mohammad Khodadad 1,∗,†, Ali Shiraee 1,∗, Mahdi Astaraki 1, Hamidreza Mahyar 1 |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 37 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-19407
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Towards Domain Specification of Embedding Models in Medicine Mohammad Khodadad 1,∗,†, Ali Shiraee 1,∗, Mahdi Astaraki 1, Hamidreza Mahyar 1 Abstract Medical text embedding models are foundational to a wide array of healthcare applications, ranging from clinical decision support and biomedical information retrieval to medical question answering, yet they remain hampered by two critical shortcomings. First, most models are trained on a narrow slice of medical and biological data, beside not being up to date in terms of methodology, making them ill suited to capture the diversity of terminology and semantics encountered in practice. Second, existing evaluations are often inadequate: even widely used benchmarks fail to generalize across the full spectrum of real world medical tasks. To address these gaps, we leverage MEDTE, a GTE model extensively fine-tuned on diverse medical corpora through self-supervised contrastive learning across multiple data sources, to deliver robust medical text embeddings. Alongside this model, we propose a comprehensive benchmark suite of 51 tasks spanning classification, clustering, pair classification, and retrieval modeled on the Massive Text Embedding Benchmark (MTEB) but tailored to the nuances of medical text. Our results demonstrate that this combined approach not only establishes a robust evaluation framework but also yields embeddings that consistently outperform state of the art alternatives in different tasks. {IEEEkeywords} Embedding, Benchmark, Medicine, Biology, Evaluation † † footnotetext: 1 Engineering Technology Building, McMaster University † † footnotetext: ∗ These authors contributed equally to this work † † footnotetext: † Corresponding author: Mohammad Khodadad, [email protected] \IEEEpeerreviewmaketitle 1 Introduction Medical text embedding models serve as fundamental building blocks for diverse clinical NLP applications, from clinical decision support and biomedical literature retrieval to medical question-answering. These models transform unstructured biomedical texts (spanning electronic health records, research literature, and clinical trial data) into meaningful vector representations, enabling machine learning algorithms to extract useful insights from the rapidly growing volumes of medical data [ 1 , 2 ] . While recent transformer-based models such as BERT [ 3 ] have revolutionized text embeddings in general NLP, their performance on domain-specific medical tasks remains suboptimal. Despite the emergence of several medical-domain embedding models such as BioBERT [ 2 ] , ClinicalBERT [ 4 ] , and Med-BERT [ 5 ] , many of these models fail to outperform even the most recent general-purpose text embedding models. In some cases, state-of-the-art general models such as E5 [ 6 ] and SBERT [ 7 ] achieve superior results on medical text benchmarks, raising concerns about the effectiveness of domain-adapted models and highlighting the need for a more comprehensive evaluation framework. These concerns not being very obvious, is a result of not having a standard high quality, comprehensive benchmark in medicine. Most papers utilize their only limited benchmark, which only covers very few aspects of embedding models. Existing benchmarks such as the Massive Text Embedding Benchmark (MTEB) [ 8 ] provide large-scale evaluations for general-domain embedding models and a limited set of medical tasks; however, they still lack a comprehensive, structured assessment tailored to medical text. Current evaluations of medical embeddings are typically narrow in scope, partly inaccessible, and fail to cover the full range of task types, leaving models tested on only a handful of datasets. As a result, it remains difficult to judge their real-world effectiveness across diverse clinical and biomedical applications. A systematic, large-scale benchmark dedicated to medical text embeddings is therefore still missing. To bridge this gap, we first develop a medical embedding model by fine-tuning a GTE embedding backbone with self-supervised contrastive learning. Training leverages an extensive corpus that combines PubMed abstracts, curated medical articles from Wikipedia, entries from ClinicalTrials.gov, clinical notes from MIMIC-IV, preprints from bioRxiv and medRxiv, and large-scale medical QA datasets such as MedMCQA and MedQA, ensuring broad domain coverage. We then introduce a comprehensive benchmark for medical text embeddings comprising 51 tasks across classification, clustering, pair classification, and retrieval. The benchmark aggregates data from multiple high-quality sources, including PubMed abstracts, Wikipedia medical pages, structured and unstructured clinical data from MIMIC-IV and ClinicalTrials.gov, and so on. Our evaluation shows that the proposed model achieves state-of-the-art performance, outperforming previous medical-domain baselines on all benchmark tasks and even exceeding leading general-purpose embedding models. These findings underscore the importance of domain-adaptive training strategies and comprehensive multi-task benchmarking in medical NLP. 1.1 Contributions • MedTE model. By applying self-supervised contrastive learning to GTE-Base on a diverse medical corpus, we developed MedTE, which achieves state-of-the-art results on our benchmark. • MedTEB benchmark. We release the first large-scale benchmark dedicated to medical text embeddings, comprising 51 tasks that span classification, clustering, pair classification, and retrieval, sourced from a diverse set of curated medical corpora. • Comprehensive evaluation. We systematically compare domain-specific and general-purpose embedding models and show that many existing medical models lag behind strong general baselines, underscoring the need for more robust domain-adaptation techniques. • Extensive analysis. We conduct an in-depth analysis of both the benchmark and our model, examining task-level performance, error patterns, and the effects of corpus composition to provide actionable insights for future research. By establishing a standardized benchmark and advancing state-of-the-art medical text embeddings, this work lays a foundation for more effective NLP applications in the biomedical and clinical domains. 2 Literature review Dense vector embeddings encode clinical narratives into low-dimensional semantic representations, enabling rapid phenotyping, automated disease coding, and cohort discovery in electronic health records (EHRs) [ 9 , 10 ] ; they also surface relevant evidence in retrieval-augmented generation (RAG) systems [ 11 ] ; and boost recall as well as ranking in biomedical information retrieval [ 12 ] . By mitigating lexical variability and contextual ambiguity, embeddings support efficient similarity search, patient clustering, and reasoning across diverse medical corpora. Consequently, embedding techniques have become integral in EHR analytics, unifying structured and unstructured data to improve patient stratification and outcome prediction [ 13 ] . Pre-trained transformer embeddings are now central to NLP. BERT [ 3 ] introduced bidirectional masked language modeling and next-sentence prediction, enabling deep contextual understanding. SciBERT [ 14 ] was built on BERT by pretraining on a scientific-article corpus, boosting performance in biology and chemistry tasks. While BERT and SciBERT excel at token-level representations, Sentence-BERT (SBERT) [ 7 ] uses Siamese and triplet networks to produce semantically optimized sentence embeddings. The E5 family [ 6 ] further advances this by applying large-scale, weakly supervised contrastive learning over billions of text-pair examples. Emerging instruction tuned and multilingual models continue this trajectory, extending transformer embeddings to broader tasks and languages [ 15 ] . Recent work merges unsupervised and supervised contrastive objectives. GTE [ 16 ] uses large‐scale unsupervised contrastive pretraining followed by supervised fine‐tuning. Nomic-Embed [ 17 ] combines masked language modeling and contrastive learning over contexts up to 8,192 tokens with FlashAttention, gradient caching, and DeepSpeed. These steps shift from bidirectional pretraining toward specialized contrastive methods, motivating our medical embedding benchmark. In the medical domain, transformer embeddings have been tailored to clinical data: Med-BERT [ 5 ] adds cross-visit pretraining on EHRs from Cerner and Truven MarketScan; ClinicalBERT [ 4 ] continues pretraining on MIMIC-III notes; BioBERT [ 2 ] augments pretraining with PubMed abstracts and PMC full texts; ExBERT [ 18 ] integrates a module for out-of-vocabulary medical terms. These domain-specialized methods further inform