Small data but Specific: Task-Aligned Embeddings for Clinical Code Retrieval

David Rey-Blanco, Roberto Cruz · 2026

Read Small data but Specific: Task-Aligned Embeddings for Clinical Code Retrieval by David Rey-Blanco, Roberto Cruz (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-30529.txt.

Introduction

Rey-Blanco and Cruz test whether task-focused embeddings beat general ones for finding clinical codes beyond English. Sentence embeddings are usually trained and judged on English, so recall collapses when records are matched to ICD-10-CM and CIE-10 codes in other tongues. They fine-tune a Spanish biomedical encoder on synthetic data generated by a large model across six languages, plus a reranking stage. The tuned retriever matches or beats a strong baseline on most measures, with Portuguese gains largest, at a small cost to English.

Catalog record

CREATORDavid Rey-Blanco, Roberto Cruz
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE59678
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.30529

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Small data but Specific: Task-Aligned Embeddings for Clinical Code Retrieval David Rey-Blanco, Roberto Cruz Abstract Clinical coding can be framed as a fine-grained semantic retrieval task in which short clinical mentions must be matched to the correct ICD-10/CIE-10 code despite subtle distinctions in severity, anatomy, temporality, or etiology. Existing sentence-embedding models are poorly adapted to this setting, particularly for non-English clinical text, while manually annotated training data remains scarce and expensive. This work studies whether compact, task-specific retrievers can be improved through high-quality synthetic supervision generated with large language models. We use a frontier Large Language Model to construct multilingual training corpora grounded in the ICD-10 hierarchy. A Spanish biomedical bi-encoder is fine-tuned on 19,502 synthetic examples across six languages, while a cross-encoder reranker is trained on 10,628 listwise groups with clinically plausible hard negatives. The resulting system is evaluated on CodiESP v4 and DISTEMIST against BM25 and several widely used sentence-transformer baselines. The proposed retriever achieves F1 = 0.709 and MAP@10 = 0.747 on CodiESP, and F1 = 0.776 and MAP@10 = 0.812 on DISTEMIST, outperforming all public baselines by a large margin. Results indicate that retrieval quality depends more on task-aligned supervision and clinically meaningful negatives than on model scale alone. Our findings suggest that carefully curated synthetic datasets can enable compact and effective clinical semantic search systems without requiring web-scale training corpora. Keywords: Semantic Search, Medical AI, Embeddings, Clinical Coding, LLM Introduction Clinical coding is a high-stakes text-to-code retrieval problem. Given a short clinical mention, complaint, or fragment of a medical note, the system must retrieve the correct entry from a controlled terminology such as ICD-10-CM or its Spanish adaptation, CIE-10. This is harder than ordinary semantic search because adjacent codes often describe clinically similar conditions that differ by a qualifier such as organism, laterality, severity, episode type, or anatomical location. A useful model must therefore retrieve the right clinical variant, not only the right disease family. Modern dense retrieval usually addresses this kind of problem with two stages. A bi-encoder first projects queries and candidate descriptions into a shared vector space and retrieves a shortlist efficiently [ 77 , 41 ] . A cross-encoder then jointly reads the query and each candidate in the shortlist to produce a more precise relevance score [ 62 , 31 ] . This recall-then-rerank architecture is common in retrieval-augmented generation [ 49 , 99 , 68 ] and in automatic ICD coding [ 60 , 37 , 103 ] . The difficulty is that most public embedding models and biomedical NLP resources are still centred on English. BioBERT [ 46 ] , ClinicalBERT [ 2 ] , and related models are strong on English EHRs and biomedical benchmarks such as PubMedQA [ 39 ] , but Spanish, Catalan, Italian, Portuguese, and other clinical languages are under-represented [ 40 ] . Multilingual encoders help [ 13 , 75 , 78 ] , but clinical retrieval remains fragile when abbreviations, morphology, surface forms, and coding conventions vary across languages [ 61 ] . The limitations of clinical coding models The central issue is not only the lack of model capacity. General-purpose encoders are usually trained to capture broad semantic relatedness, whereas clinical coding requires fine-grained equivalence. For example, “Brucelosis debida a Brucella suis” and “Brucelosis debida a Brucella melitensis” are semantically close but not interchangeable as codes. The same tension appears across thousands of CIE-10 descriptions: the nearest clinical neighbour is often the wrong answer. Four practical limitations follow from this setting. First, tokenisation and pretraining corpora under-cover non-English clinical morphology, which can flatten important lexical distinctions. Second, biomedical pretraining does not necessarily transfer if it was learned in another language or on a different clinical distribution [ 46 , 2 , 10 ] . Third, cross-lingual transfer degrades in dense retrieval because small representation shifts can change the top-1 result [ 45 , 98 ] . Fourth, two-stage systems can be misaligned: a reranker cannot recover the correct code if the bi-encoder never retrieves it, and a high-recall bi-encoder is insufficient if it cannot order clinically adjacent candidates. The data problem is equally important. Labelled clinical data is expensive because annotation requires both medical and linguistic expertise. In multilingual coding, the data must also match the local coding vocabulary and the intended level of granularity. A corpus labelled at chapter level does not teach full-code distinctions, and an English ICD corpus does not necessarily contain the Spanish, Catalan, or Portuguese expressions that a local coding system must handle. Proposal, results, and contributions We study whether LLM-generated synthetic supervision can make domain- and language-specific clinical retrievers practical under these data constraints. Rather than using synthetic data only as generic augmentation, we use an LLM as a tutor for a controlled label space: given a code, definition, neighbouring codes, and target language, the generator creates clinical mentions, paraphrases, and hard negatives that force the model to learn the decision boundary. This follows recent work on LLM-generated instruction and retrieval data [ 96 , 7 , 16 ] and on clinical synthetic text generated from domain knowledge or labels [ 73 , 100 , 50 , 44 ] . Concretely, we train a two-stage retriever for Spanish CIE-10/ICD-10 search. The bi-encoder is fine-tuned from PlanTL-GOB-ES/bsc-bio-ehr-es [ 10 ] on about 19,500 multilingual synthetic pairs covering English, Spanish, Catalan, Italian, Portuguese, and French. The cross-encoder reranker is trained on roughly 10,600 listwise groups with clinically plausible hard negatives. We evaluate the resulting system on two held-out Spanish clinical coding datasets, CodiESP v4 and DISTEMIST, at full-code, category, and chapter resolution. The main result is that targeted synthetic supervision and reranking produce a large improvement over public baselines. On CodiESP, the complete system reaches 70.9% top-1 exact accuracy, improving by 48.4 percentage points over the best public encoder tested, MiniLM-L6-v2, and by 51.6 points over the in-domain English S-BioBERT baseline. On DISTEMIST, it reaches 77.6% top-1 exact accuracy. The results also show that larger embeddings and biomedical pretraining alone are not enough: MPNet-v2 underperforms MiniLM-L6-v2, while S-BioBERT underperforms both despite domain-specific pretraining. The paper makes three contributions: 1. It documents a reproducible recipe for training a two-stage ICD retriever from multilingual LLM-generated supervision. 2. It shows that language coverage and task-aligned synthetic curation can matter more than generic biomedical pretraining for Spanish clinical code retrieval. 3. It quantifies the role of cross-encoder reranking in resolving qualifier-level distinctions that are common in ICD-10-CM and CIE-10-ES. Synthetic data is not treated here as a substitute for real clinical evaluation, thus generated examples can be too regular, biased toward the generator’s preferred phrasing, or detached from the noise of real notes [ 54 , 86 ] . For that reason, we use synthetic data only for supervision and evaluate exclusively on held-out human clinical coding datasets (using human-anotated sets only for this task). This paper is structured as follows. The methodology section describes the synthetic data generation process, the bi-encoder, and the cross-encoder. The experimental design section defines the CIE-10 hierarchy, the search backend, the baseline models, the evaluation datasets, and the metrics. The results section reports performance on CodiESP and DISTEMIST. The discussion compares the findings with public benchmarks and related ICD coding literature, and the final sections

Read the complete text