ClinicalEncoder26AM: A Multlilingual Diagnosable ColBERT Model; Evidences from the MultiClinNER Shared Task
François Remy · 2026
Read ClinicalEncoder26AM: A Multlilingual Diagnosable ColBERT Model; Evidences from the MultiClinNER Shared Task by François Remy (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-28521.txt.
Introduction
François Remy presents ClinicalEncoder26AM, a multilingual ColBERT model for clinical and biomedical texts. It aligns token-level meaning with a clinical latent space at several levels, building on BGE-M3 with synthetic notes, patient-doctor dialogues, and annotated resources. As a BIO tagger for symptoms, disorders, and procedures in the MultiClinNER shared task, it uses a small convolutional head to sharpen span boundaries. The simple system handles most documents in one 8192-token window and reaches top-five overall with state-of-the-art multilingual entity recall.
Catalog record
| CREATOR | François Remy |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 19276 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.28521
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
ClinicalEncoder26AM: A Multlilingual Diagnosable ColBERT Model; Evidences from the MultiClinNER Shared Task François Remy Abstract ClinicalEncoder26AM is a multilingual Diagnosable ColBERT ( 9 ) for clinical and biomedical texts, which aligns at multiple levels its token-level semantic with ClinicalMap25, a clinical latent space inspired by BioLORD-2023 and enriched with synthetic and annotated supervision. The post-training recipe builds upon BGE-M3 ( 2 ) , and combines synthetic clinical notes, patient–doctor conversations, and annotated resources such as MedMentions ( 7 ) , while considering both named-entity-level and sentence-level representations in a multi-adapter distillation, along with a ColBERT-style retrieval objective. In this system demonstration paper, we evaluate the model in the MultiClinNER shared task by finetuning it as a BIO tagger for patient symptoms, disorders, and procedure spans, using a lightweight two-layer CNN head to improve local boundary detection. The resulting system remains simple, processes most documents in a single 8192-token window, and achieves state-of-the-art multilingual entity recall, while achieving Top 5 overall across all entity types and languages in Character-weighted F1 scores. Training curves further show that ClinicalEncoder26AM is markedly more data-efficient than the base M3 model, supporting the usefulness of its clinical post-training for downstream information extraction. 1 Introduction ClinicalEncoder26AM is the multilingual extension and improvement of the ClinicalEncoder series of interpretable clinical encoders designed to support both high-quality retrieval and token-level semantic inspection. The model follows the Diagnosable ColBERT paradigm, in which each token is not only encoded for late interaction, but also aligned with a clinically grounded latent space that serves as a semantic reference during analysis and debugging ( 9 ) . This design is motivated by the difficulty of evaluating the alignment between ColBERT retrieval models and expert judgment over concept similarities. While ColBERT models are intrepretable at the document–query interaction level, their embeddings themselves remain opaque. By contrast, a diagnosable model can expose whether a token or span is being mapped toward the appropriate region of a domain-specific semantic space, making failures easier to address. ClinicalEncoder26AM builds on ColBERT-style late interaction, but adapts it to multilingual clinical and biomedical text through post-training on top of BGE-M3 ( 2 ) . Its semantic grounding relies on ClinicalMap25, a latent clinical space inspired by BioLORD-2023 ( 8 ) and enriched with additional synthetic and annotated signals. The supervision used for this post-training includes synthetic clinical notes, patient–doctor conversations, and annotated biomedical corpora such as MedMentions ( 7 ) . The overall goal is not only to produce strong multilingually-aligned representations, but also to retain a meaningful alignment between local token semantics, higher-level sentence information, and retrieval-oriented document representations. In this paper, we focus on the MultiClinNER shared task 4 of SMM4H-HeaRD 6 rather than on the post-training details. Our main objective is to describe how a clinically post-trained multilingual encoder can be adapted into a practical BIO tagging system for extracting symptoms, disorders, and procedures from clinical texts. This downstream setting is particularly informative because it highlights an important distinction: even when token representations are rich enough to support ontology-grounded interpretation, they are not automatically optimized for precise span boundary detection. In other words, knowing what a token means in context is not the same as deciding exactly where entities begin and end in running text. For that reason, we finetune ClinicalEncoder26AM with a lightweight task-specific tagging head instead of attempting to recover spans directly from ontology mappings. This choice keeps the system simple, benefits from the shared task annotations, and provides a clean way to evaluate whether the post-trained multilingual encoder transfers effectively to structured clinical information extraction. As we show later, this simple approach yields a strong recall-oriented system across all languages, with competitive overall rankings and no major degradation in languages such as Czech that were not exercised in the original post-training. 2 Methodology ClinicalEncoder26AM is built from the multilingual BGE-M3 encoder ( 2 ) , whose backbone follows an XLM-RoBERTa-style architecture with 24 Transformer layers, a hidden size of 1024, 16 attention heads, and supports up to 8192 simultaneous tokens (about 32kB of raw English text). In practice, this context size is sufficient to encode almost all documents in the MultiClinNER corpus a single forward pass. Unlike generative language models, ClinicalEncoder26AM computes all document representations in one pass, making it suitable for both large-scale retrieval and clinical information extraction. The central methodological idea is to couple late-interaction retrieval representations derived by down-projection from embeddings aligned with a clinically grounded reference space. More precisely, each token is encouraged to align with ClinicalMap25, a latent semantic space designed to capture fine-grained clinical meaning at the token, mention, and local contextual levels. ClinicalMap25 is inspired by BioLORD-2023 ( 8 ) , but extends that line of work with additional supervision signals and more recent synthetic data sources in order to represent clinical concepts more precisely in actual clinical language. This alignment is intended to make the model more diagnosable: instead of only inspecting token-to-token interaction scores after retrieval, one can also inspect whether the token embeddings themselves are organized in a clinically sensible way. A mixture of synthetic clinical notes, patient–doctor conversations, and annotated biomedical documents such as MedMentions ( 7 ) are used as part of this post-training, including curated translations. These resources expose the encoder to both biomedical terminology and more natural clinical phrasing, including context-sensitive formulations that are common in real-world documentation. During this stage, we consider supervision at multiple granularities, including named-entity-level representations and sentence-level representations, before applying a ColBERT-style late-interaction training objective. This multi-granular training setup is meant to preserve local semantics while maintaining compatibility with document-level retrieval behavior. A second important component is the use of multi-adapter distillation to combine complementary supervision signals into a single multilingual model. Rather than relying on one homogeneous source of supervision, the post-training transfers knowledge from multiple task-specific or data-specific views into a unified encoder. This strategy is particularly useful in the clinical domain, where no single dataset fully captures ontology grounding, contextual phrasing, retrieval behavior, and multilingual variation at once. It also helps retain the multilingual alignment inherited from BGE-M3 while adapting the model for clinical semantics. As a result, ClinicalEncoder26AM can be used in several closely related ways. The same encoder can support multilingual retrieval, direct token-to-concept mapping against ontologies such as SNOMED CT ( 3 ) or UMLS ( 1 ) , and downstream token classification after task-specific finetuning. From a modeling perspective, this is possible because the post-training is not restricted to one output format: it jointly encourages clinically meaningful local embeddings, sentence-aware contextualization, and compatibility with ColBERT-style late interaction. In the present paper, we focus on the final use case, namely transfer to BIO tagging for shared-task named entity recognition. 3 Experimental Setup For the MultiClinNER shared task, we finetune ClinicalEncoder26AM as a BIO sequence tagger for symptom, disorder, and procedure extraction in seven European languages. All target languages are supported by the multilingual base model, although only a subset was explicitly represented in the original post-training mixture. Following the shared-task setup, we train one model per entity type rather than a single joint tagger, as result of the format of the annotation data. The tagging architecture is intentionally lightweight. On top of the encoder outputs, we add a two-layer