An end-to-end system for explainable clinical coding across languages and diverse medical data sources Authors: Alicia Ramirez-Arrabe, Juan Martinez-Romo, Andres Duque Published: 2026 ABSTRACT AbstractBackground and objectiveThe International Classification of Diseases (ICD) coding system standardizes clinical texts to improve the information extraction process in healthcare and research. However, most existing automated coding systems operate as “black boxes”, highlighting the need for explainable approaches that offer transparency and interpretable solutions. In this work, we present an end-to-end system that provides explainable clinical coding predictions.MethodsWe develop a three-phase system for explainable clinical coding. In Phase 1, text spans potentially describing ICD codes are detected using different Named Entity Recognition (NER) models. Phase 2 applies a supervised text classification model with a confidence threshold, while low-confidence cases are classified in Phase 3 using a semantic similarity model built from ICD code descriptions and related keyphrases.ResultsThe system is evaluated on four corpora in Spanish and English, annotated with ICD codes from three variants (ICD-10-CM, ICD-10-PCS, and ICD-O-3) and their corresponding textual mentions. Overall, the system proves to be robust and competitive with state-of-the-art approaches, outperforming most of them and achieving an average F1-score improvement of 3.42%.ConclusionsThis study presents one of the most comprehensive evaluations of an explainable clinical coding system across languages and ICD variants. The proposed approach demonstrates strong robustness and generalization, effectively handling unseen codes as well as discontinuous and overlapping entities.Keywords: Automated Clinical Coding, Explainable Artificial Intelligence, ICD-10 codes, Interpretability, Clinical Natural Language Processing Introduction Healthcare systems generate vast amounts of clinical texts every day worldwide. This information is an important resource but often contains unstructured data in the form of free text, and its transformation into structured data remains a challenge [1]. The application of Natural Language Processing (NLP) techniques to clinical documents, such as discharge summaries, radiology reports or Electronic Health Records (EHRs), has proven to be highly valuable across various areas, like symptom identification [2] or extraction of medical concepts [3], among others. In this work, we focus on a strongly important NLP task of the biomedical domain: automated clinical coding. The main objective of clinical coding is to label the text written by doctors in clinical reports with standardized codes representing the patient’s medical information, which may be very useful to improve the efficiency of information management for healthcare professionals, researchers and managers of clinical institutions [4]. Nonetheless, manual coding of clinical texts is a complex, time-consuming, and error-prone task [5], making the implementation of automated solutions essential. Specifically, automated clinical coding systems leverage NLP techniques to identify medical terminology in clinical reports and assign the corresponding standardized codes. Several coding systems exist for this purpose, such as the Systematized Nomenclature of Medicine-Clinical Terms (SNOMED CT) [6], Current Procedural Terminology (CPT) [7], and the International Classification of Diseases (ICD) [8], among others. This work focuses on the latter, specifically on its tenth revision (ICD-10), or CIE-10 in Spanish, which is the most widely used in hospitals to describe diagnoses (ICD-10-CM) and medical procedures (ICD-10-PCS). The first versions of ICD-10 were published in the 1990s; however, its implementation in most countries, including Spain, did not begin until the 2010s. The ICD-10-CM code set allows for up to 155,000 diagnosis codes but currently includes around 69,000. Similarly, ICD-10-PCS contains approximately 72,000 procedure codes. In addition to ICD-10 codes, this research also addresses the clinical coding of a domain-specific extension of the ICD system for oncological terminology: the International Classification of Diseases for Oncology (ICD-O), or CIE-O in Spanish, which describes the morphological characteristics of malignant neoplasms, that is, cancers. The third revision of ICD-O (ICD-O-3), which is the most commonly used version in hospitals nowadays and is the one considered in this work, includes approximately 14,000 possible codes. Regarding the currently existing clinical coding systems, a major drawback of most of them is that they operate as “black boxes”, that is, they generate code predictions without providing any explanation for their decisions. This lack of transparency raises concerns regarding their reliability and trustworthiness [9], creating the need to apply eXplainable Artificial Intelligence (XAI) techniques. In this context, the concepts of explainability and interpretability are frequently employed. Although some authors distinguish between them [10] and also introduce the related notion of comprehensibility [11], many researchers treat explainability and interpretability as synonymous [12]. In this work, we follow this latter view, using explainability and interpretability interchangeably. Therefore, the task of explainable clinical coding represents a significant step forward by introducing interpretability into clinical coding systems. From this perspective, such systems should not only assign ICD codes to each clinical report, but also identify the specific text spans that justify each prediction. That is, within the context of automatic clinical coding, we consider a system to be explainable in terms of correct code+entity predictions. We have selected this methodology and evaluation to ensure comparability with state-of-the-art results, following approaches proposed in previous studies [13–17]. With this goal in mind, we have designed an explainable automated clinical coding system that focuses on two main tasks: (1) assigning the correct ICD codes to clinical reports, and (2) allowing for the interpretability of the resulting ICD code predictions by identifying the text spans that justify each coding decision. Due to the nature of the problem and the corpora employed in this study, both tasks present several challenges. The main issues of the first task are the following. Each of the ICD code variants considered includes thousands of possible labels; that is, the set of possible codes is very large. This leads to challenges like data sparsity and scalability, since models must handle a huge number of labels efficiently [18]. Each of the ICD code variants considered includes thousands of possible labels; that is, the set of possible codes is very large. This leads to challenges like data sparsity and scalability, since models must handle a huge number of labels efficiently [18]. The problem is multilabel, meaning that multiple codes may be assigned to each document. Additionally, the exact number of codes assigned per document is unknown. The problem is multilabel, meaning that multiple codes may be assigned to each document. Additionally, the exact number of codes assigned per document is unknown. Many ICD codes are extremely rare. Indeed, in most corpora, over half of the codes appear only once across the entire dataset. Thus, supervised classification models may only see a single instance of most codes during training and must be able to deal with this severe label imbalance. Many ICD codes are extremely rare. Indeed, in most corpora, over half of the codes appear only once across the entire dataset. Thus, supervised classification models may only see a single instance of most codes during training and must be able to deal with this severe label imbalance. As a direct consequence of the previous challenge, the test sets contain codes that are not present in the training or development sets, implying that classification cannot rely solely on a fully supervised approach. In fact, a major drawback of most automatic clinical coding systems, either explainable or not, is that they overlook unseen codes and focus only on those that appear during training [19]. As a direct consequence of the previous challenge, the test sets contain codes that are not present in the training or development sets, implying that classification cannot rely solely on a fully supervised approach. In fact, a major drawback of most automatic clinical coding systems, either explainable or not, is that they overlook unseen codes and focus only on those that appear during training [19]. Due to these factors, the task is highly complex and can be considered an eXtreme Multilabel Text Classification (XMTC) problem. As for the second task, it is essential for developing an explainable system, but it also presents several challenges. While most textual ICD mentions are flat entities, some may be nested or overlap with others. Additionally, some entities can be discontinuous, and their components may appear in separate parts of the text, sometimes close together, and other times spread across different or unrelated sections of the clinical report. Most existing clinical coding systems do not consider non-flat entities, leaving many relevant mentions outside their scope. Taking these challenges into consideration, we designed a system organized into three phases. In the first phase, the system employs a NER-based system for identifying parts of the text that could potentially describe an ICD code. The second and third phases rely on classification models, based on supervised and unsupervised learning, respectively, that assign each textual evidence its most probable code. The main contributions of this work are the following: We present a robust end-to-end system for explainable clinical coding in medical reports, achieving state-of-the-art results and providing one of the most complete analyses to date of a system addressing this task. Our system also tackles two important aspects often overlooked in previous studies: (1) codes not seen during training, and (2) overlapping and discontinuous entities. We present a robust end-to-end system for explainable clinical coding in medical reports, achieving state-of-the-art results and providing one of the most complete analyses to date of a system addressing this task. Our system also tackles two important aspects often overlooked in previous studies: (1) codes not seen during training, and (2) overlapping and discontinuous entities. We provide a systematic analysis of the task by reviewing most of the existing corpora, as well as recent techniques proposed in the literature. In particular, we explore four publicly available corpora in Spanish and English that, as far as we know, are the only datasets annotated with both ICD codes and their corresponding textual mentions in these languages. These corpora cover three ICD variants (ICD-10-CM, ICD-10-PCS, and ICD-O-3) and include entities with diverse characteristics, such as flat, discontinuous, and overlapping annotations. We provide a systematic analysis of the task by reviewing most of the existing corpora, as well as recent techniques proposed in the literature. In particular, we explore four publicly available corpora in Spanish and English that, as far as we know, are the only datasets annotated with both ICD codes and their corresponding textual mentions in these languages. These corpora cover three ICD variants (ICD-10-CM, ICD-10-PCS, and ICD-O-3) and include entities with diverse characteristics, such as flat, discontinuous, and overlapping annotations. We ensure explainability by explicitly incorporating it into the system’s architecture, making each coding decision interpretable for clinicians and researchers, and representing a contribution to the field of XAI. To this end, we evaluate three different NER-based systems with distinct architectures and characteristics, analyzing which performs best for the types of entities present in the studied corpora. We ensure explainability by explicitly incorporating it into the system’s architecture, making each coding decision interpretable for clinicians and researchers, and representing a contribution to the field of XAI. To this end, we evaluate three different NER-based systems with distinct architectures and characteristics, analyzing which performs best for the types of entities present in the studied corpora. The rest of this article is structured as follows: Section “Background and previous work” provides a brief overview of the task background, including an analysis of the available corpora and a review of related work in the literature. Section “Materials and methods” describes in detail the corpora used, the implemented system, and the evaluation metrics. In Section” Experiments and results”, we present the obtained results, compare them with state-of-the-art systems, and include an analysis of the main errors identified. Finally, Section “Conclusions and future work” summarizes the conclusions and suggests potential directions for future research. Background and previous work Over the years, several corpora and studies focused on the task of clinical coding have been published. However, most of them address only the code assignment task, without incorporating an explainable component to justify predictions and help healthcare professionals to better interpret the results. The main corpora and systems proposed for explainable clinical coding are discussed below. Annotated corpora in explainable clinical coding For the task of explainable clinical coding, some corpora have been publicly released that explicitly include explainability annotations. The CodiEsp shared task [13] at CLEF eHealth 2020 introduced one of the first corpora focused on explainable ICD coding of medical reports in Spanish, specifically requiring the classification of diagnoses and medical procedures. It was later followed by Cantemist [14], developed within the SEPLN IberLEF 2020 shared task, which focused on the explainable coding of neoplasms in a collection of clinical reports written in Spanish. Similarly, the FALP corpus [15] also centers on oncology reports in Spanish. Regarding other languages, MDACE [16] is an English-language corpus annotated with text spans corresponding to diagnoses and procedure codes, derived from MIMIC-III clinical texts [20]. Additional contributions in other languages include the BRONCO corpus [21], a collection of German discharge summaries annotated with diagnosis codes, and ElCardioCC [17], released for the CLEF BioASQ 2025 challenge, which focuses on explainable automated ICD-10 coding of Greek cardiology discharge letters. Apart from clinical coding corpora that explicitly include textual annotations for each code, other valuable resources exist for ICD-10 text classification. Since this work focuses on the explainability component and, as will be explained later, the designed system architecture requires data with annotated textual mentions, these resources unfortunately cannot be used directly in this study. For Spanish, the CARES (Corpus of Anonymised Radiological Evidences in Spanish) corpus [22] contains radiology reports manually labeled with ICD-10 codes. In English, the MIMIC-III corpus [20], and its updated version MIMIC-IV [23], provide discharge summaries and radiology reports from over 350,000 patients, annotated with ICD-10 diagnosis and procedure codes. From 2016 to 2018, the CLEF eHealth evaluation lab conducted shared tasks on coding death certificates with ICD-10 codes in French, English, Hungarian, and Italian [24–26]. The tasks involved mapping sentences with one or more diagnoses to ICD-10 codes. Additionally, CLEF eHealth 2019 published a corpus of over 8,000 German non-technical summaries of planned animal experiments, manually annotated with ICD-10 codes [27]. Methodologies and systems for explainable clinical coding While it is true that explainable ICD-10 coding corpora are scarce, several studies have attempted to address this issue using different approaches. In [28] the authors evaluated six different biomedical Transformer models for classifying the ICD codes of neoplasms in a private corpus containing over 20,000 oncology health records. To incorporate the explainability component, they applied Local Interpretable Model-agnostic Explanations (LIME) [29] to highlight the most relevant words associated with the predicted codes. There exists another work [30] that introduces an interpretable ICD-10 coding system using keyphrase extraction to represent codes with both literal and statistically inferred keyphrases from a corpus of 12,966 Spanish EHRs. In [19] the authors developed a Longformer-based system to classify Spanish EHRs and applied three explainability methods: Local Interpretable Model-agnostic Explanations (LIME), SHapley Additive exPlanations (SHAP), and Integrated Gradients (IG), to identify parts of the text influencing each prediction. They also proposed a new metric, called Leberage, to evaluate the contribution of the explainability techniques, concluding that LIME provides the most effective explanations. Another study [31] used a hierarchical approach with two components: Medical Entity Recognition (MER) to identify clinical entities, and Medical Entity Normalization (MEN) to classify these entities according to ICD-10 codes. They utilized the experimental framework of the CodiEsp and Cantemist corpora. Since such corpora are annotated with ICD code mentions, this system is one of the few whose explainability results can be evaluated with quantifiable metrics, enabling an objective comparison. Apart from these diverse proposals for explainable systems, one of the most widespread approaches to interpretability in clinical coding has been the use of attention mechanisms, which are used to assign specific weights to the tokens of the text for each code prediction. Using Convolutional Neural Networks (CNNs) in combination with attention mechanisms [32], assigns ICD-10 codes to Spanish EHRs while highlighting the specific text segments that contribute to each code’s prediction. Attention-based approaches for interpretable predictions have also been employed in several studies using the older ICD-9 coding system. Examples include CAML [33] and DLAC [34], which use label-wise attention with code descriptions; hierarchical models such as HA-GRU [9], HiLAT [35], and HLAN [36], which incorporate word- and sentence-level attention; and a recent approach [37], that uses a Longformer-based model to encode long clinical documents and applies multi-synonym and attention to capture code associations. Materials and methods This section describes the technical and methodological aspects of the study. It presents the main characteristics of the collected corpora and details the system developed to assign ICD codes to clinical reports, ensuring that each prediction is provided with an interpretable explanation. ICD variants The corpora collected for this work include annotations for a total of three different ICD variants. ICD-10-CM codes (CIE-10-ES Diagnósticos in Spanish) are used to identify medical diagnoses. These codes follow a hierarchical structure, ranging from a minimum of three to a maximum of seven characters (see Fig. 1-A). ICD-10-CM codes (CIE-10-ES Diagnósticos in Spanish) are used to identify medical diagnoses. These codes follow a hierarchical structure, ranging from a minimum of three to a maximum of seven characters (see Fig. 1-A). ICD-10-PCS codes (CIE-10-ES Procedimientos in Spanish) are aimed at coding medical procedures. These codes follow a multiaxial structure of seven characters, where each character represents a specific attribute, such as anatomical site, body system, or type of procedure (see Fig. 1-B). ICD-10-PCS codes (CIE-10-ES Procedimientos in Spanish) are aimed at coding medical procedures. These codes follow a multiaxial structure of seven characters, where each character represents a specific attribute, such as anatomical site, body system, or type of procedure (see Fig. 1-B). ICD-O-3 codes (CIE-O-3 in Spanish) are used to classify the morphology and behaviour of neoplasms. ICD-O-3 codes also follow a multiaxial seven-character format, capturing some of the tumor’s attributes such as histology, differentiation grade, and behavior (see Fig. 1-C). ICD-O-3 codes (CIE-O-3 in Spanish) are used to classify the morphology and behaviour of neoplasms. ICD-O-3 codes also follow a multiaxial seven-character format, capturing some of the tumor’s attributes such as histology, differentiation grade, and behavior (see Fig. 1-C). Structure of the three ICD variants: A. hierarchical structure of ICD-10-CM code S52.521A. B. multiaxial structure of ICD-10-PCS code 02713D6. C. multiaxial structure of ICD-O-3 code 8120/31 Corpora In this work, four clinical corpora were collected and studied. Each corpus consists of clinical documents annotated with ICD codes and their corresponding textual mentions, using a specific variant of the ICD coding system. The corpora used in this study vary in language (Spanish or English), and may contain overlapping or discontinuous mentions. Textual mentions including non-adjacent spans were considered discontinuous entities, while mentions that shared at least one token were considered overlapping entities. The four corpora used in this study are described below. Other valuable clinical coding corpora were mentioned in Subsection 2.1, such as CARES [22] in Spanish, and MIMIC-III [20] and MIMIC-IV [23] in English. However, because the focus of this work is on the explainability component, the system proposed and evaluated in this study requires data with annotated textual mentions; that is, code–mention alignments, which unfortunately prevents their direct use in this work. CodiEsp The CodiEsp1 corpus [13] was collected for the CodiEsp shared task of CLEF eHealth 2020 [38]. It contains 1,000 clinical reports written in Spanish, covering a broad range of medical specialties, and a total of 18,435 ICD code mentions. These mentions correspond either to diagnoses, annotated with their respective ICD-10-CM codes, or to procedures, which are mapped to ICD-10-PCS codes. The dataset is organized into three subsets: training (500 documents), development (250 documents), and test (250 documents) (see Table 1). CodiEsp consists of two corpora, each with a distinct ICD variant: CodiEsp-D focuses on the automatic coding of ICD-10-CM codes, specifically the 2018 Spanish version2. In total, 2,557 ICD-10-CM codes appear at least once throughout the CodiEsp-D corpus. CodiEsp-D focuses on the automatic coding of ICD-10-CM codes, specifically the 2018 Spanish version2. In total, 2,557 ICD-10-CM codes appear at least once throughout the CodiEsp-D corpus. CodiEsp-P involves the automatic assignment of ICD-10-PCS codes, in particular the 2018 Spanish version3. The CodiEsp-P corpus covers a total of 870 unique ICD-10-PCS codes. CodiEsp-P involves the automatic assignment of ICD-10-PCS codes, in particular the 2018 Spanish version3. The CodiEsp-P corpus covers a total of 870 unique ICD-10-PCS codes. Table 1Summary statistics of the CodiEsp corpusCodiEsp-DCodiEsp-PLanguageICD variantTotal ICD codesSpanishICD-10-CM2557SpanishICD-10-PCS870TrainDevTestTrainDevTestDocuments500250250500250250Total ICD code mentions720934313665197210461112Unique ICD codes to each set807351363293129143ICD codes per document (mean)14.4213.7214.663.944.184.45Overlapping entities (pairs)1586683802668356388Discontinuous entities1047509524740457430 Summary statistics of the CodiEsp corpus The average number of ICD codes per document is notably high in CodiEsp-D, with 13 to 14 codes assigned per clinical report (see Table 1). Moreover, the test sets of both CodiEsp corpora include a large number of unseen codes, in particular, 363 in CodiEsp-D and 143 in CodiEsp-P, and a big percentage of discontinuous and overlapping textual mentions. Figure 2-A illustrates several annotation examples from both of the CodiEsp corpora. Spanish corpora examples: A. CodiEsp (ICD-10-CM codes in blue and ICD-10-PCS in red), B. Cantemist (ICD-O-3 codes in green), C. FALP (ICD-10-CM codes in blue and ICD-O-3 codes in green) Cantemist Like CodiEsp, the Cantemist4 corpus [14] was also created for a shared task, the Cantemist track of IberLEF 2020. It comprises 1,301 oncological clinical case reports written in Spanish, with a total of 16,032 ICD code mentions. The corpus is annotated with tumor morphology mentions, each associated with its corresponding ICD-O-3 code. The Cantemist corpus is divided into three subsets: training (501 documents), development (500 documents), and test (300 documents) (see Table 2). The ICD-O-3 version used for annotation is the first revision of the third Spanish edition (CIE-O-3.15). In total, 850 ICD-O-3 codes are covered in the Cantemist corpus, of which 107 are not present in the training or development sets. The corpus does not contain discontinuous textual mentions. Furthermore, the number of overlapping entities is quite low. For instance, the development set, which contains the highest number of overlaps, has 110 overlapping cases out of a total of 6,001 mentions. Figure 2-B presents an example of the annotated data in the Cantemist corpus. Table 2Summary statistics of the Cantemist corpusCantemistLanguageICD variantTotal ICD codesSpanishICD-O-3850TrainDevTestDocuments501500300Total ICD code mentions639660013635Unique ICD codes to each set183193107ICD codes per document (mean)12.7712.0012.12Overlapping entities (pairs)7911072Discontinuous entities000 Summary statistics of the Cantemist corpus FALP The FALP6 corpus [15] was created by collecting 1,101 anonymized oncological reports and includes 16,659 ICD code mentions. All annotations in the FALP corpus refer to tumor mentions and correspond either to morphological entities, mapped to their respective ICD-O-3 codes, or to topographical characteristics, annotated with ICD-10-CM codes. The corpus itself is not divided into subsets; however, in [39], a partition is proposed, consisting of 869 documents for training, 124 for development, and 108 for testing. Although the exact distribution of documents across subsets is not publicly available, this partition was adopted in the present study to ensure comparability with previous work, and a cross-validation approach will be followed when applying the system. FALP is organized into the following two corpora: FALP Morphology entails the automatic coding of ICD-O-3 codes, using the same version as the Cantemist corpus (CIE-O-3.1). A total of 226 ICD-O-3 codes appear at least once. FALP Morphology entails the automatic coding of ICD-O-3 codes, using the same version as the Cantemist corpus (CIE-O-3.1). A total of 226 ICD-O-3 codes appear at least once. FALP Topography involves the automatic classification of ICD-10-CM codes based on the 2020 Spanish edition7 [40]. Since the corpus focuses on mentions of malignant neoplasms, the number of possible codes is considerably reduced, simplifying the classification task. Furthermore, this corpus contains 153 unique ICD-O-3 codes, fewer than those found in the previously mentioned corpora, which may also facilitate the classification process. FALP Topography involves the automatic classification of ICD-10-CM codes based on the 2020 Spanish edition7 [40]. Since the corpus focuses on mentions of malignant neoplasms, the number of possible codes is considerably reduced, simplifying the classification task. Furthermore, this corpus contains 153 unique ICD-O-3 codes, fewer than those found in the previously mentioned corpora, which may also facilitate the classification process. Both test sets of the FALP corpora include a small number of unseen codes, 8 in the Morphology corpus and 4 in the Topography corpus (see Table 3). This suggests that a primarily supervised classification process may already yield sufficiently good results. None of the corpora contain discontinuous entities. However, it is worth noting the extremely high proportion of overlapping cases in the FALP Morphology corpus. For instance, the training set contains 5,368 overlapping mention pairs out of a total of 7,627 mentions. Figure 2-C presents a few annotation examples from both FALP corpora. Table 3Summary statistics of the FALP corpusFALP MorphologyFALP TopographyLanguageICD variantTotal ICD codesSpanishICD-O-3226SpanishICD-10-CM153TrainDevTestTrainDevTestDocuments869124108869124108Total ICD code mentions766310789475516780681Unique ICD codes to each set11311115344ICD codes per document (mean)8.828.698.776.356.236.31Overlapping entities (pairs)52688066492252928Discontinuous entities000000 Summary statistics of the FALP corpus MDACE The MDACE8 corpus [16] is composed of a subset of hospital records written in the English language from the MIMIC-III [20] dataset. It contains 560 Inpatient reports, corresponding to information created during a patient visit to a medical facility, and 578 Profee reports, which describe shorter outpatient visits and the provider’s work and payment. Most reports also include ICD-9 and CPT codes, but these were excluded from the study since the focus was on ICD-10 codes. The ICD-10-PCS codes from MDACE-D Inpatient were also omitted, as there were too few to train the models effectively. The Inpatient reports include 3,662 ICD code mentions, while the Profee reports contain 5,217. Thus, the MDACE corpus is divided into the following two corpora: MDACE-D Inpatient presents ICD-10-CM codes, including a total of 904 unique codes. MDACE-D Inpatient presents ICD-10-CM codes, including a total of 904 unique codes. MDACE-D Profee focuses on the coding of ICD-10-CM codes and contains 673 unique ICD codes. MDACE-D Profee focuses on the coding of ICD-10-CM codes and contains 673 unique ICD codes. None of the MDACE corpora contain discontinuous textual mentions, and they all include very few overlapping cases (see Table 4). Examples of annotations from the MDACE corpus are not shown, as the texts in the MIMIC-III corpus cannot be made public. Table 4Summary statistics of the MDACE corpusMDACE-D InpatientMDACE-D ProfeeLanguageICD variantTotal ICD codesEnglishICD-10-CM904EnglishICD-10-CM673TrainDevTestTrainDevTestDocuments332112116322112144Total ICD code mentions2245720697275011711296Unique ICD codes to each set4181061012809597ICD codes per document (mean)6.766.436.018.5410.469Overlapping entities (pairs)21912141Discontinuous entities000000 Summary statistics of the MDACE corpus System description As previously mentioned, the goal of this study is not only to assign the appropriate ICD codes to each document, but also to integrate an explainability feature within the system that justifies each predicted code. This explainability component works by providing the specific textual mentions or evidence corresponding to each code, and helps improve and ensure the robustness and reliability of the system. Figure 3 presents a schematic overview of the proposed system, which is organized into three phases. Phase 1 employs a NER-based model to identify textual mentions or evidence that could potentially lead to the assignment of an ICD code. Three distinct NER models, each with unique characteristics and arquitectures, and capable of detecting different entity types (flat, discontinuous, overlapping), were studied and evaluated. Subsequently, the extracted mentions from Phase 1 are then passed to Phase 2, which uses a supervised classification model to assign the most appropriate ICD code to each mention. This step allows us to simplify the XMTC problem by transforming it into a single-label approach, that is, assigning one code per entity instead of multiple codes per text. Moreover, because the corpora include unseen codes during training and validation, mentions with low-confidence code predictions are filtered and sent to Phase 3. In this final phase, a semantic similarity model employs a knowledge base built from code descriptions and enhanced with a keyphrase-based approach to assign the most similar ICD code. The proposed research constitutes an extension of the approach described in [41]. Flow diagram of the proposed system, showing its three phases Phases 1 and 2 of the system were addressed using Transformer-based models, which capture bidirectional context and generate richer semantic and syntactic representations. In recent years, several studies have employed this type of architecture to identify medical concepts [42–44] and assign ICD codes [31, 35, 45], showing notable improvements in most cases. All the Transformer-based models used were pretrained in the biomedical or clinical domain and publicly available on the HuggingFace platform9. For training and inference on the Spanish corpora, four pretrained models were utilized, while two were employed for the English corpora (see Table 5). Table 5Pretrained transformer-based models used in phases 1 and 2Spanishbsc-bio-ehr-es [46]RoBERTa-based model with 12 layers, pretrained on two large Spanish corpora: over 514,000 clinical documents and 2.5 million biomedical texts.roberta-base-biomedical-clinical-es [47]Based on RoBERTa base Spanish model, pretrained on multiple clinical Spanish corpora from public sources, web crawlers, and a real-world dataset with over 278,000 documents.EriBERTa-base [48]RoBERTa-based model (125 M parameters), pretrained on clinical texts in Spanish and English from the Multilingual Medical Corpus [49], including SPACCC [50] and EMEA [51] datasets.longformer-base-4096-biomedical-clinical-esLongformer-based model for long sequences (up to 4,096 tokens), pretrained on the same corpora as bsc-bio-ehr-es, enabling processing of longer clinical documents.Englishbiobert-v1.1 [52]BERT-based model pretrained on large biomedical texts: PubMed abstracts and PubMed Central full articles.Bio_ClinicalBERT [53]Initialized from BioBERT, pretrained on clinical notes from the MIMIC-III database [20]. Pretrained transformer-based models used in phases 1 and 2 Phase 1: entity extraction The first stage of the system employs a NER-based approach to identify text spans that may potentially correspond to an ICD code. In total, three different NER systems were evaluated. They are described below: SimpleNER is a token classification model developed from scratch. Texts were annotated at the token level using the BIO (Beginning, Inside, Outside) [54] annotation scheme. As the annotation process allows only one label per token, in cases of overlapping mentions, the longest span was selected, as recommended in [55].The models were trained for a maximum of 50 epochs, using the F1-micro score for validation and applying early stopping based on the performance on the development set. SimpleNER is a token classification model developed from scratch. Texts were annotated at the token level using the BIO (Beginning, Inside, Outside) [54] annotation scheme. As the annotation process allows only one label per token, in cases of overlapping mentions, the longest span was selected, as recommended in [55]. The models were trained for a maximum of 50 epochs, using the F1-micro score for validation and applying early stopping based on the performance on the development set. GliNER [56] is a model for entity extraction that uses a Bidirectional Language Model (BiLM) like BERT, along with entity type prompts marked by a special ENT token. GLiNER has been shown to outperform LLMs like ChatGPT in zero-shot tasks while requiring fewer resources, and it also performs well across multiple languages.For GliNER, the pretrained multilingual gliner_multi model was fine-tuned on the corpora for 25 epochs. It was selected after evaluating the performance of three publicly available GliNER multilingual models on HuggingFace. During decoding, GliNER uses a greedy span selection strategy, considering only entity spans with scores above a predefined threshold. In this study, the threshold was set to 0.5, following the recommendation in [56]. GliNER [56] is a model for entity extraction that uses a Bidirectional Language Model (BiLM) like BERT, along with entity type prompts marked by a special ENT token. GLiNER has been shown to outperform LLMs like ChatGPT in zero-shot tasks while requiring fewer resources, and it also performs well across multiple languages. For GliNER, the pretrained multilingual gliner_multi model was fine-tuned on the corpora for 25 epochs. It was selected after evaluating the performance of three publicly available GliNER multilingual models on HuggingFace. During decoding, GliNER uses a greedy span selection strategy, considering only entity spans with scores above a predefined threshold. In this study, the threshold was set to 0.5, following the recommendation in [56]. WNER [57] is a model that reformulates the NER task as a word-word relation classification problem. It builds a 2D grid of word pairs and uses relations like Next-Neighboring-Word (NNW) and Tail-Head-Word (THW). The architecture combines a BERT and BiLSTM encoder, and 2D convolutions to classify word-word relations. The original paper reports that WNER achieves state-of-the-art performance on 14 benchmark datasets across flat, overlapped, and discontinuous NER tasks in both English and Chinese.Due to its higher complexity, the training process in WNER typically takes longer than in simpler NER-based systems. Therefore, training was run for up to 150 epochs, selecting the model from the epoch that achieved the highest F1-micro score on the development set. Additionally, the pretrained model longformer-base-4096-biomedical-clinical-es could not be used, as the WNER architecture only supports BERT-based models. WNER [57] is a model that reformulates the NER task as a word-word relation classification problem. It builds a 2D grid of word pairs and uses relations like Next-Neighboring-Word (NNW) and Tail-Head-Word (THW). The architecture combines a BERT and BiLSTM encoder, and 2D convolutions to classify word-word relations. The original paper reports that WNER achieves state-of-the-art performance on 14 benchmark datasets across flat, overlapped, and discontinuous NER tasks in both English and Chinese. Due to its higher complexity, the training process in WNER typically takes longer than in simpler NER-based systems. Therefore, training was run for up to 150 epochs, selecting the model from the epoch that achieved the highest F1-micro score on the development set. Additionally, the pretrained model longformer-base-4096-biomedical-clinical-es could not be used, as the WNER architecture only supports BERT-based models. The textual ICD mentions in each of the corpora used in this study present different characteristics. Some corpora contain both discontinuous and overlapping mentions, while others include only different amounts of overlapping entities. As summarized in Table 6, the SimpleNER model supports only flat mentions. However, since it uses the BIO annotation schemes, certain discontinuous mentions that do not overlap with others may still be considered. Hence, we theorize that SimpleNER could obtain good results for Cantemist, FALP Topography, and MDACE, which contain mostly flat entities with minimal overlapping cases. Moreover, GliNER handles flat and overlapping mentions, so it may be well suited for FALP Morphology, where overlapping entities are more frequent. Finally, the WNER system supports all types of mentions, and thus is expected to perform well across all corpora, with a clear advantage in CodiEsp, which includes flat, discontinuous, and overlapping mentions. Comparing the three NER-based models allows us to balance performance and resource usage, providing insight into which model is more suitable depending on the level of complexity of the corpora. Table 6Key characteristics of the NER-based systemsSimpleNERGliNERWNERSupports flat entitiesSupports overlapping entitiesSupports discontinuous entitiesSupports only discontinuous entities that do not overlap with other entities Key characteristics of the NER-based systems Supports only discontinuous entities that do not overlap with other entities Phase 2: supervised classification Once the potential ICD code mentions are extracted from the clinical reports, a supervised multi-class classification model assigns a code to each mention. In the case of ICD-10-CM codes, some state-of-the-art studies have designed classification systems that leverage their hierarchical structure [31, 39]. However, one of the main goals of this study is to build a unified and robust system for the three ICD code variants considered. Since the structures of ICD-10-PCS and ICD-O-3 are not hierarchical but multiaxial, a hierarchical-based approach was not considered. Therefore, for each corpus, a classification model was finetuned to assign a complete ICD code to each extracted mention. The Optuna framework [58] was employed for hyperparameter optimization. Each model was trained for up to 50 epochs, with early stopping based on the F1-micro score on the development set. The set of possible labels for the classification models included only those ICD codes present in the training and development sets. However, the test sets across the eight corpora contain codes that were not seen during training or validation. In such cases, the model cannot possibly identify the correct code. To address this issue, a probability threshold-based strategy was applied. For each mention, if the predicted code’s softmax probability was greater than or equal to a predefined threshold (), the prediction was accepted and included in the final output. On the other hand, if the probability was below the threshold (), the instance was rejected and passed on to Phase 3. This approach ensures that predictions where the classification model lacks sufficient confidence (most of which corresponding to unseen codes that cannot be correctly assigned) are filtered out and handled in the next phase. Two probability thresholds () were optimized for each corpora, corresponding to the two main metrics used to evaluate the results: F1-micro and Mean Average Precision (MAP). Phase 3: semantic similarity classification In this final stage, the model receives only the textual mentions that were discarded during Phase 2. Most of these likely correspond to codes not present within the training or development sets, as the classification model was not confident enough to assign them a label. Since no predefined set of possible labels exists in this phase, an unsupervised approach must be implemented. To develop this unsupervised system, it was first necessary to gather as much information as possible about each ICD code. The organizers of the CodiEsp and Cantemist tasks provided a list of valid codes for each ICD variant: 98,288 ICD-10-CM codes, 87,170 ICD-10-PCS codes, and 58,062 ICD-O-3 codes. However, not all of the codes were given a description and, in addition, the code descriptions alone may not provide enough information to assign the correct code to each textual mention, as many descriptions are phrased differently than how they appear in the reports. To enrich the information available for each code, the system proposed in [30] was implemented. This methodology combines two keyphrase extraction approaches. The first makes use of a taylored regular expression for identifying Spanish medical noun and prepositional keyphrases, extracted after some pre-processing steps such as lemmatization and POS-tagging and finally filtered and selected using TF-IDF techniques. The second employs IXAMedTagger [59] to annotate terms from SNOMED-CT across categories. Both approaches are complementary, since the regular expression captures more general keyphrases, whereas IXAMedTagger ensures the inclusion of specific medical terms. Then, the knowledge base is created by assessing the co-occurrence of the extracted keyphrases and ICD-10 codes across the reports. In particular, the statistical significance of each pair (keyphrase, ICD-code) is calculated. The result is a list of keyphrases associated to each ICD-10 code, ordered by the significance of that association. The described system was applied to all four corpora using the training and development sets, enabling the creation of a knowledge base. Following the results of some exploratory experiments, the two most representative keyphrases per code, according to their significance, were selected for populating the knowledge base. The result was a list of the three ICD code variants considered in this study, in which each code may include one of the following types of information: A description and its two most significant keyphrases, if we have a description of the code and keyphrases were found in the corpus. A description and its two most significant keyphrases, if we have a description of the code and keyphrases were found in the corpus. Only a description, if the organizers of the corpora provided a description but no keyphrases were found, likely because the code was not present in the training and development sets of the four corpora. Only a description, if the organizers of the corpora provided a description but no keyphrases were found, likely because the code was not present in the training and development sets of the four corpora. Only the two most significant keyphrases, if keyphrases were computed for the code but no description was available. Only the two most significant keyphrases, if keyphrases were computed for the code but no description was available. To further enrich the information attached to each code, the keyphrases computed from the dataset used in [30] were also incorporated. This corpus consists of 12,966 Spanish medical discharge reports from 2016 and includes 7,592 ICD-10-CM and ICD-10-PCS codes. Moreover, because the knowledge base was built in Spanish, when the system is applied to the English MDACE corpus, the text spans processed in Phase 3 are translated into Spanish to compute their embeddings and cosine similarity with the knowledge base information. It is worth noting that this introduces a limitation and may result in a translation bias, potentially causing changes in semantics and structure and affecting the results. For example, even words written in a different order may yield a different cosine similarity. However, since our knowledge base is in Spanish, one of the components must be translated, and we consider that translating only the entities that reach Phase 3 likely has less impact than translating the entire knowledge base. Having collected the list of possible ICD codes along with their corresponding descriptions and keyphrases for each corpus, the embeddings of the entities, descriptions, and keyphrases were computed using the Arctic-Embed 2.0 multilingual embedding model [60]. For each entity, its embedding was compared to the embeddings of all ICD code descriptions and keyphrases using the Sentence Transformers framework [61]. For each comparison pair, the cosine similarity was computed. Figure 4 shows a few examples of how Phase 3 works. Example diagram illustrating the process in phase 3 Although the code with the highest similarity was selected, it is possible that the similarity score was low or that some gold standard codes from the test set were not present in the list of possible codes. To address this issue, another threshold-based strategy was applied, using cosine similarity as the criterion. If the cosine similarity between an entity and its assigned code was greater than or equal to a predefined threshold (), the entity and its assigned code were included in the final predictions. Conversely, if the similarity was below the threshold (), such prediction was permanently discarded and excluded from the output, as none of the classification models from Phases 2 and 3 were able to confidently assign a code to it. For each corpora, two thresholds () were optimized, corresponding to the two metrics used for evaluation. Metrics and evaluation Two main metrics were used to evaluate the models’ performance. The first was the F1-micro score, computed from Precision and Recall, as detailed below: , where is the number of true positives, the false positives, and the false negatives. The second main evaluation metric was Mean Average Precision (MAP), widely used for ranking tasks: , where is the precision at position , and is an indicator function that equals 1 if the item at rank is a relevant document, and 0 otherwise. The explainability results were only evaluated using the F1 metric, and not other entity linking metrics, since a code and its textual mention only need to be correctly predicted once to be considered correct. Optimized thresholds As the task considers two main metrics, F1 and MAP, the thresholds and of Phases 2 and 3, respectively, were evaluated on the development set to optimize each metric. Table 7 presents the resulting thresholds that optimize each metric in each corpus. We can observe that the threshold values that maximize the F1 scores are higher, that is, more restrictive, than those that maximize the MAP metric. This introduces a trade-off: higher thresholds improve Precision and F1 by imposing stricter confidence requirements and filtering more instances, while lower thresholds allow more instances through, improving Recall and MAP. Table 7Optimized thresholds per corpus to maximize the F1 and MAP metricsMax. F1Max. MAPCodiEsp-D0.500.850.400.70CodiEsp-P0.550.900.350.80Cantemist0.600.950.300.75FALP Morphology0.500.850.400.70FALP Topography0.500.900.350.75MDACE-D Inpatient0.500.900.400.75MDACE-D Profee0.500.950.400.80 Optimized thresholds per corpus to maximize the F1 and MAP metrics Experiments and results This section presents the experiments conducted to evaluate the proposed approach and the results obtained. Additionally, an error analysis of the system is performed to highlight its main areas for improvement. Results The results are organized into two main sections. First, we present the clinical coding results, where we evaluate only the classification of ICD codes for each text report. Then, we provide the results of the explainability task, in which, in addition to the ICD code classification, the predicted text spans associated with each code are also evaluated. Clinical coding When evaluating the task of clinical coding, only the correct assignment of ICD codes to each text was evaluated. For each corpus, different configurations were considered: the three NER-based models described in Section “Phase 1: Entity Extraction”, along with their corresponding ensemble versions. The ensemble strategy employed is a union, in which all instances predicted by the respective models are included. Since threshold optimization introduces a trade-off between precision-oriented performance (F1) and ranking quality (MAP), each system was evaluated under both criteria by optimizing the thresholds separately for F1 and MAP. For each system, results for Precision (P), Recall (R), F1-score (F1), and Mean Average Precision (MAP) metrics are reported. Table 8 presents the results of the clinical coding task for the CodiEsp corpus. The clinical coding of diagnoses (CodiEsp-D) appears to be an easier task than the coding of medical procedures (CodiEsp-P), as reflected by the higher results obtained. Moreover, Precision tends to be higher than Recall, but the F1 values, which balance both metrics, are generally satisfactory, as will be shown later in the comparison with state-of-the-art results. The WNER MF1 model achieves the best Precision overall and the highest F1 in CodiEsp-D. Ensemble systems that include WNER, optimized for the MAP metric (MMAP), achieve the highest Recall and MAP in both tasks. Table 8Clinical coding results for the two CodiEsp subcorpora: CodiEsp-D and CodiEsp-PSystemCodiEsp-DCodiEsp-PPRF1MAPPRF1MAPSimpleNER MF10.7890.5330.6360.4900.7640.3620.4910.457GliNER MF10.7720.6260.6910.5650.6980.4110.5170.449WNER MF10.8010.6520.7190.5900.7830.4690.5860.540SimpleNER + GliNER MF10.7390.6650.7000.5930.6720.4310.5250.464SimpleNER + WNER MF10.7620.6790.7180.6150.7390.4870.5870.545GliNER + WNER MF10.7420.6880.7140.6200.6830.4980.5760.520SimpleNER MMAP0.7360.5460.6270.4990.7480.3720.4970.460GliNER MMAP0.7160.6330.6720.5700.6620.4300.5210.449WNER MMAP0.7500.6610.7030.5970.7460.4810.5850.544SimpleNER + GliNER MMAP0.6770.6760.6770.5990.6360.4510.5270.467SimpleNER + WNER MMAP0.7050.6910.6980.6210.7030.5020.5860.558GliNER + WNER MMAP0.6810.6970.6890.6250.6430.5180.5730.515MF1 systems use the probability and similarity thresholds that maximize the F1-score, whereas MMAP systems use the thresholds that maximize MAP. The best performance for each metric and evaluation is highlighted in bold, and the second-best is underlined Clinical coding results for the two CodiEsp subcorpora: CodiEsp-D and CodiEsp-P MF1 systems use the probability and similarity thresholds that maximize the F1-score, whereas MMAP systems use the thresholds that maximize MAP. The best performance for each metric and evaluation is highlighted in bold, and the second-best is underlined In the Cantemist corpus (see Table 9), Precision and Recall are much more balanced than in CodiEsp, where a large number of false negatives were observed. As in CodiEsp-D, the WNER MF1 system achieves the highest Precision and F1, while the ensemble system GliNER + WNER MMAP obtains the best Recall and MAP. Table 9Clinical coding results for the Cantemist corpusSystemCantemistPRF1MAPSimpleNER MF10.8860.7380.8050.745GliNER MF10.8750.8040.8380.788WNER MF10.8970.8200.8570.821SimpleNER + GliNER MF10.8480.8320.8400.812SimpleNER + WNER MF10.8680.8340.8510.832GliNER + WNER MF10.8530.8420.8470.839SimpleNER MMAP0.8210.7520.7850.755GliNER MMAP0.8060.8110.8090.793WNER MMAP0.8340.8310.8320.827SimpleNER + GliNER MMAP0.7690.8460.8060.819SimpleNER + WNER MMAP0.7920.8490.8200.840GliNER + WNER MMAP0.7750.8520.8120.844MF1 systems use the probability and similarity thresholds that maximize the F1-score, whereas MMAP systems use the thresholds that maximize MAP. The best performance for each metric and evaluation is highlighted in bold, and the second-best is underlined Clinical coding results for the Cantemist corpus MF1 systems use the probability and similarity thresholds that maximize the F1-score, whereas MMAP systems use the thresholds that maximize MAP. The best performance for each metric and evaluation is highlighted in bold, and the second-best is underlined The results of the FALP corpus (see Table 10) are considerably higher than those of the previous corpora, with the majority of results scoring over 0.9. This suggests that the task is less complex, likely because the total number of ICD codes in the FALP corpus is smaller than in other corpora. As in CodiEsp-D and Cantemist, the WNER MF1 model achieves the best Precision in both tasks and the best Recall and MAP results are obtained in both corpora by two ensemble systems maximized for the MAP metric. The best F1 scores are achieved in this case by the SimpleNER + WNER MF1 model in both tasks. In FALP, there is little difference when using different thresholds, because there are less codes and very few are absent from the training and development sets (see Table 3). As a result, the supervised model has a high level of confidence in most instances, so, regardless of the threshold used, Phase 2 already selects over 95% of the codes. For FALP Topography, in Phase 3, instead of using the entire knowledge base (including all ICD-10-CM codes), we apply the prerequisite that the selected code must begin with character C, since this corpus describes neoplasms and all its codes fulfill this criteria. Table 10Clinical coding results for the two FALP subcorpora: FALP morphology and FALP topographySystemFALP MorphologyFALP TopographyPRF1MAPPRF1MAPSimpleNER MF10.9450.3750.5370.5190.9080.6150.7330.793GliNER MF10.9480.8980.9220.8990.8330.8180.8250.843WNER MF10.9490.9280.9380.9270.9100.8950.9020.909SimpleNER + GliNER MF10.9410.9120.9260.9110.8190.8690.8430.874SimpleNER + WNER MF10.9430.9370.9400.9330.9030.9040.9040.914GliNER + WNER MF10.9370.9410.9390.9370.8180.8980.8610.914SimpleNER MMAP0.9270.3780.5360.5210.9010.6130.7350.785GliNER MMAP0.9370.8990.9180.9000.8190.8190.8180.839WNER MMAP0.9370.9290.9340.9280.9070.8960.9020.909SimpleNER + GliNER MMAP0.9250.9130.9190.9110.8060.8700.8360.871SimpleNER + WNER MMAP0.9280.9380.9330.9340.8980.9070.9020.915GliNER + WNER MMAP0.9200.9420.9310.9370.8060.9120.8560.915MF1 systems use the probability and similarity thresholds that maximize the F1-score, whereas MMAP systems use the thresholds that maximize MAP. The best performance for each metric and evaluation is highlighted in bold, and the second-best is underlined Clinical coding results for the two FALP subcorpora: FALP morphology and FALP topography MF1 systems use the probability and similarity thresholds that maximize the F1-score, whereas MMAP systems use the thresholds that maximize MAP. The best performance for each metric and evaluation is highlighted in bold, and the second-best is underlined In the MDACE corpus (see Table 11), the results for MDACE-D Profee are slightly higher than those for MDACE-D Inpatient. In the MDACE-D Inpatient subcorpus, both F1 and MAP scores remain below 0.5. As in previous corpora, the WNER MF1 system achieves the highest Precision in both subcorpora. In contrast, when optimizing for the MAP metric, the ensemble of GliNER and WNER yields the best Recall and MAP performance. Table 11Clinical coding results for the two MDACE subcorpora: MDACE-D Inpatient and MDACE-D ProfeeSystemMDACE-D InpatientMDACE-D ProfeePRF1MAPPRF1MAPSimpleNER MF10.3900.2730.3210.2020.5780.2760.3740.348GliNER MF10.3600.4840.4130.2710.4760.4990.4880.406WNER MF10.5920.3670.4540.3110.6370.3910.4840.429SimpleNER + GliNER MF10.3230.5240.4000.2910.4540.5310.4900.418SimpleNER + WNER MF10.4440.4450.4440.3050.5780.4700.5190.445GliNER + WNER MF10.3650.5260.4310.3310.4710.5420.5040.455SimpleNER MMAP0.3120.2810.2960.2200.4970.2790.3580.351GliNER MMAP0.2860.4650.3540.2910.3930.5090.4440.408WNER MMAP0.5420.3780.4460.3240.5740.3970.4690.447SimpleNER + GliNER MMAP0.2480.5210.3360.2980.3670.5370.4360.424SimpleNER + WNER MMAP0.3640.4570.4050.3170.5020.4780.4900.450GliNER + WNER MMAP0.2930.5320.3780.3430.3870.5510.4550.466MF1 systems use the probability and similarity thresholds that maximize the F1-score, whereas MMAP systems use the thresholds that maximize MAP. The best performance for each metric and evaluation is highlighted in bold, and the second-best is underlined Clinical coding results for the two MDACE subcorpora: MDACE-D Inpatient and MDACE-D Profee MF1 systems use the probability and similarity thresholds that maximize the F1-score, whereas MMAP systems use the thresholds that maximize MAP. The best performance for each metric and evaluation is highlighted in bold, and the second-best is underlined Finally, as can be seen in the overall results, when optimizing for F1 (MF1 systems), the thresholds become stricter, resulting in fewer selected instances, higher Precision, and lower Recall and MAP. In contrast, when optimizing for MAP (MMAP systems), the thresholds are more relaxed, leading to more selected instances, lower Precision, and higher Recall. Moreover, if the priority is Precision or F1, the most appropriate system is WNER MF1, whereas if the focus is on Recall or MAP, the most suitable system in most cases is an ensemble, particularly GliNER + WNER MMAP. Explainability The task of explainable clinical coding requires not only the correct assignment of ICD codes to each clinical report, but also the identification of the text spans corresponding to each ICD prediction. Each corpus was evaluated using the same configurations as before: the three NER-based models described in Section “Phase 1: Entity Extraction”, along with their corresponding ensemble systems, each using the respective thresholds that maximize the F1 metric, which is the metric evaluated in this task. Again, the same ensemble strategy as in Subsection “Clinical Coding” has been employed, that is, all items predicted by the respective models are included. For each system, results for Precision (P), Recall (R), and F1-score (F1) metrics are reported. Table 12 presents the results of the explainability task for the CodiEsp corpus. The predictions of CodiEsp-D and CodiEsp-P have been combined using a union strategy into CodiEsp-X, as state-of-the-art results are reported this way. The best F1-score is achieved by WNER MF1. As shown in Table 6, WNER is the only system that supports all types of entities. Since the CodiEsp corpus contains a considerable percentage of overlapping and discontinuous entities, this may be the main reason for its superior performance. The GliNER method allows the inclusion of overlapping entities, so its results are higher than those of the systems that use the SimpleNER models. The system WNER MF1 obtains the best Precision and F1 values, 0.768 and 0.684, respectively, while the GliNER + WNER MF1 achieves the highest Recall, 0.617. Table 12Explainability results for the CodiEsp corpusSystemCodiEsp-XPRF1SimpleNER0.7450.4680.575GliNER0.7320.5540.631WNER0.7680.5850.684SimpleNER + GliNER0.7000.5870.639SimpleNER + WNER0.7310.6090.665GliNER + WNER0.7030.6170.657 Explainability results for the CodiEsp corpus The Cantemist corpus does not contain discontinuous entities (see Table 2), so the difference of results between WNER and GliNER systems is not quite high (see Table 13). Moreover, the corpus has some overlapping entities, so the SimpleNER systems obtain lower results. The system WNER MF1 obtains the best Precision and F1 values, while GliNER + WNER MF1 achieves the highest Recall. Table 13Explainability results for the Cantemist corpus. The best performance for each metric and evaluation is highlighted in bold, and the second-best is underlinedSystemCantemistPRF1SimpleNER0.8500.7080.772GliNER0.8510.7820.815WNER0.8760.8020.837SimpleNER + GliNER0.8250.8110.818SimpleNER + WNER0.8470.8150.831GliNER + WNER0.8320.8220.827For comparability, the CodiEsp-D and CodiEsp-P predictions have been combined. The best performance for each metric and evaluation is highlighted in bold, and the second-best is underlined Explainability results for the Cantemist corpus. The best performance for each metric and evaluation is highlighted in bold, and the second-best is underlined For comparability, the CodiEsp-D and CodiEsp-P predictions have been combined. The best performance for each metric and evaluation is highlighted in bold, and the second-best is underlined The FALP corpus does not contain discontinuous entities but includes overlapping cases, particularly in FALP Morphology, where the proportion of overlapping entities is extremely high (see Table 3). This is evident in the results (see Table 14), as the SimpleNER method, which does not support overlapping entities, achieves very low Recall values in both subcorpora compared to those obtained by the GliNER and WNER systems. The WNER MF1 system achieves the highest Precision in both subcorpora and the highest F1 score in FALP Topography. In contrast, the GliNER + WNER MF1 system attains the highest Recall and F1 in FALP Morphology, while in FALP Topography, the best Recall is achieved by the SimpleNER + WNER MF1 system. Table 14Explainability results for the two FALP subcorpora: FALP morphology and FALP topographySystemFALP MorphologyFALP TopographyPRF1PRF1SimpleNER0.8940.3550.5080.8630.5800.693GliNER0.9280.8800.9030.8110.7970.803WNER0.9330.9120.9220.8860.8700.878SimpleNER + GliNER0.9220.8950.9080.7930.8420.816SimpleNER + WNER0.9250.9200.9230.8770.8780.877GliNER + WNER0.9320.9270.9260.7930.8420.816The best performance for each metric and evaluation is highlighted in bold, and the second-best is underlined Explainability results for the two FALP subcorpora: FALP morphology and FALP topography The best performance for each metric and evaluation is highlighted in bold, and the second-best is underlined Table 15 presents the results of the explainability task for the MDACE corpus. The percentage of overlapping entities in the corpus is very low, and it does not contain any discontinuous entities (see Table 4). Consequently, the differences in F1 values between WNER and GliNER are relatively small, while SimpleNER continues to show limited performance. The explainability task in the MDACE corpus appears to be more challenging than in the previous corpora, as both subcorpora exhibit low overall performance, with F1 scores below 0.5. Table 15Explainability results for the two MDACE subcorpora: MDACE-D Inpatient and MDACE-D ProfeeSystemMDACE-D InpatientMDACE-D ProfeePRF1PRF1SimpleNER MF10.2680.1880.2210.4670.2330.302GliNER MF10.2750.3690.3150.4150.4350.424WNER0.5020.3120.3850.5560.3410.423SimpleNER + GliNER MF10.2520.4080.3110.3950.4620.426SimpleNER + WNER MF10.3630.3640.3640.5000.4070.449GliNER + WNER MF10.2750.3690.3150.4210.4830.450The best performance for each metric and evaluation is highlighted in bold, and the second-best is underlined Explainability results for the two MDACE subcorpora: MDACE-D Inpatient and MDACE-D Profee The best performance for each metric and evaluation is highlighted in bold, and the second-best is underlined Handling unseen codes As previously mentioned, the end-to-end system proposed in this study is organized into three phases. Phase 2 consists of a supervised classification model trained to predict the set of codes observed during training. Therefore, by design, Phase 2 cannot classify any unseen codes, and, as stated in Subsection “Corpora”, all the corpora used in this study contain such unseen codes. The problem of unseen codes in clinical coding has not been widely addressed, which motivates the creation of Phase 3 to enable their classification. Table 16 shows, for Phases 2 and 3, the proportion of correctly predicted codes among those accepted in each phase; that is, predictions that surpass the threshold of the corresponding phase and are included in the final set of predictions. For illustration, we present the results from a single system, specifically the WNER MF1 system, because it achieved the highest F1 score in most tasks. The table also shows the number of unseen codes predicted in Phase 3. As noted, Phase 2 does not allow for the prediction of unseen codes, so Phase 3 primarily handles the most challenging instances. Table 16For each corpus, the proportion of correctly predicted codes among those selected at each thresholdCorpusProportion of correctly predicted codesUnseen codes correctly predictedPhase 2Phase 3CodiEsp-D86.26%44.62%26CodiEsp-P82.03%35.36%2Cantemist94.96%68.57%19FALP Morphology97.69%83.33%3FALP Topography95.78%0.00%0MDACE-D Inpatient63.06%38.00%5MDACE-D Profee66.10%72.15%31The number of unseen codes correctly recovered is also included. The results correspond to those obtained by the WNER MF1 system For each corpus, the proportion of correctly predicted codes among those selected at each threshold The number of unseen codes correctly recovered is also included. The results correspond to those obtained by the WNER MF1 system We observe that the MDACE-D Profee corpus is where the most unseen codes have been successfully recovered. Figure 5 illustrates the flow of its instances through each phase of the system. Numbers in brackets indicate the number of instances at each stage. Most instances are accepted in Phase 2, while the rejected ones proceed to Phase 3. In this context, “accepted” means that they have surpassed the threshold of the corresponding phase, regardless of whether they are correct or not. Phase 3’s threshold accepts 79 instances, of which 57 are correctly predicted, including 31 unseen codes and 26 codes seen during training. In contrast, the FALP Topography corpus did not recover any unseen codes in Phase 3. This is because its test set contains only four unseen codes, while the rest were seen during training. Consequently, the classification task in Phase 2 is much easier, and the model is likely confident in most predictions; in fact, only five instances are passed on to Phase 3. Sankey diagram illustrating the flow of the predicted instances through the three phases of the proposed system on the MDACE-D profee corpus. Numbers in brackets indicate the number of instances State-of-the-art comparison Most of the results obtained in this study can be compared with those reported in the state of the art. As previously, we first present the results of the clinical coding task, followed by those corresponding to the explainability task. To avoid redundancy and focus on the most relevant results, only the system with the highest F1 score is reported for each task and corpus. For the clinical coding task specifically, the system with the highest MAP value is also included. To ensure a fair comparability, the evaluation script used for all tasks and corpora was the one provided by the CodiEsp organizers10, except for the Cantemist task, where we employed the script released by the Cantemist organizers11. Clinical coding For each system, results for F1-score (F1) and Mean Average Precision (MAP) metrics are reported. Table 17 presents a state-of-the-art comparison of results for the clinical coding task on the CodiEsp corpus. In both CodiEsp subcorpora, our results outperform most state-of-the-art systems, with improvements of 4.66% and 1.38% in the F1 scores of CodiEsp-D and CodiEsp-P, respectively, and an increase of 2.57% in the MAP value of CodiEsp-P. Regarding the MAP in CodiEsp-D, [63] achieved a higher score by using an ensemble of three Transformer-based models pre-trained on Spanish oncology clinical cases and fine-tuned for a multilabel sentence classification task. Table 17Comparison of clinical coding results for the two best systems (the one with highest F1 and with highest MAP) and the three best state-of-the-art systems for the CodiEsp subcorpora: CodiEsp-D and CodiEsp-PCodiEsp-DCodiEsp-PSystemF1MAPSystemF1MAPWNER MF10.7190.590SimpleNER + WNER MF10.5870.545GliNER + WNER MMAP0.6890.625SimpleNER + WNER MMAP0.5860.558IAM [62]0.6870.521Clinical Coding [63]0.5790.544Clinical Coding [63]0.6770.662IAM [62]0.5220.493IXA-AAA [64]0.0090.530The Mental Strokers [65]0.4880.445MF1 systems use the probability and similarity thresholds that maximize the F1-score, whereas MMAP systems use the thresholds that maximize MAP. The best performance for each metric and evaluation is highlighted in bold Comparison of clinical coding results for the two best systems (the one with highest F1 and with highest MAP) and the three best state-of-the-art systems for the CodiEsp subcorpora: CodiEsp-D and CodiEsp-P MF1 systems use the probability and similarity thresholds that maximize the F1-score, whereas MMAP systems use the thresholds that maximize MAP. The best performance for each metric and evaluation is highlighted in bold In the Cantemist corpus, WNER MF1 obtains the best F1 value, which represents an improvement of 0.23% (see Table 18). In terms of the MAP metric [63], remains the strongest, surpassing our highest MAP. The study [66] also obtains a higher MAP, and for their approach they use a BERT token classification model, and generate four predictions per token: the BIO tag, the first three digits of the ICD-O-3 code, the fourth digit, and the last positions of the code (see Figure 1). Table 18Comparison of clinical coding results for the two best systems (the one with highest F1 and with highest MAP) and the three best state-of-the-art systems for the Cantemist corpusSystemCantemistF1MAPWNER MF10.8570.821GliNER + WNER MMAP0.8120.844Vicomtech [66]0.8550.847Clinical Coding [63]0.8420.884Fadi [67]0.8320.797MF1 systems use the probability and similarity thresholds that maximize the F1-score, whereas MMAP systems use the thresholds that maximize MAP. The best performance for each metric and evaluation is highlighted in bold Comparison of clinical coding results for the two best systems (the one with highest F1 and with highest MAP) and the three best state-of-the-art systems for the Cantemist corpus MF1 systems use the probability and similarity thresholds that maximize the F1-score, whereas MMAP systems use the thresholds that maximize MAP. The best performance for each metric and evaluation is highlighted in bold With respect to the corpus FALP, we can only make comparisons with the subcorpus FALP Morphology, as its results are the only ones reported (see Table 19). Our results achieve the highest F1 score with the SimpleNER + WNER MF1 system, which represents an improvement of 3.64%. However, the best MAP is obtained by the state-of-the-art system [39]. In their study, they propose a novel architecture that leverages the hierarchical structure of ontologies to cluster concepts, estimate the likelihood of each document belonging to a cluster, and then rank codes within each cluster based on these probabilities. Table 19Comparison of clinical coding results for the two best systems (the one with highest F1 and with highest MAP) and the three best state-of-the-art systems for the FALP subcorpora: FALP morphology and FALP topographySystemFALP MorphologyF1MAPSimpleNER + WNER MF10.9400.933GliNER + WNER MMAP0.9310.937Divide and Conquer [39]0.9070.964MF1 systems use the probability and similarity thresholds that maximize the F1-score, whereas MMAP systems use the thresholds that maximize MAP. The best performance for each metric and evaluation is highlighted in bold Comparison of clinical coding results for the two best systems (the one with highest F1 and with highest MAP) and the three best state-of-the-art systems for the FALP subcorpora: FALP morphology and FALP topography MF1 systems use the probability and similarity thresholds that maximize the F1-score, whereas MMAP systems use the thresholds that maximize MAP. The best performance for each metric and evaluation is highlighted in bold For MDACE, the only reported results are for the F1 score in the MDACE-D Inpatient subcorpus, as shown in Table 20. Moreover, the state-of-the-art results make direct use of the gold standard entities, so we evaluated Phases 2 and 3 directly on them. Therefore, it is important to note that these results reflect the performance of the classification phases only, assuming perfect NER, and thus cannot be interpreted as the performance of a fully end-to-end system. Under these conditions, PerfectNER, a version of our system that assumes perfect entity extraction by using the gold mentions, achieves the highest F1 score, outperforming the state-of-the-art system by 3.39%. In [68], they evaluated the performance of three Large Language Models (LLMs), such as BioMistral [69], using different prompts, when directly given the gold standard entities. Table 20Comparison of clinical coding results for the two best systems and the top state-of-the-art system on MDACE-D inpatient corpusMDACE-D Inpatient (with gold entities)SystemF1MAPPerfectNER MF10.6710.564PerfectNER MMAP0.6550.581Entity linking [68]0.649-For comparability, results were computed by directly applying phases 2 and 3 to the gold standard entities, a configuration called PerfectNER, which assumes perfect entity extraction. MF1 systems use the probability and similarity thresholds that maximize the F1-score, whereas MMAP systems use the thresholds that maximize MAP. The best performance for each metric and evaluation is highlighted in bold. The blank cell indicates that the respective research did not report a result for the MAP metric Comparison of clinical coding results for the two best systems and the top state-of-the-art system on MDACE-D inpatient corpus For comparability, results were computed by directly applying phases 2 and 3 to the gold standard entities, a configuration called PerfectNER, which assumes perfect entity extraction. MF1 systems use the probability and similarity thresholds that maximize the F1-score, whereas MMAP systems use the thresholds that maximize MAP. The best performance for each metric and evaluation is highlighted in bold. The blank cell indicates that the respective research did not report a result for the MAP metric Explainability Table 21 presents a state-of-the-art comparison of the results for the explainability task on the CodiEsp and Cantemist corpora. It is not possible to compare the explainability results for the rest of the corpora, as no results have been reported. For each system, results for the F1-score (F1) metric are reported. Table 21Comparison of explainability results for the best system and the three best state-of-the-art systems for the CodiEsp and cantemist corporaCodiEsp-XCantemistSystemF1SystemF1WNER0.684WNER0.837IAM [62]0.611HITSZ-ICRC [70]0.825FLE [71]0.611Vicomtech [66]0.821Explainable Clinical Coding [31]0.633Explainable Clinical Coding [31]0.849For comparability, the CodiEsp-D and CodiEsp-P predictions have been combined. The best performance for each metric and evaluation is highlighted in bold Comparison of explainability results for the best system and the three best state-of-the-art systems for the CodiEsp and cantemist corpora For comparability, the CodiEsp-D and CodiEsp-P predictions have been combined. The best performance for each metric and evaluation is highlighted in bold The WNER MF1 system achieves the highest F1 score for CodiEsp-X, obtaining an increase of 8.06%, while the system proposed in [31] obtains the best performance on Cantemist. Their result was achieved by pre-training three Transformer models on clinical data and fine-tuning them using a hierarchical approach that handled medical entity recognition and normalization. Error analysis To identify the primary types of errors made by the system and detect potential areas for improvement, an in-depth evaluation was conducted on the misclassified medical reports from the test set. The following analysis highlights the main issues observed and suggests possible ways to address them. Low-frequency codes: As stated in previous sections, clinical coding is an extreme multilabel classification problem, not only because it involves an extremely large set of possible codes, but also because these codes tend to occur at very low frequencies. For instance, out of the 2,557 codes that comprise CodiEsp-D, 1,162 (45.44%) appear only once across the entire corpus, while MDACE-D Inpatient contains 468 codes (51.77%) that occur only once among its 904 total codes. Examining the model’s performance, we can observe a tendency to make more mistakes when classifying entities whose codes are infrequent. The main reason is that the supervised model in Phase 2 has encountered very few training examples for these rare codes and thus cannot generalize well. Similarly, when building the knowledge base in Phase 3, there was likely insufficient information to assign accurate and significant keyphrases to such infrequent codes, as also highlighted in [30], and therefore, the unsupervised similarity model is more prone to misclassify those entities.This type of error could potentially be addressed by gathering additional annotated data to train the supervised model and enrich the knowledge base with more robust keyphrases. Moreover, data augmentation techniques, such as generating synthetic data, or applying transfer learning from other corpora, could help improve the model’s robustness toward rare codes. Low-frequency codes: As stated in previous sections, clinical coding is an extreme multilabel classification problem, not only because it involves an extremely large set of possible codes, but also because these codes tend to occur at very low frequencies. For instance, out of the 2,557 codes that comprise CodiEsp-D, 1,162 (45.44%) appear only once across the entire corpus, while MDACE-D Inpatient contains 468 codes (51.77%) that occur only once among its 904 total codes. Examining the model’s performance, we can observe a tendency to make more mistakes when classifying entities whose codes are infrequent. The main reason is that the supervised model in Phase 2 has encountered very few training examples for these rare codes and thus cannot generalize well. Similarly, when building the knowledge base in Phase 3, there was likely insufficient information to assign accurate and significant keyphrases to such infrequent codes, as also highlighted in [30], and therefore, the unsupervised similarity model is more prone to misclassify those entities. This type of error could potentially be addressed by gathering additional annotated data to train the supervised model and enrich the knowledge base with more robust keyphrases. Moreover, data augmentation techniques, such as generating synthetic data, or applying transfer learning from other corpora, could help improve the model’s robustness toward rare codes. Discontinuous mentions: As previously mentioned, the NER system GliNER does not support discontinuous mentions, and SimpleNER only does it under specific circumstances. Hence, only the WNER system allows for the complete inclusion of any discontinuity. Among the four corpora, only CodiEsp contains discontinuous entities. Specifically, the test sets of CodiEsp-D and CodiEsp-P include 524 and 430 such entities, respectively, of which WNER correctly detects 30.92% and 25.81%. It is worth noting that WNER is the only system among the three evaluated that supports discontinuous entities. Therefore, although these percentages may appear low, they represent a significant improvement over the other systems. Furthermore, an analysis of the discontinuities identified by the model reveals that it tends to detect discontinuous entities more accurately when their spans are closer to each other. In particular, the predicted entities have an average span separation of 27.20 characters in CodiEsp-D and 111.95 characters in CodiEsp-P, whereas the average separation in the gold standard is 140.76 and 253.02 characters, respectively. This suggests that the greater the separation between entities, the more difficult they are to detect. Additionally, some discontinuous entities in the corpus are divided into three segments throughout the clinical reports, and WNER was unable to detect any of these. However, this limitation affects only a small number of cases, specifically, 14 and 37 entities in CodiEsp-D and CodiEsp-P, respectively.All of these undetected discontinuous entities directly affect the Recall metric, decreasing its value while increasing the number of false negatives. Discontinuous mentions: As previously mentioned, the NER system GliNER does not support discontinuous mentions, and SimpleNER only does it under specific circumstances. Hence, only the WNER system allows for the complete inclusion of any discontinuity. Among the four corpora, only CodiEsp contains discontinuous entities. Specifically, the test sets of CodiEsp-D and CodiEsp-P include 524 and 430 such entities, respectively, of which WNER correctly detects 30.92% and 25.81%. It is worth noting that WNER is the only system among the three evaluated that supports discontinuous entities. Therefore, although these percentages may appear low, they represent a significant improvement over the other systems. Furthermore, an analysis of the discontinuities identified by the model reveals that it tends to detect discontinuous entities more accurately when their spans are closer to each other. In particular, the predicted entities have an average span separation of 27.20 characters in CodiEsp-D and 111.95 characters in CodiEsp-P, whereas the average separation in the gold standard is 140.76 and 253.02 characters, respectively. This suggests that the greater the separation between entities, the more difficult they are to detect. Additionally, some discontinuous entities in the corpus are divided into three segments throughout the clinical reports, and WNER was unable to detect any of these. However, this limitation affects only a small number of cases, specifically, 14 and 37 entities in CodiEsp-D and CodiEsp-P, respectively. All of these undetected discontinuous entities directly affect the Recall metric, decreasing its value while increasing the number of false negatives. Inconsistency in procedure annotation: CodiEsp-P is the only corpus in this study that considers medical procedures (ICD-10-PCS). An examination of its annotations revealed a primary inconsistency that makes it a particularly challenging task. Although the corpus contains a total of 870 unique codes, 229 (26.32%) of them consist of only four characters, as annotators considered the available contextual information insufficient to assign the full seven-character code. As shown in Figure 1, the structure of ICD-10-PCS codes is multiaxial, and, according to the manual of the version used in CodiEsp12, the character Z indicates None or Not Applicable, meaning that it is used when the procedure does not require further specification. However, there exist 20 codes in CodiEsp-P that sometimes appear in their 4-character form and other times as full 7-character codes ending in ZZZ, without any reasonable explanation. Hence, this inconsistency represents a major issue for detecting those codes. For instance, the entity “ecografía urinaria” (“urinary ultrasound”) is sometimes labeled with the code BT4J and other times assigned the code BT4JZZZ, even though both indicate the same procedure. These 20 codes account for 10.07% of the total instances in the test set and are particularly confusing for both the supervised model in Phase 2 and the unsupervised similarity model in Phase 3, as it is unclear which version of the code should be selected in each case because both represent the same medical procedure.This issue directly affects the Precision and MAP metrics, as the models may predict the code with four characters instead of the other one, or vice versa. It also decreases the Recall by introducing false negatives when the correct code is not predicted. No straightforward solution exists for this problem, other than unifying the annotations by using only one of the possible codes in each case either in training or evaluation. Inconsistency in procedure annotation: CodiEsp-P is the only corpus in this study that considers medical procedures (ICD-10-PCS). An examination of its annotations revealed a primary inconsistency that makes it a particularly challenging task. Although the corpus contains a total of 870 unique codes, 229 (26.32%) of them consist of only four characters, as annotators considered the available contextual information insufficient to assign the full seven-character code. As shown in Figure 1, the structure of ICD-10-PCS codes is multiaxial, and, according to the manual of the version used in CodiEsp12, the character Z indicates None or Not Applicable, meaning that it is used when the procedure does not require further specification. However, there exist 20 codes in CodiEsp-P that sometimes appear in their 4-character form and other times as full 7-character codes ending in ZZZ, without any reasonable explanation. Hence, this inconsistency represents a major issue for detecting those codes. For instance, the entity “ecografía urinaria” (“urinary ultrasound”) is sometimes labeled with the code BT4J and other times assigned the code BT4JZZZ, even though both indicate the same procedure. These 20 codes account for 10.07% of the total instances in the test set and are particularly confusing for both the supervised model in Phase 2 and the unsupervised similarity model in Phase 3, as it is unclear which version of the code should be selected in each case because both represent the same medical procedure. This issue directly affects the Precision and MAP metrics, as the models may predict the code with four characters instead of the other one, or vice versa. It also decreases the Recall by introducing false negatives when the correct code is not predicted. No straightforward solution exists for this problem, other than unifying the annotations by using only one of the possible codes in each case either in training or evaluation. Unseen codes during training: All test sets of the four corpora contain a certain number of codes that do not appear in either the training or the development sets. For example, CodiEsp-P includes 143 unseen codes (16.44%) among its 870 total codes, while MDACE-D Profee contains 97 unseen codes (14.41%) out of 673. In contrast, the proportion is the lowest in FALP Morphology and FALP Topography, with 11 (4.87%) and 4 (2.61%) unseen codes, respectively. Although the main purpose of Phase 3 is to address these specific cases of unseen code occurrences, there remains considerable room for improvement, as a common type of prediction error involves missing some of these codes. One main reason for this is that, since these codes do not appear during training, their keyphrases are unavailable, unless they are found in the other corpora. In some cases, even their descriptions are missing, so the unsupervised model lacks sufficient information to classify them confidently and accurately.One possible solution for this type of error is to improve the unsupervised classification by using more sophisticated techniques. Moreover, enriching the knowledge base with additional keyphrases and descriptions for unseen codes would also be beneficial. Unseen codes during training: All test sets of the four corpora contain a certain number of codes that do not appear in either the training or the development sets. For example, CodiEsp-P includes 143 unseen codes (16.44%) among its 870 total codes, while MDACE-D Profee contains 97 unseen codes (14.41%) out of 673. In contrast, the proportion is the lowest in FALP Morphology and FALP Topography, with 11 (4.87%) and 4 (2.61%) unseen codes, respectively. Although the main purpose of Phase 3 is to address these specific cases of unseen code occurrences, there remains considerable room for improvement, as a common type of prediction error involves missing some of these codes. One main reason for this is that, since these codes do not appear during training, their keyphrases are unavailable, unless they are found in the other corpora. In some cases, even their descriptions are missing, so the unsupervised model lacks sufficient information to classify them confidently and accurately. One possible solution for this type of error is to improve the unsupervised classification by using more sophisticated techniques. Moreover, enriching the knowledge base with additional keyphrases and descriptions for unseen codes would also be beneficial. Mismatch between predicted entities and gold mentions: Some of the errors in the explainability task evaluation arise from mismatches of one or a few words in the identified entities. For instance, in the Cantemist corpus, the model predicts “sarcoma de Ewing extraesquelético” (“extraskeletal Ewing sarcoma”), whereas the gold standard mention is “sarcoma de Ewing” (“Ewing sarcoma”). The predicted code for the entity (9260/3) is right, which means that in the clinical coding task it is considered correct, while in the explainability task it counts as a mistake since the text span does not match.This type of error negatively affects both Precision and Recall, as it is counted as a false positive (the predicted span is considered incorrect) and a false negative (the gold standard span is considered missed). In the evaluation of the CodiEsp corpus, a tolerance of 10 characters is allowed regarding the entities’ spans. However, in the case of the Cantemist corpus, such tolerance was not applied, in order to ensure full comparability with the state of the art. Hence, one possible way to mitigate this issue would be to introduce flexible matching criteria that allow minor variations in entities. Mismatch between predicted entities and gold mentions: Some of the errors in the explainability task evaluation arise from mismatches of one or a few words in the identified entities. For instance, in the Cantemist corpus, the model predicts “sarcoma de Ewing extraesquelético” (“extraskeletal Ewing sarcoma”), whereas the gold standard mention is “sarcoma de Ewing” (“Ewing sarcoma”). The predicted code for the entity (9260/3) is right, which means that in the clinical coding task it is considered correct, while in the explainability task it counts as a mistake since the text span does not match. This type of error negatively affects both Precision and Recall, as it is counted as a false positive (the predicted span is considered incorrect) and a false negative (the gold standard span is considered missed). In the evaluation of the CodiEsp corpus, a tolerance of 10 characters is allowed regarding the entities’ spans. However, in the case of the Cantemist corpus, such tolerance was not applied, in order to ensure full comparability with the state of the art. Hence, one possible way to mitigate this issue would be to introduce flexible matching criteria that allow minor variations in entities. Multiple labeling of some entities: The annotations of the CodiEsp corpus include instances whose text span positions match exactly, but are assigned two different codes. For example, in CodiEsp-D, the entity “reacción alérgica al contraste con manifestaciones cutáneas” (“allergic reaction to contrast with skin manifestations”) is annotated with two completely different ICD-10-CM codes: T50.8X5 and L27.0. Another example, in CodiEsp-P, is the entity “hepática quimioembolización” (“hepatic chemoembolisation”), which is assigned the ICD-10-PCS codes 04L33DZ and 3E05305. The percentage of test set instances affected by this issue is 0.98% in CodiEsp-D and 4.41% in CodiEsp-P.This problem cannot be addressed using the pipeline proposed in this work, as in Phases 2 and 3 only one code is assigned to each entity. This may lead to a decrease in Recall, since for entities associated with multiple codes, only one is predicted. A possible solution would be that the classification models consider the possible existence of multilabel entities. However, this must be done with caution, as the phenomenon does not occur very frequently and it could result in many false positives. Multiple labeling of some entities: The annotations of the CodiEsp corpus include instances whose text span positions match exactly, but are assigned two different codes. For example, in CodiEsp-D, the entity “reacción alérgica al contraste con manifestaciones cutáneas” (“allergic reaction to contrast with skin manifestations”) is annotated with two completely different ICD-10-CM codes: T50.8X5 and L27.0. Another example, in CodiEsp-P, is the entity “hepática quimioembolización” (“hepatic chemoembolisation”), which is assigned the ICD-10-PCS codes 04L33DZ and 3E05305. The percentage of test set instances affected by this issue is 0.98% in CodiEsp-D and 4.41% in CodiEsp-P. This problem cannot be addressed using the pipeline proposed in this work, as in Phases 2 and 3 only one code is assigned to each entity. This may lead to a decrease in Recall, since for entities associated with multiple codes, only one is predicted. A possible solution would be that the classification models consider the possible existence of multilabel entities. However, this must be done with caution, as the phenomenon does not occur very frequently and it could result in many false positives. Conclusions and future work This paper presents a three-stage explainable system designed to address the clinical coding task on medical reports annotated with the International Classification of Diseases (ICD) coding system. The proposed approach stands out for its interpretability, as it provides the text spans that justify each coding decision. To evaluate the robustness of the system, four corpora, organized into seven subcorpora, were collected. They comprise medical reports written in Spanish and English and include three different ICD variants: diagnoses (ICD-10-CM), medical procedures (ICD-10-PCS), and oncological (ICD-O-3) codes. These datasets are, as far as we know, the only publicly available English and Spanish corpora that provide annotations for both ICD codes and their corresponding textual mentions, that is, they are annotated for the task of explainable clinical coding. In addition, three NER-based approaches were evaluated, each built with a different architecture and characterized by distinct features and levels of complexity, allowing us to evaluate their suitability depending on the characteristics of each corpus. As far as we know, this study constitutes the most extensive analysis so far of a system addressing the task of explainable clinical coding by being evaluated across four different tasks, represented by four distinct corpora in two languages, and covering three ICD variants. Overall, the system performs well in many scenarios, demonstrating robustness and generalization, while highlighting challenges in more difficult settings. Furthermore, the system explicitly incorporates explainability, ensuring that each coding decision is interpretable by clinicians and researchers. Additionally, as previously mentioned in Section “Introduction”, two main limitations are commonly found in explainable clinical coding systems: (1) unseen codes are typically not considered, and (2) discontinuous and/or overlapping entities are often ignored. The system proposed in this work addresses both issues, allowing models to detect and handle such cases effectively. The experiments show that the system performs well overall, with balanced and generally satisfactory F1 values across corpora. As noted earlier, threshold optimization ( and ) introduces a trade-off between optimizing for F1 and optimizing for MAP. The best overall Precision and F1 scores were achieved by the WNER model using more restrictive thresholds (MF1), while ensemble methods, particularly WNER + GLiNER with less restrictive thresholds (MMAP), obtained the highest Recall and MAP. The experiments have been conducted within the experimental framework and using the same evaluation metrics as those of the reported state-of-the-art results. Among the systems that have outperformed the reported results, the average percentage improvement is 3.42%, with the highest being obtained in the explainability task of CodiEsp (CodiEsp-X, see Table 21), with a percentage improvement of 8.06%. Among the corpora, CodiEsp and MDACE proved to be the most challenging, with MDACE in particular showing lower explainability performance and notable differences between Precision and Recall. Performance remains limited in scenarios involving low-frequency or unseen codes and in cases with long or discontinuous entities. Several potential directions for future work have been discussed throughout the paper. One of the main proposals is to expand the experiments to additional corpora, including the use of synthetic data, since the availability of publicly annotated clinical data, particularly with explainability annotations, is very limited due to privacy and sensitivity concerns. Moreover, a promising research direction is the application of transfer learning to other valuable ICD-10 corpora not used in this study due to the lack of textual annotations, such as the Spanish CARES corpus [22], the CARMEN-I corpus [72] or an extended version of the MIMIC-III corpus [20]. Furthermore, to align the system with current research trends, integrating LLMs poses a promising direction for future work, especially in Phase 3 to handle unseen codes more effectively. Finally, for comparability, this study follows the evaluation setup of state-of-the-art systems, which is the most widely used evaluation for automatic clinical coding. This means that explainability was evaluated strictly in terms of correct code+entity predictions. Future work could benefit from clinician validation to better assess the quality and clinical relevance of all predicted entities, especially those not marked as correct by this study’s evaluation criteria. Acknowledgements Not applicable. Author contribution A.R.A. designed the methodology, developed the software, carried out the experiments, and wrote the manuscript. J.M.R. and A.D. designed the methodology, supervised the methods, experiments and analysis, and reviewed and edited the manuscript. All authors read and approved the final manuscript. Funding This work has been partially supported by the Spanish Ministry of Science and Innovation within the EDHER-MED Project under grant PID2022-136522OB-C21 as well as by the Universidad Nacional de Educación a Distancia (UNED), Spain within project SICAMESP (2023-VICE-0029). Data availability All data used in this study are publicly available. Declarations Ethics approval and consent to participate Not applicable. Consent for publication Not applicable. Competing interests The authors have no competing interests to disclose. Footnotes https://zenodo.org/records/3837305 https://www.sanidad.gob.es/estadEstudios/estadisticas/normalizacion/CIE10/CIE10ES_2018_norm_MANUAL_CODIF_DIAG_.pdf https://www.sanidad.gob.es/estadEstudios/estadisticas/normalizacion/CIE10/CIE10ES_2018_norm_MANUAL_CODIFICACION_PROCEDIMIENTOS_EDICION_2018.pdf https://zenodo.org/records/3978041 https://www.sanidad.gob.es/estadEstudios/estadisticas/normalizacion/CIE10/CIEO-3.1._ACCESIBLE.pdf https://zenodo.org/records/5555432 https://www.sanidad.gob.es/estadEstudios/estadisticas/normalizacion/CIE10/Clasif_Inter_Enfer_CIE_10_rev_3_ed.diag.pdf https://github.com/3mcloud/MDACE https://huggingface.co/models https://github.com/TeMU-BSC/codiesp-evaluation-script https://github.com/TeMU-BSC/cantemist-evaluation-library https://www.sanidad.gob.es/estadEstudios/estadisticas/normalizacion/CIE10/CIE10ES_2018_norm_MANUAL_CODIFICACION_PROCEDIMIENTOS_EDICION_2018.pdf Publisher’s Note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. References Associated Data Data Availability Statement All data used in this study are publicly available.