Multilingual Clinical NER for Diseases and Medications Recognition in Cardiology Texts using BERT Embeddings
Manuela Daniela Danu, George Marica, Constantin Suciu, Lucian Mihai Itu, Oladimeji Farri · 2025
Read Multilingual Clinical NER for Diseases and Medications Recognition in Cardiology Texts using BERT Embeddings by Manuela Daniela Danu, George Marica, Constantin Suciu, Lucian Mihai Itu, Oladimeji Farri (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-17437.txt.
Introduction
Hospital case reports hold rich medical detail, but that detail sits locked in free text in many languages. Manuela Daniela Danu, with four co-authors, builds systems that pull out mentions of diseases and medications from cardiology reports written in English, Spanish, and Italian. Their work formed part of a shared evaluation task on multilingual clinical text, where smaller languages have far less training material.
Danu and her co-authors compare several BERT based language models, some trained on one language and some on many, all starting from general text. Their systems beat the average and middle scores of the contest leaderboard on every subtask, for both disease and medication spotting. Spanish disease recognition proved hardest, while medication spotting scored highest across the three languages.
Catalog record
| CREATOR | Manuela Daniela Danu, George Marica, Constantin Suciu, Lucian Mihai Itu, Oladimeji Farri |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 29 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-17437
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Multilingual Clinical NER for Diseases and Medications Recognition in Cardiology Texts using BERT Embeddings Manuela Daniela Danu, George Marica, Constantin Suciu, Lucian Mihai Itu, Oladimeji Farri Abstract The rapidly increasing volume of electronic health record (EHR) data underscores a pressing need to unlock biomedical knowledge from unstructured clinical texts to support advancements in data-driven clinical systems, including patient diagnosis, disease progression monitoring, treatment effects assessment, prediction of future clinical events, etc. While contextualized language models have demonstrated impressive performance improvements for named entity recognition (NER) systems in English corpora, there remains a scarcity of research focused on clinical texts in low-resource languages. To bridge this gap, our study aims to develop multiple deep contextual embedding models to enhance clinical NER in the cardiology domain, as part of the BioASQ MultiCardioNER shared task. We explore the effectiveness of different monolingual and multilingual BERT-based models, trained on general domain text, for extracting disease and medication mentions from clinical case reports written in English, Spanish, and Italian. We achieved an F1-score of 77.88% on Spanish Diseases Recognition (SDR), 92.09% on Spanish Medications Recognition (SMR), 91.74% on English Medications Recognition (EMR), and 88.9% on Italian Medications Recognition (IMR). These results outperform the mean and median F1 scores in the test leaderboard across all subtasks, with the mean/median values being: 69.61%/75.66% for SDR, 81.22%/90.18% for SMR, 89.2%/88.96% for EMR, and 82.8%/87.76% for IMR. [[email protected],] *1 [[email protected],] [[email protected],] [[email protected],] [[email protected],] keywords MultiCardioNER ,BioASQ ,Cardiology ,Named Entity Recognition ,NER ,unstructured data ,BERT ,Multilingual ,English ,Spanish ,Italian † † copyrightyear: 2024 † † copyright: Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). † † venue: CLEF 2024: Conference and Labs of the Evaluation Forum, September 09–12, 2024, Grenoble, France † † titlenote: Disclaimer: The concepts and information presented in this paper are based on research results that are not commercially available. † † address: Advanta, Siemens SRL, 15 Noiembrie Bvd, 500097 Brasov, Romania † † address: Automation and Information Technology, Transilvania University of Brasov, 5 Mihai Viteazul Street, 500174 Brasov, Romania † † address: Digital Technology and Innovation, Siemens Healthineers, 755 College Rd E, 08540 Princeton, NJ, United States † † corresp: Corresponding author. 1 Introduction With the increasing amount of available electronic health record (EHR) data, clinical natural language processing (NLP) tasks have become significantly important for extracting valuable information from unstructured clinical texts Rubel Schneider et al. 2020 . Named Entity Recognition (NER) is a key NLP task used to identify meaningful entities within these texts, such as anatomical structures, diseases and disorders, signs and symptoms, procedures, and medications Rubel Schneider et al. 2020 ; Kundeti et al. 2016 . Consequently, this facilitates various data analysis applications, ranging from predicting future clinical events Jin et al. 2018 to summarization Riccio et al. 2023 and relation extraction between entities (e.g., drug-to-drug interactions Zaikis and Vlahavas 2020 , symptom-disease relationship Abulaish et al. 2019 , patient-procedure association Rink et al. 2011 , etc.) Despite recent advances in deep learning methods for NER Lee et al. 2020 ; Kanakarajan et al. 2021 , extracting structured information from the vast amounts of unstructured and often noisy clinical documents in EHR systems remains challenging due to the highly specialized medical language, which varies considerably across different medical specialties, as well as due to the prevalence of misspellings, abbreviations, and use of synonyms to express clinical concepts Rubel Schneider et al. 2020 . While contextualized language models have recently improved the performance of NER systems for English corpora Lee et al. 2020 ; Alsentzer et al. 2019 ; Li et al. 2019 , there is a notable lack of research focused on clinical texts in low-resource languages. To address this gap, our study aims to develop multiple deep contextual embedding models for English, Spanish, and Italian to enhance clinical NER in the cardiology domain, as part of the MultiCardioNER shared task Lima-López et al. 2024 ; Nentidis et al. 2024a ; Nentidis et al. 2024b . The MultiCardioNER task is part of the twelfth edition of the large-scale biomedical semantic indexing and question answering challenge (BioASQ) Nentidis et al. 2024a ; Nentidis et al. 2024b , a long-standing initiative aiming to advance research by developing methods and tools that leverage the vast amount of online information to meet the needs of biomedical researchers and practitioners. This initiative seeks to provide efficient and rapid access to the continuously expanding resources and knowledge in the biomedical field. MultiCardioNER Lima-López et al. 2024 ; Nentidis et al. 2024a ; Nentidis et al. 2024b is a shared task that aims to automatically identify two key clinical concepts in medical documents pertaining to cardiology, namely diseases and medications. This task focuses on adapting clinical NER systems to effectively work across multiple languages - primarily Spanish, English, and Italian - for two different subtasks: (1) diseases recognition in Spanish cardiology texts, and (2) medications recognition in cardiology texts written in Spanish, English, and Italian. Both subtasks involve reading and analyzing clinical texts to identify the clinical entities mentioned in the text and using the BRAT format to mark the starting and ending positions of these entities. In this paper, we created four different monolingual models: (1) Spanish Diseases Recognition (SDR), (2) Spanish Medications Recognition (SMR), (3) English Medications Recognition (EMR), and (4) Italian Medications Recognition (IMR). Additionally, we developed two multilingual models: one specialized for Spanish Diseases Recognition (Multi-SDR) and another for Medications Recognition across all three targeted languages (Multi-MMR). We applied transfer learning techniques by fine-tuning BERT-based Devlin et al. 2018 contextual embeddings, originally trained on general domain text in each of the three languages, for the biomedical domain to extract diseases and medications from clinical reports. 2 Related Work In clinical and biomedical NER, recent studies have explored various methodologies to enhance performance. A key model in this domain is multilingual BERT (M-BERT) Devlin et al. 2018 , trained on 104 Wikipedia languages, which excels in various tasks without explicit cross-lingual alignment Pires et al. 2019 , outperforming models based on cross-lingual embeddings Wu and Dredze 2019 . Tian et al. 2020 improved biomedical NER by incorporating syntactic information, enhancing recognition of complex entity relationships (ORCID). Pérez-Díez et al. 2021 focused on de-identifying Spanish medical texts via NER and entity randomization, achieving high recall rates on radiology reports and MEDDOCAN Marimon et al. 2020 challenge data. Kanakarajan et al. 2021 developed BioELECTRA, a biomedical text encoder using discriminators, which outperformed several baselines on multiple biomedical NER benchmarks by leveraging ELECTRA’s efficiency and accuracy in text encoding. Kocaman and Talby 2021 developed a scalable NER system for large biomedical datasets, emphasizing real-time processing and high accuracy. Carrino et al. 2022 focused on pre-trained biomedical language models for clinical NLP in Spanish, addressing the need for multilingual capabilities in biomedical NER. Zhang et al. 2022 optimized a bi-encoder for NER using contrastive learning, introducing dynamic thresholding to improve accuracy, especially for nested entities, with significant gains on datasets like ACE Walker and Consortium 2005 and GENIA Ohta et al. 2002 . Khandelwal et al. 2022 used a novel schema with distant supervision to enhance NER accuracy, showing that domain-specific schema can supplement limited annotated data effectively. Hu