A pretrained biomedical large language model for Persian biomedical text mining Authors: Baqer M. Merzah, Tania Taami, Salman Asoudeh, Saeed Mirzaee, Amir Reza Hossein Pour, Amir Ali Bengari, Ehsan Karimi Shahmarvandi Published: 2026 ABSTRACT Large language models (LLMs) have recently gained attention in the life sciences due to their capacity to model, extract, and apply complex biological information. Beyond their classical use as chatbots, these systems are increasingly used for complex analysis and problem-solving in specialized fields, including bioinformatics. First, we introduce BIOPARS-BENCH, a dataset from over 10,000 scientific articles, textbooks, and medical websites. BioParsQA was also introduced to evaluate the proposed model, which consists of 5231 Persian medical questions and answers. This study then introduces BioPars, a simple but accurate measure designed to assess LLMs for three main abilities: acquiring subject-specific knowledge, interpreting and synthesizing such knowledge, and demonstrating proper evidence. Comparing ChatGPT, Llama, and Galactica, our study highlights their ability to remember and retrieve learned knowledge but also reveals shortcomings in addressing higher-level, real-world questions and fine-grained inferences. These findings indicate the need for further fine-tuning to address the capabilities of LLM in bioinformatics tasks. To our knowledge, BioPars is the first application of LLM in Persian medical QA, especially for generating long answers. Evaluation of four selected medical QA datasets shows that BioPars has achieved remarkable results compared to comparative approaches. The model on BioParsQA achieved a ROUGE-L score of 29.99, which is an improvement over GPT-4 1.0. The model achieved a BERTScore of 90.87 with the MMR method. The MoverScore and BLEURT values were also higher in this model than the other three models. In addition, the reported scores for the model are MoverScore = 60.43 and BLEURT = 50.78. Introduction Early benchmarks in NLP were usually oriented towards specific, relatively simple tasks. As Large Language Models (LLMs) outgrew the constraints of previous, small-scale architectures, researchers extended their testing scope to encompass progressively sophisticated abilities. Large-scale benchmarks such as MMLU1, BIG-bench2, and C-Eval3 seek to integrate many different NLP challenges in order to provide an overall evaluation of LLMs. Concurrently, various scientific fields have developed their own discipline-specific tests–varying from geoscience4 to radiology5 and ocean science6–to meet certain needs of each field and prove the broader context of application for LLMs. Some recent evaluation frameworks point towards new skills that become more prominent with model size–e.g., reasoning, advanced math1, and code generation7. While benchmarks now challenge how LLMs comprehend and use text information in a broad scope of fields, such as medicine and biology89, they might provide only bounded comprehension of expert activities. This gap in assessment constrains our capacity to compare the domain-specific strengths of the models, and reliance on multiple-choice formats can be missing the broader set of sophisticated capabilities these systems possess. In this paper, we concentrate on examining the higher-level proficiencies of foundational models within bioinformatics. Bioinformatics has been one especially intriguing area for applying Large Language Models (LLMs) to biological research10–12. There has been a variety of research exploring different facets in this intersection, from computer science applications–e.g., medical text processing13, public health data analysis14, GNN benchmarking15–to biological and medical inquiries, e.g., KVPLM16, DNA-Pretrain17, proteome benchmarks18–20, and peptide property prediction21. Despite these attempts, much of the LLM community has concentrated on medical text processing, with comparatively limited channels for generative AI to meaningfully assist specialized bioinformatics research. Nonetheless, the convergence of bioinformatics and LLMs promises significant advancements in scientific discovery and technological innovation. Bioinformatics is a field that combines biology and computational science to gather, examine, and interpret high-level biological data, with a special focus on genomics and molecular biology22. The emergence of large language models (LLMs) has significantly transformed how scientists manage biological data, opening up new avenues for discovery. Accordingly, mounting demand is for a full-coverage benchmark that such models could be evaluated based on how able they are to solve central bioinformatics tasks. Such a standard would enhance the capacity of current LLMs to facilitate sequence analysis, clinical phenotype prediction, and the discovery of biological insights more effectively. In order to fill the knowledge gap between the development of LLM and biomedical text mining-specific requirements of the Persian language, we introduce BioPars–an open but powerful testing platform for thoroughly evaluating state-of-the-art reasoning and domain knowledge. Using this platform, we benchmark popular state-of-the-art LLMs, like ChatGPT, Llama, and Galactica. While these models are excellent at multiple-choice type situations, they perform poorly when it comes to more advanced, professional environments. When asked to respond to very domain-specific questions, their responses turn out to be not of tangible utility. Having seen these results, it’s clear that targeted training on authentic Persian biomedical data is essential in order to set LLMs properly in viable applications, rather than solely utilizing zero-shot or generic text-generating capabilities. Eventually, in addition to serving as a standard benchmark, we see BioPars as a multifunctional, modular platform. For practical research as well as application, this approach allows for a deeper, multidimensional evaluation of large language models’ comprehension and utilization of domain-specific knowledge. It also helps developers by demystifying each model’s areas of strength in the bioinformatics domain, which will ultimately pave the way to more efficient support to researchers in the field. Related work AI methodologies may benefit all patient-care stages, from initial screening to recovery post-surgery23. Recent biomedical research increasingly blends data-centric analytics with interactive, simulation-based methods. An illustrative case is the MATLAB/Simulink experiential module on vestibular-system dynamics developed by24, which lets learners tune physiological parameters and observe real-time balance responses. Neuromorphic-event cameras coupled with a lightweight deep neural network have likewise enabled real-time positional tracking of moving objects in augmented-reality workspaces while sharply reducing latency and power consumption25. Such examples highlight the diverse methodological landscape that BioPars seeks to complement with its Persian-language biomedical LLM layer. In a different data-driven domain, logarithmic-polynomial modelling of quality-rate convex hulls has demonstrated high accuracy, with low RMSE and high \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$R^{2}$$\end{document} values; these findings suggest that the convex hull of one codec can predict the performance of others and thereby streamline adaptive-streaming workflows26. The growing volume of biomedical corpora and domain-specific AI models has revolutionized medical NLP and text mining. However, biomedical text processing is still sparsely researched in Persian, underscoring the importance of establishing pre-trained Persian biomedical language models, with BioPars as an example. Large public datasets such as the eICU Collaborative Research Database (eICU-CRD) have propelled critical-care NLP by providing high-resolution structured clinical data that enable predictive analytics and decision support 27. Likewise, the MedDialog corpus–3.4 M Chinese and 0.26 M English dialogues–has been used to train conversational medical AI, demonstrating how multiturn dialogue corpora boost language-model performance 28. Yet most pre-trained models are built on English or Chinese data, leading to poor domain adaptation for Persian. Moreover, the Multi-Scale Attentive Interaction Networks (MSAIN) framework, which improves Chinese medical QA via biGRUs, CNNs, and attention layers, suggests that language-specific model architectures are fundamental 29. Therefore, dedicated Persian biomedical corpora and models must be developed. A similar lesson emerges from medical computer vision: specialized architectures consistently outperform off-the-shelf models. Evaluation on a diverse dataset of breast-ultrasound (BUS) and mammography images shows that the U-Net model surpasses a pretrained Segment-Anything (SAM) architecture in accurately identifying and segmenting both benign and malignant tumours 30. U-Net remains superior even for irregular shapes, weak boundaries, and high heterogeneity. These findings reinforce the broader principle that domain-tailored solutions–whether for images or text–yield significant performance gains, motivating the design of BioPars as a Persian-specific biomedical LLM. To resolve this disparity, both the evaluation benchmark and fine-tuning of biomedical LLMs demand a critical multilingual and domain-specific approach. On one hand, MultiMedQA provides a robust framework for assessing LLM clinical reasoning, utilizing six medical QA datasets in conjunction with HealthSearchQA to enable structured evaluations of biomedical text comprehension and AI safety31. On the other hand, human evaluations indicate that even efficient models such as Flan-PaLM generate potentially harmful responses in medical question-answering, underscoring the urgent need for risk-mitigation strategies and culturally sensitive training corpora. Additionally, the Indiana Chest X-ray Collection has proven valuable for biomedical retrieval tasks by aligning radiology images with annotated text reports5. With continued exploration in Persian biomedical NLP, datasets such as eICU-CRD, MedDialog, and MultiMedQA will play a pivotal role in developing specialized Persian medical corpora, driving advancements in biomedical text mining, clinical entity recognition, and domain-specific machine translation5,28,29,31 . These facts reinforce the critical demand for Persian-specific biomedical datasets and fine-tuned language models. Thus, BioPars is positioned to mitigate medical text mining challenges in low-resource languages. BioMedGPT32 introduces a multimodal generative pre-trained transformer designed to bridge the language of life (e.g., molecules, proteins) with human natural language. The model, BioMedGPT-10B, integrates molecular graphs (encoded via GraphMVP) and protein sequences (encoded via ESM-2) with natural language by aligning their feature spaces using a fine-tuned biomedical language model (BioMedGPT-LM-7B), derived from Llama2-Chat-7B and trained on 4.2 million biomedical articles from S2ORC. Evaluated on biomedical QA (MedMCQA, PubMedQA, USMLE), molecule QA (ChEBI-20), and protein QA (UniProtQA) tasks, BioMedGPT-10B outperforms general-purpose models like ChatGPT and Llama2, achieving human-expert-level accuracy on PubMedQA and significant improvements in cross-modal tasks through curated datasets (PubChemQA, UniProtQA). The results highlight its potential in accelerating drug discovery and biomedical research. BioinspiredLLM33 presents a conversational large language model (LLM) fine-tuned on 1,034 peer-reviewed articles in biological and bio-inspired materials mechanics. Built on the Orca-2-13b architecture, the model employs Q-A processed datasets to enhance dialogue coherence and integrates retrieval-augmented generation (RAG) to access external sources and trace references. Evaluated on a specialized 100-question exam, BioinspiredLLM achieved 82% accuracy (rising to 90% with chain-of-thought prompting and higher with RAG), outperforming base models in knowledge recall (e.g., mechanical trends in horse hoof hydration) and hypothesis generation (e.g., predicting jackfruit thorn mechanics later validated experimentally). The model also demonstrated utility in collaborative workflows with generative AI tools (e.g., Stable Diffusion) to design bio-inspired materials, highlighting its role in accelerating interdisciplinary research by bridging domain knowledge and enabling creative, data-driven insights. BioMistral34 emerges as an open-source collection of medical language models built on Mistral 7B Instruct, refined through extensive training on PubMed Central’s biomedical research (3 billion tokens). Tested across 10 specialized medical question-answering benchmarks–including clinical knowledge, genetics, and anatomy–the models surpassed leading open-source alternatives like MedAlpaca and PMC-LLaMA, achieving 50.3% accuracy in few-shot evaluations and 57.3% after task-specific tuning, rivaling proprietary tools such as GPT-3.5. Innovations like parameter merging (SLERP/DARE) and memory-efficient quantization boosted adaptability, while multilingual testing (7 languages) revealed strengths and limitations tied to translation quality. While excelling in English medical reasoning, gaps in calibration and cross-lingual consistency highlight avenues for future refinement, positioning BioMistral as a versatile yet evolving resource for healthcare NLP applications. Guo et al.35 tackles the challenge of efficiently extracting detailed chemical reaction information from scientific journals, aiming to replace the slow, labor-intensive manual methods currently in use. The researchers developed a two-step system: first, identifying key reaction products using specialized neural networks (achieving 76% precision), and then extracting supporting elements such as catalysts and solvents with 79% accuracy. By training their models on a small set of annotated examples and leveraging chemistry-specific pretraining, the team was able to navigate the unstructured nature of journal text and capture intricate experimental details. This approach offers a practical solution for processing complex, non-standardized descriptions, making it a valuable tool for speeding up drug development and improving chemical database curation as the volume of scientific literature continues to grow. MatSciBERT36 is a language model specialized for materials science, trained on a corpus of 285 million words from over 150,000 journal articles spanning fields such as ceramics, alloys, and cement. It was developed to overcome the challenges that generic models like SciBERT face with domain-specific terminology. Built on a Transformer architecture, MatSciBERT starts with SciBERT’s weights and then undergoes additional pretraining focused on materials science content. In evaluations, it achieved a Macro-F1 score of 65.92% on the fine-grained NER task (using the SOFC-Slot dataset), which is about 4% better than SciBERT, and set new performance standards. It also excelled in classifying glass-related abstracts with a 93.57% F1 score and showed strong results in relation extraction. The pretrained weights and code are now publicly available to support further developments in materials-specific NLP applications. Shetty et al.37 presents a comprehensive pipeline for automatically extracting material property data from polymer literature, addressing the challenge of mining valuable information from the rapidly growing corpus of materials science publications. The authors curated a dataset of 2.4 million materials science abstracts–of which around 650,000 were polymer-relevant–and manually annotated 750 abstracts using a custom ontology covering eight entity types. They then fine-tuned a BERT-based model, MaterialsBERT (initialized from PubMedBERT), for named entity recognition and relation extraction tasks, achieving superior performance compared to other baseline models on three out of five NER datasets. Using this pipeline, they extracted approximately 300,000 material property records from 130,000 abstracts in just 60 hours with a single GPU, demonstrating its efficacy in accelerating data curation and enabling the discovery of key trends in applications like polymer solar cells, fuel cells, and supercapacitors. Zhao et al.38 introduces ChemDFM, a large language model (LLM) specialized for chemistry, built by fine-tuning LLaMa-13B through domain pre-training (34B tokens from 3.8M papers and 1.4K textbooks) and instruction tuning (2.7M task-specific prompts). Evaluated on tasks like molecule recognition, property prediction, reaction analysis, and text-based molecule design, ChemDFM outperforms major open-source LLMs (e.g., LLaMa-2, Galactica) and even surpasses GPT-4 in many benchmarks despite its smaller size. It demonstrates robust performance in free-form dialogues, assisting with chemical research tasks such as paper comprehension and experimental design, validated through both quantitative metrics (e.g., 73% accuracy in SMILES-to-formula tasks) and qualitative real-world scenario testing. The model’s weights and code are open-sourced to foster further research. Zhang et al.39 introduces ChemLLM, the first large language model (LLM) specifically designed for chemistry, alongside ChemData–a dataset of 7 million instruction-tuning QA pairs derived from structured chemical databases (e.g., PubChem, ChEMBL)–and ChemBench, a benchmark with 4100 multiple-choice questions spanning nine core chemical tasks (e.g., molecular property prediction, retrosynthesis). Built on InternLM2-Base-7B, ChemLLM undergoes two-stage tuning: general capability enhancement followed by domain-specific training on ChemData. Results show ChemLLM matches GPT-4’s performance on chemical tasks, achieves state-of-the-art results on ChemBench, and outperforms similar-sized models (e.g., LLaMA2, Mistral) in general benchmarks (MMLU, C-Eval) and specialized NLP tasks like chemical literature translation, demonstrating robust domain expertise while retaining broad linguistic proficiency.40 presents LlaSMol, a suite of large language models tailored for chemistry using the SMolInstruct dataset–a carefully curated collection that covers 14 chemistry tasks, including name conversion, molecule generation, and retrosynthesis, with more than 3 million samples. The team fine-tuned several base models (Mistral, Llama 2, Code Llama, and Galactica) with LoRA, resulting in models that not only outperform GPT-4 and Claude 3 Opus on key chemistry tasks but also achieve state-of-the-art results on benchmarks like molecule captioning and property prediction. Among these, LlaSMol-Mistral stood out as the top performer, underscoring how domain-specific tuning using a diverse and well-curated dataset like SMolInstruct can produce models that are both highly effective for specialized tasks and versatile enough for broader applications. Chen et al.41 introduces PharmaGPT, a suite of domain-specific large language models (13B and 70B parameters) tailored for bio-pharmaceutical and chemical domains, trained on a diverse corpus of specialized data (patents, research papers, clinical reports, exams like NAPLEX, and multilingual resources). The models, built on LLaMA with extended vocabulary and continued pretraining, underwent instruction fine-tuning and reinforcement learning with human feedback (RLHF) to enhance domain expertise. Evaluations on benchmarks such as the North American Pharmacist Licensure Examination (NAPLEX), Chinese Pharmacist Exams, and biomedical translation tasks demonstrated that PharmaGPT outperforms general-purpose models (e.g., GPT-3.5, GPT-4) in accuracy and precision, achieving scores of 66–76% on NAPLEX and surpassing GPT-4 in medical translation tasks, despite having fewer parameters. Results highlight its efficacy in specialized pharmaceutical and chemical applications. Pollard et al.27 introduces the eICU Collaborative Research Database, a freely accessible, multi-center database containing high-granularity clinical data from over 200,000 ICU admissions across 208 US hospitals (2014–2015), including vital signs, lab results, medications, diagnoses, treatments, and APACHE IV severity scores. Designed to support critical care research, the database facilitates tasks such as developing machine learning algorithms, clinical decision tools, and benchmarking studies. Structured into 26 relational tables (e.g., patient, lab, medication), the dataset is rigorously deidentified and validated for integrity, with data completeness varying by hospital (e.g., vital signs available for >94% of patients). Hosted on PhysioNet, access requires ethical training and a data-use agreement, enabling researchers to explore ICU outcomes, treatment patterns, and predictive modeling while ensuring patient privacy. In conjunction with the development of LLMs, there has been a growing demand for domain-specific evaluation benchmarks, such as those in biomedical text mining contexts. There are a number of general-purpose benchmarks, such as SciEval and Xiezhi, which assess scientific knowledge and interdisciplinary reasoning42,43; Consequently, they don’t address the challenges in biomedical NLP, where domain-specific terminology, abbreviations, and context-sensitive interpretation are crucial. Moreover, as indicated by C-EVAL, Chinese-language models require linguistic adaptation. Thus, it illustrates that non-English contexts are challenging even for advanced models in terms of higher-order reasoning44. This is of particular relevance to the mining of biomedical texts in Persian, since there is a serious lack of both annotated biomedical datasets and pre-trained language models in Persian. Furthermore, AGIEval, evaluating reasoning by human-standardized exams, stresses that though LLMs do great in structured question-answering tasks, they underperform in fields which require deep, domain-specific knowledge and thus reinforce the need for specialized biomedical benchmarks45. As a benchmark that focuses on domain expertise, BIOINFO-BENCH offers a rich evaluation of LLM performance in bioinformatics-related fields including sequence analysis, disease classification, and analysis of biomedical data46. The findings from BIOINFO-BENCH shed light on the fact that even efficient models like ChatGPT and Galactica struggle with real-world biomedical problem-solving, suggesting that general pretraining on extensive corpora is inadequate for specialized biomedical applications. In addition, SCIENCEQA provides a multimodal evaluation framework that underscores the importance of interpretable AI in biomedical NLP by utilizing not only text but also diagrams and domain-specific scientific knowledge47. These benchmarks collectively reveal that biomedical language models call for domain-specific pretraining, fine-tuning on comprehensive medical corpora, and evaluation metrics tailored to sophisticated biomedical reasoning rather than general natural language understanding. Accordingly, the limitations identified by MMLU in tasks requiring mathematical and scientific problem-solving indicate that LLMs still struggle with structured reasoning and are therefore not optimal for biomedical NLP applications that demand clinical decision support and automated diagnosis1. To mitigate these challenges, precise benchmarking systems that employ reasoning-intensive biomedical questions and domain-specific adaptation are essential for biomedical LLMs. The AI2 Reasoning Challenge (ARC) has demonstrated that LLMs perform poorly on inference-based science questions, suggesting that domain knowledge falls short without robust reasoning capabilities48. This insight is particularly relevant to Persian biomedical NLP, where the scarcity of large-scale annotated datasets further complicates model development and evaluation. Therefore, future research should focus on creating biomedical benchmarks for Persian language models, incorporating clinical, pharmaceutical, and genetic data while maintaining interpretability and robustness in text mining applications. BioPars strives to bridge this gap by taking cues from both general and biomedical benchmarks to develop a Persian biomedical LLM that enhances biomedical text mining, supports clinical decision-making, and advances biomedical research in resource-limited contexts. BIOPARS-BENCH This section explains the method and goals of a detailed medical database collected from Persian medical books, journals, and teaching materials. The collection includes many medical subjects, from basic to advanced healthcare fields, and is a useful tool for researchers, doctors, and tech developers creating medical tools in Persian. The dataset is compiled from authoritative Persian medical literature, including textbooks, clinical guidelines, and peer-reviewed articles. 1. Key topics covered includeHematology: Iron Deficiency Anemia, Hemophilia, Hematology, Blood DonationNeurology & Genetics: Parkinson’s Disease Genetics, MigraineInfectious Diseases: COVID-19 Prevention Guidelines, InfluenzaPharmacology: Antibiotics, Diuretic Drugs, Anti-Nausea Medications, Pharmacological PrinciplesDiagnostics & Imaging: Radiology, Interpretation of Laboratory TestsMaternal & Pediatric Health: Breastfeeding and Infant Nutrition, Neonatal CareChronic Conditions: Hypertension Self-Care Education, Autoimmune DisordersSurgical & Clinical Care: Burn Management, Hemorrhoid Treatment, Vaginal InfectionsNursing & General Medicine: Nursing Practices, Comparison of Intravenous (IV) SolutionsThe dataset includes 20+ core categories with granular subtopics, ensuring broad coverage of clinical, preventive, and therapeutic medical knowledge.Fig. 1Statistical analysis of the research data set collected from Tehran University of Medical Sciences Journal between 2019 and 2025. 1. Key topics covered includeHematology: Iron Deficiency Anemia, Hemophilia, Hematology, Blood DonationNeurology & Genetics: Parkinson’s Disease Genetics, MigraineInfectious Diseases: COVID-19 Prevention Guidelines, InfluenzaPharmacology: Antibiotics, Diuretic Drugs, Anti-Nausea Medications, Pharmacological PrinciplesDiagnostics & Imaging: Radiology, Interpretation of Laboratory TestsMaternal & Pediatric Health: Breastfeeding and Infant Nutrition, Neonatal CareChronic Conditions: Hypertension Self-Care Education, Autoimmune DisordersSurgical & Clinical Care: Burn Management, Hemorrhoid Treatment, Vaginal InfectionsNursing & General Medicine: Nursing Practices, Comparison of Intravenous (IV) SolutionsThe dataset includes 20+ core categories with granular subtopics, ensuring broad coverage of clinical, preventive, and therapeutic medical knowledge. Statistical analysis of the research data set collected from Tehran University of Medical Sciences Journal between 2019 and 2025. Scientific articles in various journals were used to collect text data. Initially, articles were collected in the Journal of Medical Sciences of the University of Tehran in 73 intervals between 2018 and 2025. Details of the collected articles are given in Table 1. These articles included specialized areas that were mostly registered as innovations and findings in the field of medicine. For more comprehensiveness, the data was collected from 6600 general articles collected from paziresh24, seebmagazine, Ibn Sina Clinical Medicine Journal, and pezeshkyemrooz. Details of general keywords and the number of words collected from scientific articles are given in Table 1. The statistical characteristics of the collected dataset are illustrated in Fig. 1.Table 1Summary of the number of words collected from different scientific and medical sources used in this study.SourceYear# Interval# Article# wordshttps://tumj.tums.ac.ir/2025170670002024121271701220231212718224320221212119022202112108167881202012124165900201912121171420https://www.paziresh24.com/––21008 Mhttps://seebmagazine.com/––20008.7 Mhttps://sjh.umsha.ac.ir/––200010.1 Mhttps://www.pezeshkyemrooz.com/—–5002 M Table 1Summary of the number of words collected from different scientific and medical sources used in this study.SourceYear# Interval# Article# wordshttps://tumj.tums.ac.ir/2025170670002024121271701220231212718224320221212119022202112108167881202012124165900201912121171420https://www.paziresh24.com/––21008 Mhttps://seebmagazine.com/––20008.7 Mhttps://sjh.umsha.ac.ir/––200010.1 Mhttps://www.pezeshkyemrooz.com/—–5002 M Summary of the number of words collected from different scientific and medical sources used in this study. BioParsQA To evaluate the proposed LLM, we need a dataset related to the context in which it is used. For this purpose, a set of medical questions and answers was collected. Answering online questions usually allows for errors due to the possibility of answers from non-specialists, which can affect the final result and the final decision. On the other hand, providing a wrong answer has irreparable consequences. For example, the drug Mifepristone is usually provided to treat high blood sugar. This drug causes fetal loss in pregnant women. For this purpose, an attempt was made to use questions and answers that at least two specialist doctors approved. 5231 medical questions and answers were collected. Examples of translated Persian medical question–answer pairs from the BioParsQA dataset are presented in Fig. 2 .Fig. 2English translations of sample Persian medical question–answer pairs from the BioParsQA dataset. English translations of sample Persian medical question–answer pairs from the BioParsQA dataset. BioPars BioPars encoder: gated attention guided by moving average Assume that a biomedical token sequence is \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$X={x_1, x_2,...,x_n}\in R^{n*d}$$\end{document}. Following a single encoder block, the representations are \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$Y={y_1,...,y_n}\in R^{n*d}$$\end{document}. Ordinary self-attention is enhanced by BioPars using a damped exponential moving average (EMA), which introduces a significant temporal bias. The outcome is then modulated by lightweight gating. High-dimensional damped EMA An expansion matrix \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\beta \in R^{d*h}$$\end{document} first lifts each scalar component of a token vector to an h-dimensional latent space:1\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} u_t^j=\beta _jx_{t,j} \end{aligned}$$\end{document}Next, an element-wise damped EMA is applied:2\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} h_t^j=\alpha _j \bigodot u_t^j+(1-\alpha _j \bigodot \delta _j)\bigodot h_{t-1}^j \end{aligned}$$\end{document}Using \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\eta \in R^{d*h}$$\end{document}, the hidden state is projected back to the original width:3\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} y_{t,j}=\eta _j^Th_t^j \end{aligned}$$\end{document} The gated attention layer Short-range dependencies are already encoded in the shared context vector provided by the EMA output. This vector is used by BioPars to construct reset- and update-gates, which are then combined with scaled dot-product attention and merged via a residual path to produce a small but expressive block. Restrictions Local context: The cost is decreased from O(n 2) to O(nc) by dividing queries, keys, and values into chunks of length c; however, inter-chunk interactions are weakened.Drift in configuration: A fragmented design space resulted from earlier versions’ requirement for distinct normalization options and gating strategies for various data types.Unknown scalability: Stability in multi-million token and billion-parameter contexts has not yet been tested. Local context: The cost is decreased from O(n 2) to O(nc) by dividing queries, keys, and values into chunks of length c; however, inter-chunk interactions are weakened. Local context: The cost is decreased from O(n 2) to O(nc) by dividing queries, keys, and values into chunks of length c; however, inter-chunk interactions are weakened. Drift in configuration: A fragmented design space resulted from earlier versions’ requirement for distinct normalization options and gating strategies for various data types. Drift in configuration: A fragmented design space resulted from earlier versions’ requirement for distinct normalization options and gating strategies for various data types. Unknown scalability: Stability in multi-million token and billion-parameter contexts has not yet been tested. Unknown scalability: Stability in multi-million token and billion-parameter contexts has not yet been tested. BioPars with large-scale, long-context improvements BioPars implements a number of architectural improvements to get around those problems: Complex-domain EMA, or C-EMA EMA can be extended to the complex plane by considering it a diagonal state-space model. Each actual decay is substituted with a complex coefficient by BioPars:4\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} h_t^j=\alpha _j(cos \theta _j+ i sin \theta _j)\bigodot u_t^j+(1-\alpha _j\bigodot \delta _j)(cos \theta _j+i sin \theta _j)\bigodot h_{t-1}^j \end{aligned}$$\end{document}5\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} y_{t,j}=Re(\eta _j^Th_t^j) \end{aligned}$$\end{document}\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha ,\delta \in R^{d*h}$$\end{document} are the real number parameters same as in EMA. Different from EMA, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\eta \in \mathbb {C}^{d \times h}$$\end{document} in CEMA are complex numbers. \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\theta _j \in \mathbb {R}^h$$\end{document}, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$j \in \{1,2,\ldots ,d\}$$\end{document} are the h arguments. Normalization of timesteps While drift along the sequence axis is ignored, LayerNorm equalizes variance across features. Thus, for small feature groups, BioPars calculates a cumulative mean and variance, producing TimestepNorm, which stabilizes activations without disclosing future tokens–essential for autoregressive pre-training. Normalized attention An l2-normalized shared vector \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{Z}$$\end{document} is the source of queries and keys:6\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} X' = \textrm{CEMA}(X), \quad X' \in \mathbb {R}^{n \times d} \end{aligned}$$\end{document}7\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} Z = X'W_z + b_{z'}, \quad Z' = \frac{Z}{\Vert Z\Vert }, \quad Z' \in \mathbb {R}^{n \times z} \end{aligned}$$\end{document}8\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} Q = k_q \odot Z' + \mu _q, \quad Q \in \mathbb {R}^{n \times z} \end{aligned}$$\end{document}9\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} K = k_k \odot Z' + \mu _k, \quad K \in \mathbb {R}^{n \times z} \end{aligned}$$\end{document}Where Q and K are computed using the normalized shared representation Z' instead of Z. Then, the attention operation is defined as:10\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} O = \textrm{softmax}(QK^T)V, \quad O \in \mathbb {R}^{n \times v} \end{aligned}$$\end{document}After which a softmax is sufficient–no explicit scale or SiLU is required. The outcome is less susceptible to saturation and more stable. Pre-norm two-hop residual BioPars returns the feed-forward branch straight to the block input rather than via a parameter-heavy gated residual:11\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \hat{Y}=Attention(Norm(X))+X \end{aligned}$$\end{document}12\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} Y=FFN(Norm(\hat{Y}))+\hat{Y}=FFN(Norm(\hat{Y}))+Attention(Norm(X))+X \end{aligned}$$\end{document}Removing extra gates while maintaining variance control even at 7 B parameters. Parallelism In addition to data, tensor, and pipeline parallelism, chunk-wise attention opens up a sequence-parallel dimension. Pre-training on Persian biomedical corpora with contexts well beyond typical Transformer limits is made possible by the fact that only the final C-EMA state and TimestepNorm statistics traverse devices, and that these transfers fully overlap with computation. Evaluation metrics Four criteria were used to evaluate the results: BERTScore49, ROUGE50, Mover Score51, and BLEURT52. The following is a general definition of these criteria. ROUGE ROUGE is a set of automatic tools to check summary quality by comparing computer-made summaries with human-written ones. It measures overlap in four main ways50:ROUGE-N: Counts matching word groups (like single words or word pairs)ROUGE-L: Finds longest word sequences in order (even with gaps)ROUGE-W: Rewards consecutive matches more than scattered onesROUGE-S: Checks word pairs with gaps (e.g., “dog cat” matches “dog ran after cat”)All versions balance50:Coverage: How much human content is includedRelevance: How much computer content is usefulROUGE focuses more on coverage (using setting β) and is widely used because it matches human judgments well, while being faster and cheaper than manual checks.ROUGE-N:\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\text {ROUGE-N} = \frac{\sum _{S \in \{\text {Refs}\}} \sum _{\text {gram}_n \in S} \text {Count}_{\text {match}}(\text {gram}_n)}{\sum _{S \in \{\text {Refs}\}} \sum _{\text {gram}_n \in S} \text {Count}(\text {gram}_n)}$$\end{document}ROUGE-L\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} R_{\text {LCS}}&= \frac{\text {LCS}(X,Y)}{m}, \quad P_{\text {LCS}} = \frac{\text {LCS}(X,Y)}{n} \\ \text {ROUGE-L}&= \frac{(1+\beta ^2)R_{\text {LCS}}P_{\text {LCS}}}{R_{\text {LCS}} + \beta ^2 P_{\text {LCS}}} \end{aligned}$$\end{document}ROUGE-W\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} R_{\text {WLCS}}&= f^{-1}\left( \frac{\text {WLCS}(X,Y)}{f(m)}\right) \\ P_{\text {WLCS}}&= f^{-1}\left( \frac{\text {WLCS}(X,Y)}{f(n)}\right) \\ \text {ROUGE-W}&= \frac{(1+\beta ^2)R_{\text {WLCS}}P_{\text {WLCS}}}{R_{\text {WLCS}} + \beta ^2 P_{\text {WLCS}}} \end{aligned}$$\end{document}ROUGE-S\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} R_{\text {SKIP2}}&= \frac{\text {SKIP2}(X,Y)}{\left( {\begin{array}{c}m\\ 2\end{array}}\right) }, \quad P_{\text {SKIP2}} = \frac{\text {SKIP2}(X,Y)}{\left( {\begin{array}{c}n\\ 2\end{array}}\right) } \\ \text {ROUGE-S}&= \frac{(1+\beta ^2)R_{\text {SKIP2}}P_{\text {SKIP2}}}{R_{\text {SKIP2}} + \beta ^2 P_{\text {SKIP2}}} \end{aligned}$$\end{document}ROUGE-SU: \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\text {ROUGE-SU} = \text {ROUGE-S} + \text {Unigram-F1}$$\end{document} ROUGE-N: Counts matching word groups (like single words or word pairs) ROUGE-N: Counts matching word groups (like single words or word pairs) ROUGE-L: Finds longest word sequences in order (even with gaps) ROUGE-L: Finds longest word sequences in order (even with gaps) ROUGE-W: Rewards consecutive matches more than scattered ones ROUGE-W: Rewards consecutive matches more than scattered ones ROUGE-S: Checks word pairs with gaps (e.g., “dog cat” matches “dog ran after cat”) ROUGE-S: Checks word pairs with gaps (e.g., “dog cat” matches “dog ran after cat”) Coverage: How much human content is included Coverage: How much human content is included Relevance: How much computer content is useful Relevance: How much computer content is useful ROUGE-N:\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\text {ROUGE-N} = \frac{\sum _{S \in \{\text {Refs}\}} \sum _{\text {gram}_n \in S} \text {Count}_{\text {match}}(\text {gram}_n)}{\sum _{S \in \{\text {Refs}\}} \sum _{\text {gram}_n \in S} \text {Count}(\text {gram}_n)}$$\end{document} ROUGE-N:\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\text {ROUGE-N} = \frac{\sum _{S \in \{\text {Refs}\}} \sum _{\text {gram}_n \in S} \text {Count}_{\text {match}}(\text {gram}_n)}{\sum _{S \in \{\text {Refs}\}} \sum _{\text {gram}_n \in S} \text {Count}(\text {gram}_n)}$$\end{document} ROUGE-L\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} R_{\text {LCS}}&= \frac{\text {LCS}(X,Y)}{m}, \quad P_{\text {LCS}} = \frac{\text {LCS}(X,Y)}{n} \\ \text {ROUGE-L}&= \frac{(1+\beta ^2)R_{\text {LCS}}P_{\text {LCS}}}{R_{\text {LCS}} + \beta ^2 P_{\text {LCS}}} \end{aligned}$$\end{document} ROUGE-L\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} R_{\text {LCS}}&= \frac{\text {LCS}(X,Y)}{m}, \quad P_{\text {LCS}} = \frac{\text {LCS}(X,Y)}{n} \\ \text {ROUGE-L}&= \frac{(1+\beta ^2)R_{\text {LCS}}P_{\text {LCS}}}{R_{\text {LCS}} + \beta ^2 P_{\text {LCS}}} \end{aligned}$$\end{document} ROUGE-W\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} R_{\text {WLCS}}&= f^{-1}\left( \frac{\text {WLCS}(X,Y)}{f(m)}\right) \\ P_{\text {WLCS}}&= f^{-1}\left( \frac{\text {WLCS}(X,Y)}{f(n)}\right) \\ \text {ROUGE-W}&= \frac{(1+\beta ^2)R_{\text {WLCS}}P_{\text {WLCS}}}{R_{\text {WLCS}} + \beta ^2 P_{\text {WLCS}}} \end{aligned}$$\end{document} ROUGE-W\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} R_{\text {WLCS}}&= f^{-1}\left( \frac{\text {WLCS}(X,Y)}{f(m)}\right) \\ P_{\text {WLCS}}&= f^{-1}\left( \frac{\text {WLCS}(X,Y)}{f(n)}\right) \\ \text {ROUGE-W}&= \frac{(1+\beta ^2)R_{\text {WLCS}}P_{\text {WLCS}}}{R_{\text {WLCS}} + \beta ^2 P_{\text {WLCS}}} \end{aligned}$$\end{document} ROUGE-S\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} R_{\text {SKIP2}}&= \frac{\text {SKIP2}(X,Y)}{\left( {\begin{array}{c}m\\ 2\end{array}}\right) }, \quad P_{\text {SKIP2}} = \frac{\text {SKIP2}(X,Y)}{\left( {\begin{array}{c}n\\ 2\end{array}}\right) } \\ \text {ROUGE-S}&= \frac{(1+\beta ^2)R_{\text {SKIP2}}P_{\text {SKIP2}}}{R_{\text {SKIP2}} + \beta ^2 P_{\text {SKIP2}}} \end{aligned}$$\end{document} ROUGE-S\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} R_{\text {SKIP2}}&= \frac{\text {SKIP2}(X,Y)}{\left( {\begin{array}{c}m\\ 2\end{array}}\right) }, \quad P_{\text {SKIP2}} = \frac{\text {SKIP2}(X,Y)}{\left( {\begin{array}{c}n\\ 2\end{array}}\right) } \\ \text {ROUGE-S}&= \frac{(1+\beta ^2)R_{\text {SKIP2}}P_{\text {SKIP2}}}{R_{\text {SKIP2}} + \beta ^2 P_{\text {SKIP2}}} \end{aligned}$$\end{document} ROUGE-SU: \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\text {ROUGE-SU} = \text {ROUGE-S} + \text {Unigram-F1}$$\end{document} ROUGE-SU: \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\text {ROUGE-SU} = \text {ROUGE-S} + \text {Unigram-F1}$$\end{document} BERTScore BERTScore helps evaluate how good computer-generated text is by comparing it to human-written versions. Instead of just counting matching words, it uses special language processing from systems like BERT to understand meaning connections. The tool gives three main ratings49:Recall (R_BERT): How much important info from human text is keptPrecision (P_BERT): How much computer text is usefulF1 Score (F_BERT): Balanced combination of both scoresUses middle layers of BERT (layer 9) for better meaning matchingAdjusts scores to 0-1 scale using random text comparisons Recall (R_BERT): How much important info from human text is kept Recall (R_BERT): How much important info from human text is kept Precision (P_BERT): How much computer text is useful Precision (P_BERT): How much computer text is useful F1 Score (F_BERT): Balanced combination of both scores F1 Score (F_BERT): Balanced combination of both scores Uses middle layers of BERT (layer 9) for better meaning matching Uses middle layers of BERT (layer 9) for better meaning matching Adjusts scores to 0-1 scale using random text comparisons Adjusts scores to 0-1 scale using random text comparisons Formulas Basic Scores \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} R_{\text {BERT}} = \frac{1}{|x|} \sum _{x_i \in x} \max _{\hat{x}_j \in \hat{x}} x_i^\top \hat{x}_j \end{aligned}$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} P_{\text {BERT}} = \frac{1}{|\hat{x}|} \sum _{\hat{x}_j \in \hat{x}} \max _{x_i \in x} x_i^\top \hat{x}_j \end{aligned}$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} F_{\text {BERT}} = \frac{2 \cdot P_{\text {BERT}} \cdot R_{\text {BERT}}}{P_{\text {BERT}} + R_{\text {BERT}}} \end{aligned}$$\end{document} IDF Weighting (Optional): \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\text {idf}(w) = -\log \left( \frac{1}{M}\sum _{i=1}^M \mathbb {I}[w \in x^{(i)}]\right)$$\end{document}IDF-Weighted Recall \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$R_{\text {BERT}} = \frac{\sum _{x_i \in x} \text {idf}(x_i) \cdot \max _{\hat{x}_j \in \hat{x}} x_i^\top \hat{x}_j}{\sum _{x_i \in x} \text {idf}(x_i)}$$\end{document} Basic Scores \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} R_{\text {BERT}} = \frac{1}{|x|} \sum _{x_i \in x} \max _{\hat{x}_j \in \hat{x}} x_i^\top \hat{x}_j \end{aligned}$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} P_{\text {BERT}} = \frac{1}{|\hat{x}|} \sum _{\hat{x}_j \in \hat{x}} \max _{x_i \in x} x_i^\top \hat{x}_j \end{aligned}$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} F_{\text {BERT}} = \frac{2 \cdot P_{\text {BERT}} \cdot R_{\text {BERT}}}{P_{\text {BERT}} + R_{\text {BERT}}} \end{aligned}$$\end{document} Basic Scores \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} R_{\text {BERT}} = \frac{1}{|x|} \sum _{x_i \in x} \max _{\hat{x}_j \in \hat{x}} x_i^\top \hat{x}_j \end{aligned}$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} P_{\text {BERT}} = \frac{1}{|\hat{x}|} \sum _{\hat{x}_j \in \hat{x}} \max _{x_i \in x} x_i^\top \hat{x}_j \end{aligned}$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} F_{\text {BERT}} = \frac{2 \cdot P_{\text {BERT}} \cdot R_{\text {BERT}}}{P_{\text {BERT}} + R_{\text {BERT}}} \end{aligned}$$\end{document} IDF Weighting (Optional): \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\text {idf}(w) = -\log \left( \frac{1}{M}\sum _{i=1}^M \mathbb {I}[w \in x^{(i)}]\right)$$\end{document} IDF Weighting (Optional): \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\text {idf}(w) = -\log \left( \frac{1}{M}\sum _{i=1}^M \mathbb {I}[w \in x^{(i)}]\right)$$\end{document} IDF-Weighted Recall \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$R_{\text {BERT}} = \frac{\sum _{x_i \in x} \text {idf}(x_i) \cdot \max _{\hat{x}_j \in \hat{x}} x_i^\top \hat{x}_j}{\sum _{x_i \in x} \text {idf}(x_i)}$$\end{document} IDF-Weighted Recall \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$R_{\text {BERT}} = \frac{\sum _{x_i \in x} \text {idf}(x_i) \cdot \max _{\hat{x}_j \in \hat{x}} x_i^\top \hat{x}_j}{\sum _{x_i \in x} \text {idf}(x_i)}$$\end{document} MoverScore metric MoverScore evaluates text generation quality by computing the semantic distance between system outputs (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\boldsymbol{x}$$\end{document}) and reference texts (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\boldsymbol{y}$$\end{document}) using contextualized embeddings and Earth Mover’s Distance (EMD) 51. Unlike n-gram metrics, it:Encodes text using contextual models (BERT/ELMo)Aggregates layer-wise representations via power means or routingUses IDF-weighted n-gram embeddings to emphasize content wordsSolves optimal transport to find minimal alignment cost Encodes text using contextual models (BERT/ELMo) Encodes text using contextual models (BERT/ELMo) Aggregates layer-wise representations via power means or routing Aggregates layer-wise representations via power means or routing Uses IDF-weighted n-gram embeddings to emphasize content words Uses IDF-weighted n-gram embeddings to emphasize content words Solves optimal transport to find minimal alignment cost Solves optimal transport to find minimal alignment cost Formulation For n-gram sequences \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\boldsymbol{x}^n = (x_1^n,\dots ,x_m^n)$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\boldsymbol{y}^n = (y_1^n,\dots ,y_k^n)$$\end{document}:13\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \text {MoverScore}(\boldsymbol{x}^n, \boldsymbol{y}^n) := \min _{\boldsymbol{F} \in \mathbb {R}_+^{m \times k}} \underbrace{\sum _{i=1}^m \sum _{j=1}^k \boldsymbol{C}_{ij} \boldsymbol{F}_{ij}}_{\text {Total Transportation Cost}} \end{aligned}$$\end{document}Subject to:14\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \boldsymbol{F}\textbf{1}&= \boldsymbol{f}_{\boldsymbol{x}^n} \quad \text {(Source marginal)} \end{aligned}$$\end{document}15\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \boldsymbol{F}^\top \textbf{1}&= \boldsymbol{f}_{\boldsymbol{y}^n} \quad \text {(Target marginal)} \end{aligned}$$\end{document}Where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\boldsymbol{C}_{ij} = \Vert E(\boldsymbol{x}_i^n) - E(\boldsymbol{y}_j^n)\Vert _2$$\end{document} is the Euclidean distance between n-gram embeddings, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$E(\boldsymbol{x}_i^n) = \sum _{t=i}^{i+n-1} \text {idf}(x_t) \cdot \phi (\boldsymbol{z}_{t,1:L})$$\end{document} is the IDF-weighted aggregated embedding, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\phi (\cdot )$$\end{document} is the layer aggregation (e.g., power means \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\frac{1}{L}\sum _{l=1}^L \boldsymbol{z}_l^{p}$$\end{document}), and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\boldsymbol{f}_{\boldsymbol{x}^n}[i] = \frac{\sum _{t=i}^{i+n-1}\text {idf}(x_t)}{Z}$$\end{document} is the normalized n-gram weights. Key variations 16\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \text {WMD}_n = {\left\{ \begin{array}{ll} \text {Word-level } (n=1) \\ \text {Bigram-level } (n=2) \\ \text {Sentence-level } (n=\text {max}) \end{array}\right. }, \quad \text {SMD} = \Vert E(\boldsymbol{x}) - E(\boldsymbol{y})\Vert _2 \end{aligned}$$\end{document} BLEURT metric BLEURT is a learned evaluation metric for text generation that leverages BERT embeddings and multi-task pre-training to align with human judgments. It first generates synthetic data by perturbing Wikipedia sentences (via mask-filling, backtranslation, and word dropping) and trains on diverse signals (BLEU, ROUGE, textual entailment, etc.) to capture lexical, syntactic, and semantic deviations. The model is then fine-tuned on human-rated data. BLEURT’s pre-training ensures robustness to domain shifts and sparse supervision52. Formulation The pre-training objective combines multiple supervision signals:17\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \ell _{\text {pre-train}} = \frac{1}{M} \sum _{m=1}^M \sum _{k=1}^K \gamma _k \ell _k(\boldsymbol{\tau }_k^m, \hat{\boldsymbol{\tau }}_k^m) \end{aligned}$$\end{document}Where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\gamma _k$$\end{document} is task-specific weighting parameters, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\ell _k$$\end{document} is loss function (regression \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\ell _2$$\end{document} or cross-entropy), \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\boldsymbol{\tau }_k^m$$\end{document} is supervision signal for task k on example m, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{\boldsymbol{\tau }}_k^m$$\end{document} is model prediction from BERT’s [CLS] embedding, M is Number of synthetic training examples, and K is the number of pre-training tasks. Evaluation metrics in response to ranking methods such as Similarity Ranking (Sim), Answer Extension Ranking (AE), and Maximum Marginal Relevance (MMR) Ranking were also used, along with Re-Ranking (RR), which is on top of Similarity Ranking. Result In order to determine a suitable performance standard, BioPars is compared to three of the more popular large language models that differ in terms of size, licensing, and intended use case.ChatGPT53: ChatGPT is a closed-source LLM lineage that consists of commercial models with proprietary internal architecture and training data that are only available through APIs. Although they are unable to view or change model weights, end users are able to generate text. There will occasionally be truncated or redacted biomedical outputs because responses are processed through proprietary trust-and-safety layers, illustrating the trade-off between transparency and ease of access.Llama 7B54: This 7 billion-parameter version of Meta AIs Llama 2 suite is a lightweight, fully open-source foundation model. Because of its permissive license, researchers can modify or expand the model for specialized domains, which makes it a viable option for laboratory-scale studies. Despite the existence of larger Llama checkpoints (13 B–70 B), the 7 B edition is featured here because it is computationally affordable and falls at the threshold where emergent reasoning first becomes noticeable.Galactica 30B55: Galactica 30B is a 30-billion-parameter LLM that was pre-trained on scientific text by Metas Papers with Code. It is optimized for academic tasks like entity extraction, formal summarization, question answering across disciplines, and structured citation prediction. Galacticas generative fluency occasionally falls short of more general-purpose models, despite its domain specificity, providing an intriguing contrast between linguistic versatility and specialization.Table 2The average statistic is the data in each data set, where Q represents Question and A represents Answer (data details are taken from56.).DatasetSentence (Q)Word (Q)Sentence (A)Word (A)LiveQA1.1514.766.96141.02ExpertQA-Bio1.2621.696.18184.38ExpertQA-Med1.3722.195.96180.55MedQA1.027.363.3871.48BioParsQA1.4225.345.9176.09 ChatGPT53: ChatGPT is a closed-source LLM lineage that consists of commercial models with proprietary internal architecture and training data that are only available through APIs. Although they are unable to view or change model weights, end users are able to generate text. There will occasionally be truncated or redacted biomedical outputs because responses are processed through proprietary trust-and-safety layers, illustrating the trade-off between transparency and ease of access. ChatGPT53: ChatGPT is a closed-source LLM lineage that consists of commercial models with proprietary internal architecture and training data that are only available through APIs. Although they are unable to view or change model weights, end users are able to generate text. There will occasionally be truncated or redacted biomedical outputs because responses are processed through proprietary trust-and-safety layers, illustrating the trade-off between transparency and ease of access. Llama 7B54: This 7 billion-parameter version of Meta AIs Llama 2 suite is a lightweight, fully open-source foundation model. Because of its permissive license, researchers can modify or expand the model for specialized domains, which makes it a viable option for laboratory-scale studies. Despite the existence of larger Llama checkpoints (13 B–70 B), the 7 B edition is featured here because it is computationally affordable and falls at the threshold where emergent reasoning first becomes noticeable. Llama 7B54: This 7 billion-parameter version of Meta AIs Llama 2 suite is a lightweight, fully open-source foundation model. Because of its permissive license, researchers can modify or expand the model for specialized domains, which makes it a viable option for laboratory-scale studies. Despite the existence of larger Llama checkpoints (13 B–70 B), the 7 B edition is featured here because it is computationally affordable and falls at the threshold where emergent reasoning first becomes noticeable. Galactica 30B55: Galactica 30B is a 30-billion-parameter LLM that was pre-trained on scientific text by Metas Papers with Code. It is optimized for academic tasks like entity extraction, formal summarization, question answering across disciplines, and structured citation prediction. Galacticas generative fluency occasionally falls short of more general-purpose models, despite its domain specificity, providing an intriguing contrast between linguistic versatility and specialization. Galactica 30B55: Galactica 30B is a 30-billion-parameter LLM that was pre-trained on scientific text by Metas Papers with Code. It is optimized for academic tasks like entity extraction, formal summarization, question answering across disciplines, and structured citation prediction. Galacticas generative fluency occasionally falls short of more general-purpose models, despite its domain specificity, providing an intriguing contrast between linguistic versatility and specialization. Table 2The average statistic is the data in each data set, where Q represents Question and A represents Answer (data details are taken from56.).DatasetSentence (Q)Word (Q)Sentence (A)Word (A)LiveQA1.1514.766.96141.02ExpertQA-Bio1.2621.696.18184.38ExpertQA-Med1.3722.195.96180.55MedQA1.027.363.3871.48BioParsQA1.4225.345.9176.09 The average statistic is the data in each data set, where Q represents Question and A represents Answer (data details are taken from56.). We also used four other medical QA datasets to evaluate the proposed method and protocol. The details of these datasets are given in Table 2. These datasets include the following: LiveQA 57: Contains health questions submitted to the National Library of Medicine. This dataset includes a training set with 634 QA pairs and a test set with 104 QA pairs used for evaluation.ExpertQA58: A high-quality long-form QA dataset with 2177 questions in 32 fields, with answers validated by experts. Among them, 504 medical questions and 96 biology questions were used for model evaluation. MedicationQA59: Contains 690 consumer-related questions related to drugs with information retrieved from reputable websites and scientific articles.In the evaluation of the LiveQA dataset (see Table 3), the GPT-4 model was able to achieve ROUGE-l = 19.44 using the RR method. It also achieved maximum BertScore = 83.01, MoverScore = 54.11, and BLEURT = 40.55. In evaluating the LLaMA2-13b model on the LiveQA dataset, the results were weaker than those of GPT-4. It achieved maximum ROUGE-L = 18.83, BertScore = 82.79, MoverScore = 53.79, and BLEURT = 40.59 in the RR, RR, Sim, and Sim evaluation methods. The Galactica 30B model achieved better results in some evaluation criteria than the LLaMA2-13b and GPT-4 models. It achieved maximum ROUGE-L = 19.68. It also achieved maximum BERTScore in the three evaluation criteria. MoverScore and BLEURT maximum results were obtained at 83.92, 54.32, and 41.52, respectively. BioPars achieved the best results in the evaluation criteria on the LiveQA data. This model achieved a maximum ROUGE- L = 19.8, which performed about 0.40 better than GPT-4. This model also achieved a maximum BERTScore = 84.92. The maximum MoverScore and BLEURT values were equal to 55.73 and 43.81, respectively. The comparative bar plot of different approaches on the LiveQA is shown in Fig. 3.Table 3ROUGE-L results across all datasets. The best results are in bold and the second-best results are underlined.DatasetMethodGPT-4LLaMA2-13bGalactica-30BMedLLaMA-13bMedGemmaBioParsLiveQAZS18.8017.7317.32––18.98Sim19.3518.5219.68––19.92AE19.2418.4519.43––19.42MMR19.3218.2519.19––19.22RR19.4418.8319.54––19.80ExpertQA-MedZS25.4524.8624.86––26.91Sim27.6126.4026.40––28.36AE27.9826.1526.15––28.72MMR27.7826.4226.43––28.44RR28.0827.4927.97––28.12BioParsQAZS27.0127.3227.6128.1327.9828.95Sim28.2127.5027.4028.4227.1229.84AE28.9927.3427.1328.9227.9029.99MMR28.9127.4627.1128.9328.7729.54RR28.7527.4928.1228.4328.6529.34 LiveQA 57: Contains health questions submitted to the National Library of Medicine. This dataset includes a training set with 634 QA pairs and a test set with 104 QA pairs used for evaluation. LiveQA 57: Contains health questions submitted to the National Library of Medicine. This dataset includes a training set with 634 QA pairs and a test set with 104 QA pairs used for evaluation. ExpertQA58: A high-quality long-form QA dataset with 2177 questions in 32 fields, with answers validated by experts. Among them, 504 medical questions and 96 biology questions were used for model evaluation. ExpertQA58: A high-quality long-form QA dataset with 2177 questions in 32 fields, with answers validated by experts. Among them, 504 medical questions and 96 biology questions were used for model evaluation. MedicationQA59: Contains 690 consumer-related questions related to drugs with information retrieved from reputable websites and scientific articles. MedicationQA59: Contains 690 consumer-related questions related to drugs with information retrieved from reputable websites and scientific articles. Table 3ROUGE-L results across all datasets. The best results are in bold and the second-best results are underlined.DatasetMethodGPT-4LLaMA2-13bGalactica-30BMedLLaMA-13bMedGemmaBioParsLiveQAZS18.8017.7317.32––18.98Sim19.3518.5219.68––19.92AE19.2418.4519.43––19.42MMR19.3218.2519.19––19.22RR19.4418.8319.54––19.80ExpertQA-MedZS25.4524.8624.86––26.91Sim27.6126.4026.40––28.36AE27.9826.1526.15––28.72MMR27.7826.4226.43––28.44RR28.0827.4927.97––28.12BioParsQAZS27.0127.3227.6128.1327.9828.95Sim28.2127.5027.4028.4227.1229.84AE28.9927.3427.1328.9227.9029.99MMR28.9127.4627.1128.9328.7729.54RR28.7527.4928.1228.4328.6529.34 ROUGE-L results across all datasets. The best results are in bold and the second-best results are underlined. Fig. 3Comparative bar plot of different approaches on the LiveQA. Comparative bar plot of different approaches on the LiveQA. The evaluation of the models on the ExpertQA-Bio dataset is also shown in Tables 3, 4, 5, 6 and 7. The GPT-4 model achieved ROUGE-L = 27.20 on this data with the maximum RR evaluation method. Also, the highest BERTScore, MoverScore, and BLEURT values for this model were 85.83, 56.79, and 45.93, which were obtained with the RR, AE, and MMR methods, respectively. The LLaMA2-13b model presented weaker results than GPT-4. This model achieved ROUGE-L = 25.79, BERTScore = 85.18, MoverScore = 56.17, and BLEURT = 45.20 in the RR evaluation method, the highest values reported for this model. Galactica 30B also achieved results close to LLaMA2-13b in some evaluation methods. This model recorded 27.32, 85.65, 56.73, and 45.87 values in the evaluation metrics. BioPars obtained a maximum ROUGE- L = 25.82, which performed worse than the GPT-4 model in this metric. This model also achieved a maximum BERTScore of 85.12, which performed worse than Galactica 30B in other evaluation metrics. The comparative bar plot of different approaches on the ExpertQA-Bio is shown in Fig. 4.Table 4BERTScore results across all datasets. The best results are in bold.DatasetMethodGPT-4LLaMA2-13bGalactica-30BMedLLaMA-13bMedGemmaBioParsLiveQAZS82.5081.9381.52––84.92Sim83.0182.7882.32––84.23AE82.9582.6081.93––84.53MMR82.9182.7083.92––84.72RR82.9482.7981.54––84.00ExpertQA-MedZS85.1184.8985.23––87.92Sim86.1085.5084.22––86.71AE86.1285.3685.12––86.23MMR86.2285.5785.73––87.91RR86.3085.8084.55––87.26BioParsQAZS86.1388.1286.2787.9887.8288.45Sim87.1588.5286.8287.5987.8789.91AE87.1388.3186.6986.4486.4089.23MMR87.2188.1986.3287.0887.1890.87RR87.3288.0186.0187.1087.4389.12 Table 4BERTScore results across all datasets. The best results are in bold.DatasetMethodGPT-4LLaMA2-13bGalactica-30BMedLLaMA-13bMedGemmaBioParsLiveQAZS82.5081.9381.52––84.92Sim83.0182.7882.32––84.23AE82.9582.6081.93––84.53MMR82.9182.7083.92––84.72RR82.9482.7981.54––84.00ExpertQA-MedZS85.1184.8985.23––87.92Sim86.1085.5084.22––86.71AE86.1285.3685.12––86.23MMR86.2285.5785.73––87.91RR86.3085.8084.55––87.26BioParsQAZS86.1388.1286.2787.9887.8288.45Sim87.1588.5286.8287.5987.8789.91AE87.1388.3186.6986.4486.4089.23MMR87.2188.1986.3287.0887.1890.87RR87.3288.0186.0187.1087.4389.12 BERTScore results across all datasets. The best results are in bold. Fig. 4comparative bar plot of different approaches on the ExpertQA-Bio. comparative bar plot of different approaches on the ExpertQA-Bio. The evaluation results of the proposed approach and the comparative approaches on the ExpertQA-Med dataset are given in Tables 3, 4, 5, 6 and 7. The GPT-4 approach achieved successful results on this dataset compared to the two LLaMA2-13b and Galactica 30B models. This approach achieved a maximum ROUGE-L = 28.08, while the two LLaMA2-13b and Galactica 30B models achieved maximum ROUGE-L of 26.49 and 27.97, respectively. In terms of BERTScore, the GPT-4 model also recorded better results. The GPT-4 model also achieved a maximum MoverScore = 57.32 and BLEURT = 47.0. The BioPars model also achieved high results on this dataset. This model achieved ROUGE-L = 28.72 using the AE method, the highest value reported in all the tested models. The model also achieved a BERTScore of 87.92, a 1.62 improvement over the GPT-4 model. In the two evaluation metrics, MoverScore and BLEURT, the BioPars model achieved 58.54 and 47.95 at maximum, respectively. The comparative bar plot of different approaches on the ExpertQA-Med is shown in Fig. 5.Table 5MoverScore results across all datasets. The best results are in bold.DatasetMethodGPT-4LLaMA2-13bGalactica-30BMedLLaMA-13bMedGemmaBioParsLiveQAZS54.0253.3753.11−−55.35Sim54.0853.7952.22−−55.73AE54.0453.7054.04−−55.23MMR54.0353.6753.23−−55.63RR54.1153.7254.32−−55.71ExpertQA−MedZS56.5055.7455.72−−58.54Sim57.1356.2355.12−−58.34AE57.2556.1755.17−−58.27MMR57.2856.2455.24−−58.26RR57.3256.5855.51−−58.38BioParsQAZS57.2356.7856.0158.6758.0159.23Sim58.1256.9056.7459.4359.1160.43AE58.2257.1156.8559.5259.1360.23MMR58.2157.2256.9459.3259.1460.43RR58.3257.5156.6559.0759.1259.01 Table 5MoverScore results across all datasets. The best results are in bold.DatasetMethodGPT-4LLaMA2-13bGalactica-30BMedLLaMA-13bMedGemmaBioParsLiveQAZS54.0253.3753.11−−55.35Sim54.0853.7952.22−−55.73AE54.0453.7054.04−−55.23MMR54.0353.6753.23−−55.63RR54.1153.7254.32−−55.71ExpertQA−MedZS56.5055.7455.72−−58.54Sim57.1356.2355.12−−58.34AE57.2556.1755.17−−58.27MMR57.2856.2455.24−−58.26RR57.3256.5855.51−−58.38BioParsQAZS57.2356.7856.0158.6758.0159.23Sim58.1256.9056.7459.4359.1160.43AE58.2257.1156.8559.5259.1360.23MMR58.2157.2256.9459.3259.1460.43RR58.3257.5156.6559.0759.1259.01 MoverScore results across all datasets. The best results are in bold. Fig. 5Comparative bar plot of different approaches on the ExpertQA-Med. Comparative bar plot of different approaches on the ExpertQA-Med. The results of the different approaches and the proposed approach on the MedicationQA dataset are shown in Tables 3, 4, 5, 6 and 7. The GPT-4 and LLaMA2-13b models achieved maximum ROUGE-L scores of 16.19 and 14.71, respectively. These two models also achieved maximum BERTScore scores of 83.59 and 82.79. The GPT-4 model performed better than LLaMA2-13b in the MoverScore evaluation criterion in the maximum case and worse than LLaMA2-13b in the BLEURT evaluation criterion in the maximum case. Galactica 30B performed worse than the other models in this dataset. This model achieved 15.73, 81.78, and 38.43 in the evaluation criteria. The BioPars model performed best among the tested models in this dataset. This model achieved ROUGE-L = 18.14, which was a 1.19 improvement over GPT-4. This model achieved a maximum ERTScore = 83.90 using the RR method. 54.32 and 39.87 were recorded in the maximum mode for MoverScore and BLEURT. The comparative bar plot of different approaches on the MedicationQA is shown in Fig. 6.Table 6BLEURT results across all datasets. The best results are in bold.DatasetMethodGPT-4LLaMA2-13bGalactica-30BMedLLaMA-13bMedGemmaBioParsLiveQAZS39.8440.4539.82––42.42Sim40.4740.5941.45––42.21AE40.1539.8041.12––43.81MMR40.5540.2241.52––42.24RR40.5039.5941.33––41.58ExpertQA-MedZS45.9846.3246.36––46.12Sim46.4746.1546.45−–47.12AE46.8046.0246.33––47.95MMR46.8446.4146.34––47.44RR47.0046.4746.41––47.43BioParsQAZS47.1247.7246.3247.0047.0048.11Sim48.3347.7146.0047.3447.3449.42AE48.8247.7246.4447.8647.8649.34MMR48.8047.8246.5447.8947.8950.78RR48.0947.8046.5647.0247.0249.66 Table 6BLEURT results across all datasets. The best results are in bold.DatasetMethodGPT-4LLaMA2-13bGalactica-30BMedLLaMA-13bMedGemmaBioParsLiveQAZS39.8440.4539.82––42.42Sim40.4740.5941.45––42.21AE40.1539.8041.12––43.81MMR40.5540.2241.52––42.24RR40.5039.5941.33––41.58ExpertQA-MedZS45.9846.3246.36––46.12Sim46.4746.1546.45−–47.12AE46.8046.0246.33––47.95MMR46.8446.4146.34––47.44RR47.0046.4746.41––47.43BioParsQAZS47.1247.7246.3247.0047.0048.11Sim48.3347.7146.0047.3447.3449.42AE48.8247.7246.4447.8647.8649.34MMR48.8047.8246.5447.8947.8950.78RR48.0947.8046.5647.0247.0249.66 BLEURT results across all datasets. The best results are in bold. Fig. 6comparative bar plot of different approaches on the MedicationQA. comparative bar plot of different approaches on the MedicationQA. The evaluation results of the different approaches on the BioParsQA dataset are given in Tables 3, 4, 5, 6 and 7. Initially, the GPT-4 approach achieved ROUGE-L = 28.99 using the AE evaluation method. This approach also achieved maximum BERTScore = 87.32, MoverScore = 58.32, and BLEURT = 48.82 using the RR, RR, and AE evaluation methods. The LLaMA2-13b approach performed worse than GPT-4 in the ROUGE-L evaluation criterion. This approach was able to achieve maximum ROUGE-L = 88.52 using the Sim method. The BERTScore values in this approach are much higher than GPT-4, and all methods achieved BERTScore above 0.88. This approach’s maximum MoverScore and BLEURT were 57.51 and 47.82, respectively. The maximum ROUGE-L in the Galactica 30B approach is 28.12. This approach also achieved maximum BERTscore, MoverScore, and BLEURT of 86.82, 56.94, and 46.56 on the BioParsQA dataset. The BioPars approach outperformed the other three models on BioParsQA. The model achieved a ROUGE-L score of 29.99, an improvement over GPT-4 1.0. The model achieved BERTscore = 90.87 with the MMR method. The MoverScore and BLEURT values were also higher than the other three models. The model achieved MoverScore = 60.43 and BLEURT = 50.78. The comparative bar plot of different approaches on the BioParsQA is shown in Fig. 7.Fig. 7The comparative bar plot of different approaches on the BioParsQA. The comparative bar plot of different approaches on the BioParsQA. In the following section, the results of the comparative approaches MedLLaMA-13b60, MedGemma61 and BioPars are presented in Table 7. The MedLLaMA-13b approach was able to achieve ROUGE-L = 28.93 using the MMR evaluation method. This approach also achieved a BERTScore of 87.98. The maximum MoverScore and BLEURT in this approach were 59.52 and 47.89, respectively. The MedGemma method was also able to achieve ROUGE-L = 28.77 using the MMR evaluation method. This approach provided weaker comparative results than MedGemma. The MedGemma approach was able to achieve a maximum BERTScore of 87.87 in Sim. Also, the MoverScore value varied between the range [58.01-59.14]. The proposed BioPars approach outperforms the two approaches MedLLaMA-13b and MedGemma in all evaluation criteria. In the ROUGE-L benchmark, the BioPars approach was able to provide 1.06 improvement over the MedLLaMA-13b approach. This approach also achieved 1.22 improvement over MedGemma. In other comparative benchmarks including BERTScore, MoverScore and BLEURT, the BioPars approach has provided better results.Table 7Automatic evaluation scores on BioParsQA: we compare MedLLaMA-13b, MedGemma, and BioPars. The superior scores among the same models are highlighted in bold.DatasetMethodsMedLLaMA-13b60MedGemma61BioParsROUGE-LBERTScoreMoverScoreBLEURTROUGE- LBERTScoreMoverScoreBLEURTROUGE- LBERTScoreMoverScoreBLEURTBioParsQAZS28.1387.9858.6747.0027.9887.8258.0147.0028.9588.4559.2348.11Sim28.4287.5959.4347.3427.1287.8759.1147.3429.8489.9160.4349.42AE28.9286.4459.5247.8627.9086.4059.1347.8629.9989.2360.2349.34MMR28.9387.0859.3247.8928.7787.1859.1447.8929.5490.8760.4350.78RR28.4387.1059.0747.0228.6587.4359.1247.0229.3489.1259.0149.66 Table 7Automatic evaluation scores on BioParsQA: we compare MedLLaMA-13b, MedGemma, and BioPars. The superior scores among the same models are highlighted in bold.DatasetMethodsMedLLaMA-13b60MedGemma61BioParsROUGE-LBERTScoreMoverScoreBLEURTROUGE- LBERTScoreMoverScoreBLEURTROUGE- LBERTScoreMoverScoreBLEURTBioParsQAZS28.1387.9858.6747.0027.9887.8258.0147.0028.9588.4559.2348.11Sim28.4287.5959.4347.3427.1287.8759.1147.3429.8489.9160.4349.42AE28.9286.4459.5247.8627.9086.4059.1347.8629.9989.2360.2349.34MMR28.9387.0859.3247.8928.7787.1859.1447.8929.5490.8760.4350.78RR28.4387.1059.0747.0228.6587.4359.1247.0229.3489.1259.0149.66 Automatic evaluation scores on BioParsQA: we compare MedLLaMA-13b, MedGemma, and BioPars. The superior scores among the same models are highlighted in bold. Table 8 shows the results of applying the BioPars approach to the relation extraction task. For this purpose, two datasets, ChemProt and BioRed, were used. It was also used to evaluate different approaches such as BioBERT, SciBERT, BioLinkBERT, PubmedBERT, BERT-GT Luo et al and PubMedBERT Luo et al. On the ChemProt dataset, the BioBERT, SciBERT, BioLinkBERT, and PubmedBERT approaches achieved Micro F1 of 79.42, 82.82, 83.54, and 82.36, respectively. Micro F1 values for BERT-GT Luo et al62 and PubMedBERT Luo et al62 are not available. The proposed BioPars approach was able to achieve Micro F = 83.98 on this dataset, which obtained better results than the comparative approaches. Micro F1 values for the BioRed dataset are only available for the two approaches, BERT-GT Luo et al and PubMedBERT Luo et al. These two approaches achieved Micro F1 of 56.5 and 59.8 respectively. The proposed approach achieved Micro F1 = 62.01 on this dataset.Table 8Performance of Relation Extraction(RE) in ChemProt and BioRED.MethodModelChemProtMicro F1BioRedMicro F1Fine-tuningBioBERT6379.42–SciBERT6382.82–BioLinkBERT6383.54–PubmedBERT6382.36–BERT-GT Luo et al62–56.5PubMedBERT Luo et al62–59.8BioPars83.9862.01 Table 8Performance of Relation Extraction(RE) in ChemProt and BioRED.MethodModelChemProtMicro F1BioRedMicro F1Fine-tuningBioBERT6379.42–SciBERT6382.82–BioLinkBERT6383.54–PubmedBERT6382.36–BERT-GT Luo et al62–56.5PubMedBERT Luo et al62–59.8BioPars83.9862.01 Performance of Relation Extraction(RE) in ChemProt and BioRED. Error analysis Despite BioPars’ overall strong performance, a closer inspection of the outputs revealed several systematic sources of parsing errors. These errors mainly arise from the inherent complexity of biomedical language and certain limitations of current transformer-based models. Multi-word and compound terms: Biomedical entities often include long, multi-component phrases (e.g., “nuclear factor kappa-light-chain-enhancer of activated B cells”). Models frequently fail to capture the full syntactic span of these entities, splitting them into unrelated tokens, leading to incorrect dependency links and misclassified relations.Nested and discontinuous modifiers: Sentences containing multiple embedded clauses or parenthetical modifiers pose serious challenges. The models often fail to capture long-distance dependencies, leading to incomplete or flattened parse structures.Abbreviations and acronym ambiguity: Biomedical texts heavily rely on abbreviations (e.g., TNF, IL-6, MAPK), which can refer to different biological entities depending on context. Without explicit abbreviation disambiguation, models tend to attach them incorrectly or skip relations entirely.Domain-specific terminology and rare constructions: Many biomedical expressions and syntactic patterns are underrepresented in the training corpora. As a result, even domain-adapted models like BioBERT and BioLinkBERT struggle with rare dependency patterns, especially in BioRED sentences that combine clinical and molecular contexts.Data sparsity and annotation inconsistency: Some relation categories in ChemProt and BioRED have few labeled examples, which limits the model’s exposure to diverse syntactic forms. Moreover, inconsistent boundary annotations propagate upstream errors into the parser.Overall, these findings indicate that parsing errors are not only due to model limitations but also reflect the intrinsic linguistic and structural complexity of biomedical literature. Although BioPars mitigates some of these issues, especially in handling nested noun phrases and long entities, the high error rates highlight the need for more domain-aware syntactic modeling and richer annotated resources in future work. Multi-word and compound terms: Biomedical entities often include long, multi-component phrases (e.g., “nuclear factor kappa-light-chain-enhancer of activated B cells”). Models frequently fail to capture the full syntactic span of these entities, splitting them into unrelated tokens, leading to incorrect dependency links and misclassified relations. Multi-word and compound terms: Biomedical entities often include long, multi-component phrases (e.g., “nuclear factor kappa-light-chain-enhancer of activated B cells”). Models frequently fail to capture the full syntactic span of these entities, splitting them into unrelated tokens, leading to incorrect dependency links and misclassified relations. Nested and discontinuous modifiers: Sentences containing multiple embedded clauses or parenthetical modifiers pose serious challenges. The models often fail to capture long-distance dependencies, leading to incomplete or flattened parse structures. Nested and discontinuous modifiers: Sentences containing multiple embedded clauses or parenthetical modifiers pose serious challenges. The models often fail to capture long-distance dependencies, leading to incomplete or flattened parse structures. Abbreviations and acronym ambiguity: Biomedical texts heavily rely on abbreviations (e.g., TNF, IL-6, MAPK), which can refer to different biological entities depending on context. Without explicit abbreviation disambiguation, models tend to attach them incorrectly or skip relations entirely. Abbreviations and acronym ambiguity: Biomedical texts heavily rely on abbreviations (e.g., TNF, IL-6, MAPK), which can refer to different biological entities depending on context. Without explicit abbreviation disambiguation, models tend to attach them incorrectly or skip relations entirely. Domain-specific terminology and rare constructions: Many biomedical expressions and syntactic patterns are underrepresented in the training corpora. As a result, even domain-adapted models like BioBERT and BioLinkBERT struggle with rare dependency patterns, especially in BioRED sentences that combine clinical and molecular contexts. Domain-specific terminology and rare constructions: Many biomedical expressions and syntactic patterns are underrepresented in the training corpora. As a result, even domain-adapted models like BioBERT and BioLinkBERT struggle with rare dependency patterns, especially in BioRED sentences that combine clinical and molecular contexts. Data sparsity and annotation inconsistency: Some relation categories in ChemProt and BioRED have few labeled examples, which limits the model’s exposure to diverse syntactic forms. Moreover, inconsistent boundary annotations propagate upstream errors into the parser. Data sparsity and annotation inconsistency: Some relation categories in ChemProt and BioRED have few labeled examples, which limits the model’s exposure to diverse syntactic forms. Moreover, inconsistent boundary annotations propagate upstream errors into the parser. Conclusion and limitations We introduce BioPars, the first comprehensive Persian medical LLM that implements the entire training pipeline from pre-training to final testing, outperforming other open-source medical LLMs. Further experiments highlight the importance of pre-training this model for the Persian medical domain. We also collected a large set of Persian medical data to train the LLM model from Persian medical references, online sites, textbooks, and scientific articles. We also created a large-scale Persian medical conversation dataset, BioParsQA. Despite these achievements, we recognize the limitations of the model. BioPars cannot guarantee the accuracy of answers in all its aspects. Given the serious consequences of incorrect data in the medical domain, we strongly suggest that users exercise caution and seek expert advice when dealing with the generated information. On the other hand, any incorrect medical suggestion and decision may lead to serious consequences. In the future, we will focus on increasing data reliability, integrating more real-time data with monitoring, and incorporating non-textual multimodal information for a more comprehensive and accurate medical service. BioPars is primarily a research model and cannot be considered a substitute for professional medical advice. This model serves as a milestone for LLMs in the Persian medical field.