Optimizing large language models for ontology-based annotation: a study on gene ontology in biomedical texts Authors: Pratik Devkota, Somya D Mohanty, Prashanti Manda Published: 2025 ABSTRACT AbstractAutomated ontology annotation of scientific literature plays a critical role in knowledge management, particularly in fields like biology and biomedicine, where accurate concept tagging can enhance information retrieval, semantic search, and knowledge integration. Traditional models for ontology annotation, such as Recurrent Neural Networks (RNNs) and Bidirectional Gated Recurrent Units (Bi-GRUs), have been effective but limited in handling complex biomedical terminologies and semantic nuances. This study explores the potential of large language models (LLMs), including MPT-7B, Phi, BiomedLM, and Meditron, for improving ontology annotation, specifically with Gene Ontology (GO) concepts. We fine-tuned these models on the CRAFT dataset, assessing their performance in terms of F1 score, semantic similarity, memory usage, and inference speed. Our results show that while Bi-GRU baselines remain competitive in raw accuracy, LLMs offer complementary strengths. LLMs exhibit qualitatively higher semantic consistency in some cases, particularly when handling complex or multi-word ontology terms. However, these observations are exploratory and not statistically verified across all model types. However, resource requirements for LLMs are notably high, raising considerations about computational efficiency. Techniques like parameter-efficient fine-tuning (PEFT) and advanced prompting were explored to address these challenges, demonstrating potential in reducing computational demands while maintaining performance. Our findings suggest that while LLMs offer advantages in annotation accuracy, practical deployment should balance these benefits with resource costs. This research highlights the need for further optimization and domain-specific training to make LLMs a feasible choice for real-world biomedical ontology annotation tasks.Keywords: large language models, gene ontology, automated annotation, natural language processing Introduction Automatically annotating scientific literature with concepts from a domain ontology is an important task for several fields, especially Biology and biomedical sciences. Automated ontology annotation of literature refers to the process of automatically tagging and linking concepts within academic or literary texts to a predefined ontology [1]. This process can significantly aid in knowledge management, literature review, and data integration across different research domains. The task of assigning semantic labels or concept tags to textual data based on existing ontologies enables the extraction and exchange of structured knowledge from unstructured text. This is critical for various applications such as information retrieval, knowledge graph representation, and semantic search. The growth of biological ontologies and their use for representing knowledge has spurred research into natural language approaches for automatic annotation of scientific literature [2, 3]. In automated ontology annotation, natural language processing (NLP) techniques are used to analyze text and identify relevant terms, phrases, or topics that can be matched to the concepts in the ontology [1]. This can enhance information retrieval, facilitate knowledge organization, and improve the ability to connect related research and ideas. Key steps in automated ontology annotation can include: Text Processing: Preprocessing the literature to clean and prepare the text for analysis. Text Processing: Preprocessing the literature to clean and prepare the text for analysis. Entity Recognition: Identifying important entities or terms within the text. Entity Recognition: Identifying important entities or terms within the text. Ontology Mapping: Matching recognized entities to concepts within the ontology. Ontology Mapping: Matching recognized entities to concepts within the ontology. Annotation: Adding metadata or tags to the text based on the mapping. Annotation: Adding metadata or tags to the text based on the mapping. Validation: Ensuring the accuracy and relevance of the annotations. Validation: Ensuring the accuracy and relevance of the annotations. Traditionally, machine learning algorithms such as Recurrent Neural Networks (RNNs) and Convolutional Neural Networks (CNNs) have been employed for automated ontology annotation of scientific literature [1, 3–7]. Prior work by our team has leveraged the sequential processing capabilities of Bidirectional Gated Recurrent Units (Bi-GRUs) and demonstrated the performance of these architectures on ontology annotation [1–3, 7–9]. A GRU-based neural network architecture for automatically identifying ontology concepts within biological literature, focusing on improving the extraction of Gene Ontology (GO) terms was presented in [2]. This approach leveraged ELMo word embeddings to enhance contextual understanding, achieving strong accuracy metrics such as F1 scores and Jaccard similarity. The model aims to address the challenges posed by complex biomedical texts where concepts are often implied indirectly. Subsequently, we introduced a novel approach for intelligent, ontology-aware annotation in biological literature [3], aimed at improving semantic understanding and concept recognition through structured ontologies like the Gene Ontology (GO). Unlike traditional models, this method incorporates hierarchical and semantic relationships from ontologies directly into training, which helps the model better distinguish related terms and capture context-specific meanings. Boosting the performance of the GRU-based architectures with a post-processing technique that leveraged structured ontologies to improve semantic understanding and annotation accuracy resulted in substantial enhancements to model accuracy [7]. By integrating hierarchical relationships—like those in the Gene Ontology—the model captured nuanced connections between related terms, improving accuracy over traditional models. The boosted framework utilized semantic similarity metrics to enhance context-specific term recognition, addressing challenges in concept variability and indirect references often present in complex biomedical texts, resulting in more accurate, context-sensitive annotations for improved literature mining [7]. In previous work, neural networks such as Bi-GRU have demonstrated competitive performance in ontology annotation with low computational demands, making them effective for many real-world applications. However, these models often rely on explicit term matching and may struggle with indirect concept references or nuanced semantic relationships—common in biomedical literature. Large language models (LLMs) are advanced AI systems designed to comprehend and generate human language by training on large-scale datasets using transformer architectures. These models—ranging from early milestones like GPT, BERT, and PaLM to today’s most advanced systems—predict sequences of tokens to generate contextually relevant outputs. Since the release of GPT-3, LLMs have seen widespread adoption across domains—content creation, coding, biomedical tasks, and more—often requiring minimal fine-tuning. Significant recent advances include OpenAI’s release of GPT-5 [10], which expanded multimodal reasoning capabilities and underpins applications such as ChatGPT and Microsoft Copilot. Anthropic’s Claude 4 family introduced hybrid models with both fast-response and extended “thinking” modes, pushing forward reasoning depth. Google’s Gemini 2.5 Pro [11] further extended context windows to 1 million tokens and enhanced multimodal reasoning through its Deep Think mode. Meanwhile, Meta’s latest Llama-4 [12] emphasized handling of more complex and contentious queries, and xAI’s Grok-3 added dynamic reasoning modes (“Think” and “Big Brain”) for adaptive inference. Despite these breakthroughs, LLMs still face challenges—including hallucinations, misalignment, bias, and significant energy demands—that necessitate continued research into alignment, efficiency, and safe deployment [13, 14]. While LLMs are resource-intensive and not guaranteed to outperform smaller models on standard benchmarks, their pretraining on broad corpora and their strong contextual modeling capabilities suggest they may excel in cases requiring deeper semantic inference, domain adaptation, or generalization with limited supervision. For example, Sung et al. [15] demonstrated the effectiveness of prompt-based methods using T5 and GPT-3 for biomedical entity normalization, treating concept linking as a text-to-text task. Similarly, Ji et al. [16] applied in-context learning with GPT-3 for SNOMED CT concept annotation in clinical notes, showing that LLMs can generalize well even with minimal supervision. In the Bio-Onto benchmark introduced by Wang et al. [17], multiple LLMs were evaluated on GO and DO annotation tasks, highlighting strengths in zero-shot generalization but also exposing limitations in hierarchical consistency. These studies collectively indicate that while LLMs offer promising performance, especially in zero- and few-shot scenarios, their effectiveness can vary based on ontology structure, domain specificity, and input formulation. Our work builds on these efforts by systematically comparing LLMs to prior neural models using fine-tuned and zero-shot setups for gene ontology annotation in scientific literature. The goal of this work is not only to evaluate whether LLMs outperform traditional models, but to explore the practical boundaries, resource trade-offs, and deployment feasibility of using LLMs for gene ontology annotation. We systematically benchmark multiple LLMs—ranging in size and architecture—against a strong Bi-GRU baseline, evaluating not just annotation quality but also memory usage, inference time, hallucination rates, and fine-tuning efficiency. Our contribution lies in surfacing when LLMs might be justified in ontology annotation, and what costs those decisions entail. In this study, we investigate the potential of LLMs for automated ontology annotation of scientific literature, specifically focusing on Gene Ontology (GO) annotations. We conducted extensive experiments, fine-tuning LLMs [18] with varying parameter sizes, architectures, and pretraining datasets. After fine-tuning, we rigorously evaluated each model’s performance using metrics such as F1 score and semantic similarity score, which assess the model’s semantic accuracy in ontology annotations. Additionally, we monitored memory and GPU utilization during fine-tuning and measured inference times to gauge computational efficiency. This analysis provides insights into LLMs’ performance improvements, resource demands, and deployment feasibility for real-world ontology annotation tasks, comparing them to recurrent neural networks. Methods Dataset selection and preparation The Colorado Richly Annotated Full-Text (CRAFT) gold standard dataset was used as the training resource for the models in this study. The CRAFT (Colorado Richly Annotated Full Text) dataset is an annotated corpus of 97 full-text biomedical literature, designed for use in natural language processing and text mining. Its ontology annotations span multiple biomedical domains, including Gene Ontology (GO), Chemical Entities of Biological Interest (ChEBI), and Sequence Ontology (SO), among others. This dataset provides comprehensive term and concept annotations, supporting tasks like named entity recognition, ontology mapping, and semantic analysis, making it valuable for training and evaluating models in biomedical text analysis. Sentence segmentation was applied to each article in the CRAFT corpus, resulting in 27,946 extracted sentences. Each sentence includes zero or more words or phrases annotated with Gene Ontology (GO) concepts, identified by unique GO IDs to ensure a structured ontology annotation framework. We split the dataset into a training set of 22,364 sentences and an evaluation set of 5582 sentences. Our goal is to utilize this dataset to evaluate and optimize large language models (LLMs) for accurately predicting GO concepts tied to words or phrases in input sentences. Baseline model selection and comparison framework In previous work, we trained various Bidirectional Gated Recurrent Unit (Bi-GRU) architectures using the CRAFT dataset, enriched with parts-of-speech tags and supplementary information from sources like NCBI’s BioThesaurus and the Unified Medical Language System (UMLS). Among these, our most successful model achieved an F1 score and a semantic similarity score of 0.84. This model will serve as the baseline for comparison with several large language models (LLMs) that we will subsequently fine-tune on the CRAFT dataset. We considered other neural architectures such as Bi-LSTMs and CNNs, which have been applied to ontology annotation. In our prior work [1, 8], we systematically compared these models and found that Bi-GRUs consistently outperformed Bi-LSTMs and CNNs in capturing long-range dependencies and producing more accurate annotations. Based on this evidence, we use Bi-GRUs as the representative deep learning baseline in this study. Additionally, we introduced a novel post-processing technique termed “Ontology Boosting”, aimed at enhancing the confidence of predictions generated by our Bi-GRU models [7]. Through the implementation of Ontology Boosting, we achieved further performance improvements, attaining an F1 score of 0.85 and a semantic similarity score of 0.90. Throughout our experimentation with LLM finetuning, we will primarily evaluate the performance and memory requirements of these models in comparison to our Bi-GRU model training. We note that the Bi-GRU model was augmented with additional resources including part-of-speech tags, NCBI BioThesaurus, and UMLS, as well as postprocessing rules. In contrast, the LLMs were evaluated without supplementary domain resources or rule-based refinements, reflecting their use as general-purpose models. This design choice highlights differences in paradigm rather than establishing a strictly resource-matched comparison. Large language model selection We began our experiments by selecting a foundational large language model to serve as a basis for optimizing different stages of our study. The initial model selection focused on finding a smaller variant of the state-of-the-art LLM available at the time, balancing computational efficiency with performance. We chose MPT-7B [19], a seven-billion-parameter, decoder-style transformer model pretrained on one trillion tokens of English text and code, due to its manageable size and enhanced throughput during training and inference. This selection minimized discrepancies in resource use, ensuring a fair comparison between RNNs and LLMs in terms of both performance and efficiency. In addition, we selected the following models for comparison with our baseline model: Phi Phi [20] represents a major advancement in the design of large language models, focusing on scaling to handle more complex reasoning tasks. It builds on the architecture of previous transformer models but introduces optimizations for more nuanced comprehension and generation, particularly in multi-step reasoning tasks. Phi’s approach allows it to produce high-quality responses across a wide range of topics, from scientific research to general knowledge. Additionally, it incorporates advanced techniques to improve its ability to learn from both structured and unstructured data, making it a powerful tool for applications that require deep understanding and contextual sensitivity. BiomedLM BiomedLM [21] is a domain-specific large language model fine-tuned on vast amounts of biomedical literature, clinical reports, and medical datasets. Its training enables it to provide highly accurate and relevant outputs in medical and health-related contexts. BiomedLM excels at extracting insights from scientific articles, making it particularly useful in areas such as drug discovery, bioinformatics, medical research, and clinical decision support. Its fine-tuning on domain-specific data gives it a significant edge in understanding complex biomedical terminologies and relationships, enabling it to assist healthcare professionals in navigating vast repositories of medical knowledge and generating hypotheses for novel medical discoveries. Falcon Falcon [22], developed by the Technology Innovation Institute, is an efficient and high-performance large language model designed for both generative and analytical tasks. Falcon is known for its ability to generate coherent and contextually accurate responses while being optimized for lower computational requirements, making it suitable for deployment in real-world applications where resources may be limited. It excels in areas such as text summarization, question-answering, and natural language generation, all while maintaining a strong balance between speed and accuracy. Falcon’s efficiency also allows it to be applied in a variety of industries, from healthcare to e-commerce, without overwhelming system resources. Meditron Meditron [23] is a specialized LLM fine-tuned for the healthcare domain, offering tailored solutions for processing medical texts and clinical data. Its architecture is optimized for understanding and interpreting complex medical terminology, including clinical notes, research papers, and diagnostic reports. Meditron can be applied in various clinical settings, helping to streamline workflows by assisting in tasks such as automated diagnosis support, clinical decision-making, and patient care recommendations. The model is designed to ensure high accuracy in medical contexts, where misinterpretation of information could have serious consequences. Llama 2 Llama 2 [12] is a series of LLMs developed by Meta, designed to provide efficient and high-performing models for general-purpose language tasks. Unlike some other models that are highly specialized, Llama 2 is intended to be a versatile, scalable solution for a variety of natural language processing (NLP) tasks, such as text generation, translation, summarization, and question-answering. Its architecture is optimized for performance and adaptability, making it suitable for deployment in both research environments and commercial applications. Mistral Mistral [24] is an open-weight, high-performance LLM that is designed to excel in multitask learning, with a focus on fine-tuning for specific domains such as programming, healthcare, and customer service. One of the standout features of Mistral is its ability to learn from diverse tasks and datasets without requiring exhaustive retraining for each new application. MPT Mosaic Pretrained Transformer (MPT) [19] is a robust, open-source LLM designed by MosaicML that stands out for its efficiency in large-scale fine-tuning. MPT is built to handle a wide range of NLP tasks, including text generation, summarization, and question-answering, with an emphasis on performance at scale. Its open-source nature allows organizations to customize the model to fit their specific needs, particularly in industries requiring specialized knowledge, such as finance, healthcare, and education. MPT’s ability to be fine-tuned with relatively smaller datasets makes it a cost-effective option for scientific research groups looking to implement advanced LLM capabilities without requiring massive computational resources. Finetuned ChatGPT Finetuned ChatGPT refers to the specialized versions of OpenAI’s popular GPT models [10], which have been fine-tuned on specific datasets or for particular tasks to improve their performance in conversational AI applications. The base ChatGPT model has been trained on vast amounts of general-purpose data, making it highly capable in a variety of domains. However, when fine-tuned, it can achieve even greater accuracy in specialized areas such as domain-specific research. This fine-tuning allows ChatGPT to produce more contextually relevant and accurate responses. After optimizing the prompt for the base large language model, we used the same system message, instruction, input sentence, and ground truth response data to fine-tune GPT-3.5. Since the GPT-3.5 fine-tuning API requires a specific input format, we converted our data to align with the following structure: All training and evaluation datasets were converted into this format. The user role contains the input sentence, while the assistant role includes the corresponding ground truth response. Once formatted, the training dataset was uploaded to OpenAI’s servers, and a fine-tuning job was created. Upon completion, OpenAI provided a unique model identifier for inference. For evaluation, we queried the fine-tuned model using OpenAI’s Chat Completion API, passing the model identifier and the messages (without the assistant’s ground truth content). The model’s responses were stored and parsed for comparison against the ground truth annotations to evaluate performance on terms, GO concepts, and hierarchical parent categories. Expansion of model selection Following the fine-tuning process for MPT-7B, we expanded our selection of models to include a diverse range, guided by considerations such as parameter size, model architecture, and the dataset used for pretraining. This expansion was driven by the need to understand the effects of various variables on the efficacy of ontology annotations while also accounting for resource utilization. We selected a varied set of models spanning a range of parameter sizes, from 1.5B to 7B parameters. The decision to cap the selection at 7B was motivated by the substantial increase in resource requirements associated with larger models. Although the next closest parameter size at the time of experimentation was 13B, opting for models of that magnitude would have demanded significantly higher resources. In contrast, 7B models offered a balanced compromise—reduced memory footprint, lower computational demands, and shorter training and inference times. This selection enabled a more meaningful and practical comparison between our baseline models and the larger language models in terms of both performance and resource utilization. Our selection was also influenced by the architectural designs and configurations of the models. We sought to explore the impact of architectural factors, such as attention mechanisms and layer structures, on the compute cost and performance of ontology annotations. Different architectures offered distinct optimization techniques, which in turn affected training and inference time, as well as memory footprint. Furthermore, our selection was guided by the type of dataset used for pretraining the large language models. We aimed to investigate whether models trained on scientific corpora would yield better performance in ontology annotation tasks compared to those trained on general English language data sourced from Common Crawl. Fine-tuning for the initial model We conducted a comprehensive fine-tuning processes of the initial model that encompassed four distinct stages: Prompt tuning We began the prompt-tuning stage to enhance generative performance and reduce the risk of hallucinations in our fine-tuned model. This process started with a single prompt task, where the model was instructed to extract terms associated with GO concepts from a given input sentence. The prompt directed the model to identify and extract any words or phrases from the sentence related to concepts in the GO hierarchy. If no associations were found, the model was expected to respond by indicating the absence of relevant terms. Below is an example of the initial prompt-response data: This prompt-response format served as a template applied to all input sentences, creating a comprehensive dataset for fine-tuning the initial model. The dataset, incorporating prompts and model responses, was integrated into both the training and evaluation sets to ensure consistent application throughout the fine-tuning process. The model was then fully fine-tuned on this updated training dataset. During evaluation, the fine-tuned model received prompts from the evaluation set, generating responses based on learned associations between input terms and GO concepts. These generated terms were then compared against ground truth annotations to assess performance. We iteratively refined the prompt formulation, adjusting its language and specificity to elicit more accurate, relevant responses from the model. Additional information was introduced to help the model better understand GO concepts by prompting it to associate terms with their unique identifiers (GO IDs). However, including these IDs led to hallucinations, with the model generating invalid identifiers. To address this, we removed the IDs from prompts and instead instructed the model to add parent concepts for each term. This adjustment improved the model’s grasp of ontology hierarchy, enhancing annotation accuracy. Furthermore, we explored the effectiveness of framing prompts within specific contexts or personas [25], such as that of a gene ontology expert. By providing contextual cues, we aimed to guide the model towards generating responses that were not only relevant and coherent but also aligned with the given prompt. This approach proved beneficial in enhancing the model’s performance, as it helped the model gain a better understanding of the context in which it was operating. Additionally, it encouraged the model to prioritize identifying words commonly found in the gene ontology domain, thereby improving the relevance and accuracy of its responses. We also experimented with the effect of prompting the model to generate responses in JSON format, as well as highlighting important keywords by representing them in all uppercase. These additional formatting techniques proved effective in enhancing the clarity and structure of the generated responses. Through these iterative adjustments to prompt tuning, we sought to optimize the effectiveness of the prompts in guiding the model’s generative behavior and enhancing its ability to extract ontology-related terms from input sentences. After thoroughly evaluating the impact of each adjustment, we finalized the prompt, which involved instructing the model to associate concepts with each identified term, include immediate parent concepts for each associated concept, and frame the context within the persona of a gene ontology expert. The final prompt thus represented the culmination of our iterative process and served as a comprehensive guideline for steering the model’s generative behavior towards producing contextually relevant and accurate ontology annotations. Architecture tuning Once we had formulated the prompt that yielded the best possible response, we moved on to the next step. In this step, we explored various supervised fine-tuning techniques to further train the pretrained large language model on our smaller training dataset, aiming to refine the model’s capabilities and improve its performance in ontology annotation. Among several fine-tuning methods, we selected two of the most common approaches: full fine-tuning and parameter-efficient fine-tuning. Full fine-tuningFull fine-tuning involved training the entire model architecture, including all layers and parameters, on our downstream ontology annotation task. This approach allowed the model to adapt its learned representations and parameters to the specific nuances of the task, potentially improving its performance and effectiveness for the annotation task. Using the final prompt template derived from the prompt tuning stage, we computed the total number of tokens across all observations in our dataset. The maximum number of tokens per observation was found to be 710. Therefore, we selected 1024 as the maximum sequence length for full fine-tuning. This choice was strategically made to balance the need to adequately encompass the dataset while mitigating memory constraints during both training and inference.We experimented with several other fine-tuning parameters. Specifically, we explored parameters such as learning rate, batch size, number of training epochs, optimizers, and schedulers to identify configurations that would yield optimal performance for our annotation task. We identified the parameter values that demonstrated superior performance: a batch size of 8, 3 training epochs, a learning rate of 5.0e-06, and the use of the decoupled AdamW optimizer with linear decay and 50 batches for warm-up.To enhance computational efficiency, our chosen model featured a capability called flash attention. This feature was utilized in our experiments to improve both computation and memory efficiency during fine-tuning and inference. To overcome memory constraints associated with fine-tuning large language models, we implemented data-parallel training using Full Sharded Data Parallel (FSDP). This data-parallel paradigm shards the optimizer states, gradients, and parameters across data-parallel workers, enabling efficient utilization of computational resources and facilitating the training of large language models. By adopting these techniques, we successfully accommodated the memory requirements of a 7-billion-parameter MPT model with a global batch size of 24 across 3 NVIDIA A6000 (48GB) GPUs. Full fine-tuning Full fine-tuning involved training the entire model architecture, including all layers and parameters, on our downstream ontology annotation task. This approach allowed the model to adapt its learned representations and parameters to the specific nuances of the task, potentially improving its performance and effectiveness for the annotation task. Using the final prompt template derived from the prompt tuning stage, we computed the total number of tokens across all observations in our dataset. The maximum number of tokens per observation was found to be 710. Therefore, we selected 1024 as the maximum sequence length for full fine-tuning. This choice was strategically made to balance the need to adequately encompass the dataset while mitigating memory constraints during both training and inference. We experimented with several other fine-tuning parameters. Specifically, we explored parameters such as learning rate, batch size, number of training epochs, optimizers, and schedulers to identify configurations that would yield optimal performance for our annotation task. We identified the parameter values that demonstrated superior performance: a batch size of 8, 3 training epochs, a learning rate of 5.0e-06, and the use of the decoupled AdamW optimizer with linear decay and 50 batches for warm-up. To enhance computational efficiency, our chosen model featured a capability called flash attention. This feature was utilized in our experiments to improve both computation and memory efficiency during fine-tuning and inference. To overcome memory constraints associated with fine-tuning large language models, we implemented data-parallel training using Full Sharded Data Parallel (FSDP). This data-parallel paradigm shards the optimizer states, gradients, and parameters across data-parallel workers, enabling efficient utilization of computational resources and facilitating the training of large language models. By adopting these techniques, we successfully accommodated the memory requirements of a 7-billion-parameter MPT model with a global batch size of 24 across 3 NVIDIA A6000 (48GB) GPUs. Parameter-Efficient Fine-Tuning (PEFT)Unlike full fine-tuning, where all parameters of the model are adjusted, parameter-efficient fine-tuning aimed to optimize the fine-tuning process to achieve comparable performance on downstream tasks while minimizing the number of parameters and computational resources required. Several approaches to parameter-efficient fine-tuning exist, with quantization and low-rank approximation being the most commonly used. One such approach, Quantized Low-Rank Adapters (QLoRA), involved quantizing a pretrained LLM to 4/8 bits, then adding a small set of learnable low-rank adapter weights that were adjusted during backpropagation.We utilized 4-bit NormalFloat quantization across all QLoRA fine-tuning experiments. Additionally, we targeted all linear layers for low-rank approximation, as adding Low-Rank Adapters (LoRA) to all linear layers was found to be crucial for matching or even improving upon the performance of full fine-tuning. Consistent with our full fine-tuning experiments, we maintained the same values for training epochs, maximum sequence length, and batch size. Flash attention was also implemented throughout all experiments.Through extensive experimentation with varying QLoRA parameter values, such as projection dimension (r), scaling factor (), dropout, and other standard fine-tuning parameters, we determined that the following parameter configuration yielded the best results in terms of fine-tuning performance: a projection dimension of 256, scaling factor of 128, dropout of 0.1, a learning rate of 2.0e-04, and the AdamW optimizer with 8-bit precision, implemented with a cosine schedule and a warm-up of 50 batches. Through meticulous parameter tuning and adaptation strategies, we were able to achieve resource-efficient model fine-tuning while maintaining performance comparable to that achieved through full fine-tuning of the LLM. Parameter-Efficient Fine-Tuning (PEFT) Unlike full fine-tuning, where all parameters of the model are adjusted, parameter-efficient fine-tuning aimed to optimize the fine-tuning process to achieve comparable performance on downstream tasks while minimizing the number of parameters and computational resources required. Several approaches to parameter-efficient fine-tuning exist, with quantization and low-rank approximation being the most commonly used. One such approach, Quantized Low-Rank Adapters (QLoRA), involved quantizing a pretrained LLM to 4/8 bits, then adding a small set of learnable low-rank adapter weights that were adjusted during backpropagation. We utilized 4-bit NormalFloat quantization across all QLoRA fine-tuning experiments. Additionally, we targeted all linear layers for low-rank approximation, as adding Low-Rank Adapters (LoRA) to all linear layers was found to be crucial for matching or even improving upon the performance of full fine-tuning. Consistent with our full fine-tuning experiments, we maintained the same values for training epochs, maximum sequence length, and batch size. Flash attention was also implemented throughout all experiments. Through extensive experimentation with varying QLoRA parameter values, such as projection dimension (r), scaling factor (), dropout, and other standard fine-tuning parameters, we determined that the following parameter configuration yielded the best results in terms of fine-tuning performance: a projection dimension of 256, scaling factor of 128, dropout of 0.1, a learning rate of 2.0e-04, and the AdamW optimizer with 8-bit precision, implemented with a cosine schedule and a warm-up of 50 batches. Through meticulous parameter tuning and adaptation strategies, we were able to achieve resource-efficient model fine-tuning while maintaining performance comparable to that achieved through full fine-tuning of the LLM. Parameter tuning for response generation Response generation by large language models is an auto-regressive process [26, 27]. Initially, all tokens in the input prompt are processed in parallel, and the model predicts the next token in the sequence based on the provided input context. Using a selection strategy, the most likely token is chosen. This process of prediction and selection is repeated iteratively to generate a sequence of tokens until a termination condition is met. Several adjustable parameters influence the behavior of large language models during response generation [28]. One such parameter is the maximum number of generation tokens. A shorter maximum length encourages the model to prioritize coherence and conciseness, while a longer maximum length can promote diversity and creativity. For our use case, we needed the response to be concise and focused, while minimizing hallucination as much as possible. Therefore, a lower maximum generation token limit was necessary. However, excessive restriction of this limit could hinder the model’s ability to identify potential missing annotations in the input sentences that were not present in the ground truth. Given that we observed a maximum response length of 182 tokens within our dataset, we selected a maximum generation token limit of 250, which provided a good balance between conciseness and the potential to identify missing annotations. It is worth noting that this value might differ slightly for different large language models due to variations in the model’s tokenizer and vocabulary [29]. As an added benefit, a lower generation limit also contributed to reduced memory utilization and faster inference times per prompt. Another major factor influencing the quality of responses generated by these models was the sampling method. Temperature, number of beams, top-k, and top-p were several parameters that provided granular control over the characteristics of the responses [30, 31]. Higher temperature values promoted creativity and novelty, while lower values produced more focused and contextually relevant responses. Similarly, top-k sampling restricted the token vocabulary to the top-k most probable tokens during generation. Limiting the vocabulary to the top-k tokens encouraged selectivity and reduced diversity in the response. The number of beams, another critical parameter in response generation, determined the number of parallel explorations of candidate sequences to identify the optimal sequence. A higher number of beams tended to uncover high-quality sequences that might otherwise have been overlooked with fewer beams, potentially enhancing response generation performance. However, a higher number of beams also increased the computational resources required for inference. We carefully considered these factors and evaluated performance across various parameter values. The optimal generation performance was achieved with a temperature of 0.001, a top-p of 0.95, and the number of beams set to 4. Response parsing for concept generation In traditional deep learning approaches, such as those using Bi-GRU models, the outputs are typically numerical. Each token in the input sentence is mapped to an n-dimensional vector representing probabilities across n predefined ontology concepts. A softmax layer is then applied to select the most probable concept for each token. In contrast, large language models (LLMs) produce free-form outputs, which require additional post-processing. With a well-structured prompt and clear instructions, we expect the LLM to return a properly formatted JSON response for each input sentence. However, in some instances, the model may generate output that deviates from the expected format. In such cases, we employ regular expression (regex)-based parsing to extract the lists of terms and associated GO concepts from the text. Once parsed, we obtain a structured list of predicted terms and their corresponding concepts. Before we evaluate the model’s predictions, we check for hallucinations—cases where the model predicts terms that were never actually mentioned in the original sentence. If a predicted term isn’t found in the input, we remove it and any related GO concepts from the evaluation. We then assess the model’s performance using only the predictions that match real terms from the input. Embedding matching on generated concepts One common issue with large language models is hallucination, where the model generates terms not present in the input or associates valid terms with incorrect concepts. In our task, this often appeared as either fabricated words or mismatched Gene Ontology (GO) concepts. In some cases, the generated concept was semantically equivalent but not syntactically identical to the correct ontology term—for example, meiotic homologous chromosome pairing versus the canonical homologous chromosome pairing at meiosis. To avoid false negatives from such variations, we applied embedding matching. Specifically, we compared embeddings of generated concepts against all GO terms using cosine similarity and selected the highest-scoring match. Embeddings were computed with all-mpnet-base-v2, a sentence-transformers model fine-tuned on one billion sentence pairs to capture semantic similarity. Performance evaluation metrics The performance of both the baseline Bi-GRU model and the boosted Bi-GRU models was evaluated using a modified F1 score and Jaccard semantic similarity score. The modified F1 score involved removing all instances of accurately predicted out-of-concept tokens to reduce potential bias introduced by the higher occurrence of out-of-concept tokens in the evaluation dataset. Out-of-concept tokens are those not associated with any specific concept and are prevalent in our dataset, often vastly outnumbering concept tokens. However, in the case of large language models (LLMs), which generate text rather than predicting individual tokens, instances of out-of-concept predictions were never encountered. Therefore, the reported scores for the evaluation of these LLMs comprised the unmodified F1 score and Jaccard semantic similarity. The F1 score quantified the model’s ability to annotate concepts precisely, while the Jaccard semantic similarity measured the model’s ability to annotate concepts with semantically similar alternatives when exact concepts were not annotated. This metric assessed the ontological distance between concepts, providing insights into the model’s understanding of semantic relationships within the ontology. When a model/LLM predicts concepts that are parent or child terms of the ground truth in the GO hierarchy, these are not counted as exact matches. Instead, we rely on semantic similarity metrics, which assign partial credit by quantifying the ontological relatedness of predicted and true terms. This approach ensures that predictions capturing semantically coherent but non-identical concepts are appropriately reflected in the evaluation. Results Annotation performance Figure 1 presents a comparison of model performance across both F1 score and semantic similarity score for ontology-based annotation. The boosted Bi-GRU baseline achieved the highest overall performance, with an F1 score of 0.85 and a semantic similarity score of 0.90, reflecting the advantage of domain-specific training and postprocessing. Among LLMs, the 7B-parameter models (Falcon, Meditron, Llama 2, Mistral, and MPT) performed competitively, with F1 scores ranging from 0.839 to 0.878 and semantic similarity scores closely aligned (0.840–0.876), indicating consistent accuracy across different architectures of similar scale. Smaller models such as Phi (1.5B) and BiomedLM (2.7B) showed moderately lower performance, while the fine-tuned ChatGPT 3.5 Turbo achieved an F1 score of 0.685 and semantic similarity of 0.699. ChatGPT 4, when used without fine-tuning, performed poorly (F1 = 0.048), underscoring the importance of adaptation for domain-specific annotation tasks. Importantly, the semantic similarity scores closely track the F1 results, highlighting that while LLMs may not surpass the Bi-GRU baseline in raw accuracy, they achieve semantically coherent predictions that reflect partial correctness when parent or child ontology terms are predicted. Performance comparison of the boosted Bi-GRU baseline and various large language models (LLMs) on ontology-based annotation tasks. Bars show F1 scores (dark green) and semantic similarity scores (light green). The Bi-GRU achieved the highest performance, while 7B-parameter LLMs performed competitively across both metrics. Semantic similarity scores highlight cases where LLMs capture semantically related but non-identical ontology concepts The Boosted Bi-GRU (38 M) baseline achieves the highest semantic similarity score at 0.900, indicating strong alignment with ontology-based annotations. Among the LLMs, Phi (1.5B) and Mistral (7B) perform comparably, achieving semantic similarity scores of 0.876 and 0.874, respectively. Models such as BiomedLM (2.7B) and Llama 2 (7B) fall in a similar range (0.858–0.860), showing that while LLMs do not always outperform the baseline, they offer competitive semantic understanding. Fine-tuned ChatGPT-3.5 Turbo (3.5B) achieved a lower semantic similarity score of 0.699, while ChatGPT-4, despite its larger parameter count, significantly underperformed on this task with a score of 0.061. These differences underscore the variability in LLM performance depending on architecture, pretraining domain, and fine-tuning regime. Although the overall accuracy of LLM-based models is comparable to, and in some cases slightly below, the Bi-GRU baseline, our analysis indicates that LLMs capture semantic nuances more effectively. While qualitative inspection suggested that LLMs occasionally captured semantically related or multi-word terms missed by the Bi-GRU, this trend was not systematically quantified. Therefore, we treat such patterns as indicative observations rather than conclusive evidence. A comparison of zero-shot and fine-tuned LLMs further illustrates the value of domain-specific adaptation. While the zero-shot ChatGPT-4 model performed poorly (F1 = 0.048, semantic similarity = 0.061), the fine-tuned ChatGPT-3.5 Turbo achieved substantially higher performance (F1 = 0.685, semantic similarity = 0.699). This highlights the importance of fine-tuning for aligning general-purpose LLMs to specialized ontology annotation tasks. It is important to interpret these results in light of the differing resource conditions: the Bi-GRU models were supplemented with curated linguistic and biomedical resources, while the LLMs operated without such external inputs. Thus, the comparison illustrates the strengths of domain-tuned baselines versus resource-agnostic LLMs, rather than a fully symmetric evaluation. Computational resource trade-offs We evaluated GPU memory usage during fine-tuning and inference (Fig. 2). As expected, LLMs require substantially more resources. The 7B models, Falcon, Meditron, Llama 2, Mistral, and MPT consume between 125 and 139 GB of GPU memory during fine-tuning and around 15–16 GB during inference. In contrast, the Boosted Bi-GRU model remains highly efficient, using just 29.4 GB for fine-tuning and 7.3 GB for inference. GPU memory usage (in GB) for fine-tuning (light green) and inference (dark green) across models. Boosted Bi-GRU is the most efficient, while 7B models require over 125 GB for fine-tuning Phi (1.5B) and BiomedLM (2.7B) fall in the mid-range, using between 45 and 57 GB during fine-tuning and under 7 GB during inference—highlighting that moderately sized models may offer a better balance between performance and cost. Inference time Figure 3 shows average inference time per prompt. Boosted Bi-GRU again demonstrates its efficiency at 2 ms, making it suitable for high-throughput or real-time annotation. Phi, despite strong semantic similarity, has the highest latency at 3976 ms, followed by MPT and Mistral, both exceeding 1000 ms. These results indicate a clear trade-off between semantic richness and latency—crucial when scaling to large corpora. Average inference time per prompt (ms) across models. Boosted Bi-GRU is the fastest by orders of magnitude, while LLMs like phi and MPT show significantly higher latency Parameter-efficient fine-tuning (QLoRA) Figure 4 compares full fine-tuning with QLoRA. Notably, BiomedLM maintains a high semantic similarity score (0.842) with QLoRA while halving GPU usage, illustrating the potential of efficient fine-tuning techniques. Other models, like Falcon, show performance degradation with QLoRA, emphasizing the need for model-specific tuning strategies. Comparison of full finetuning and QLoRA finetuning training time across models. While QLoRA generally reduces training time compared to full finetuning for Llama 2 and Mistral, Falcon exhibits a substantial increase in training time with QLoRA. BiomedLM shows minimal difference between the two methods Hallucinations and beam search Figures 5 and 6 explore hallucination counts. Increasing the number of beams generally reduces hallucinations—for instance, Meditron drops from 45 (1 beam) to 26 (4 beams). However, more training epochs tend to increase hallucination rates, particularly in Meditron and Mistral. These findings suggest a nuanced trade-off between training depth and output reliability. Effect of beam size (1 vs. 4) on hallucination counts. Increasing the number of beams generally reduces hallucinations across models, especially for meditron and mistral Effect of training epochs (3 vs. 10) on hallucination counts. Increases in hallucinations are observed with more training, particularly in Meditron and Mistral Error analysis To better understand the limitations of our models, we conducted a qualitative error analysis on predictions made against the CRAFT corpus. A consistent pattern emerged: errors were disproportionately concentrated among Gene Ontology (GO) terms that appeared fewer than 20 times in the training data. This sparsity led to weak or unstable associations between terms and their corresponding ontology concepts, which reduced both precision and recall for these cases. For example, in the sentence: “The AVPR2-deficient pups exhibited impaired renal water reabsorption,” the ground truth annotation links “renal water reabsorption” to GO:0003091 (renal water reabsorption). While the Bi-GRU correctly assigned the term, several LLMs instead predicted the broader parent category GO:0003011 (renal system process). This suggests that limited exposure to the specific fine-grained concept in training encouraged fallback to higher-level ontology terms. Another case involved the sentence: “Spindle pole body duplication was disrupted in the mutant strain,” where the gold annotation is GO:0051300 (spindle pole body duplication). Models with fewer than 10 training examples of this concept frequently predicted GO:0007059 (chromosome segregation) or GO:0000226 (microtubule cytoskeleton organization). These predictions were semantically related but failed to capture the precise concept. A similar trend was observed with low-frequency developmental terms. In the sentence: “Failure of somite rostral–caudal patterning was observed in the embryo,” the correct annotation is GO:0030902 (somite rostral/caudal axis specification). However, models often defaulted to GO:0009952 (anterior/posterior pattern specification), a more generic developmental process that occurs more frequently in training data. Across all error cases, we found that concepts with fewer than 20 annotated instances accounted for over 70% of misclassifications. This reinforces the well-known challenge of long-tail distributions in ontology annotation: while common biological processes are consistently recognized, rare or highly specific concepts remain difficult for both traditional neural models and LLMs. Future work should explore strategies such as ontology-aware data augmentation, curriculum learning, or transfer from related high-frequency terms to address this imbalance. Discussion The findings from our evaluation reveal notable differences in the performance of general-purpose large language models (LLMs) and traditional neural architectures on the task of gene ontology (GO) annotation. Unlike standard F1 scores, semantic similarity (Fig. 1) provides a more nuanced measure of performance by accounting for the hierarchical structure of GO and offering concept-level alignment between model predictions and gold-standard annotations [32, 33]. LLMs vs. Bi-GRU: ontology-specific performance The Boosted Bi-GRU achieved the highest semantic similarity score (0.900), outperforming all LLMs despite its smaller parameter size. This aligns with previous findings that smaller, task-specific models can outperform larger architectures when domain-specific supervision and structured input features are leveraged [34]. Its sequential modeling and explicit training on GO-labeled data make it particularly effective in disambiguating biomedical concepts. By contrast, general-purpose LLMs performed inconsistently. Surprisingly, ChatGPT-4 achieved the lowest semantic similarity score (0.061), despite its strong performance on general language understanding benchmarks [35]. This result underscores the limitations of general-purpose LLMs in structured biomedical tasks, where they often generate semantically plausible but non-canonical or unverifiable terms [36]. In contrast, domain-adapted models such as Phi (1.5B) [37] and BiomedLM [38], which are pretrained on biomedical corpora, demonstrated stronger alignment with ontology concepts. These findings highlight the importance of domain-specific pretraining for improving ontology-aware semantic grounding [39]. While general-purpose LLMs do not outperform domain-specific neural architectures in overall accuracy, they occasionally generate semantically coherent near-miss predictions that align with related ontology concepts. We interpret this behavior as suggestive, though not conclusive, evidence of semantic robustness. Because our study did not explicitly stratify performance by concept frequency or term length, we refrain from making empirical claims regarding advantages on rare or multi-word ontology terms and instead present these trends as qualitative observations for future investigation. Efficiency trade-offs Beyond performance, resource usage is a critical factor in model selection. The Boosted Bi-GRU required significantly less GPU memory and inference time compared to 7B LLMs like Mistral, Falcon, and Llama 2 (Figs. 2 and 3). While LLMs may offer competitive semantic performance, their computational demands may render them impractical for real-time or large-scale annotation workflows, particularly in clinical or embedded environments. Parameter-efficient fine-tuning and hallucinations Fine-tuning strategies also play a pivotal role in real-world deployment. Our QLoRA experiments (Fig. 4) showed that BiomedLM preserved semantic performance while halving memory usage—an encouraging result that supports the promise of parameter-efficient fine-tuning [40]. However, other models like Falcon showed performance degradation, suggesting that quantization effectiveness varies by architecture. Hallucination analysis (Figs. 5 and 6) revealed that increasing beam width generally reduces hallucinations, consistent with prior observations that beam search stabilizes generation [41]. However, prolonged training can increase hallucination rates, particularly in larger models like Meditron and Mistral, suggesting a trade-off between learning depth and generation fidelity. Interestingly, our results show that in some cases, such as with the Falcon model, full finetuning was faster than parameter-efficient finetuning using QLoRA. While PEFT methods are generally expected to reduce training time by updating only a small subset of model parameters [42], they can introduce additional computational overhead. This includes less optimized hardware utilization [43], memory bottlenecks from managing quantized weights, and architectural mismatches that impact efficiency. In the case of Falcon, these factors may have outweighed the benefits of reduced parameter updates, highlighting that the practical speed gains of PEFT methods can be model-dependent and should be empirically validated rather than assumed [44]. Implications and future work These findings suggest that larger LLMs do not inherently yield better performance for biomedical ontology annotation. Instead, domain-specific models—whether small like the Bi-GRU or moderately scaled like BiomedLM—offer better alignment with GO concepts and more efficient deployment profiles. Our results advocate for continued research into domain adaptation, symbolic reasoning integration, and fine-tuning methods tailored to ontology-based tasks. Future work should explicitly quantify whether domain-pretrained LLMs maintain these qualitative advantages over general-purpose LLMs on long-tail and compositional ontology mentions. While we explored multiple configurations of learning rate, batch size, epochs, and optimizer settings, the final set of hyperparameters reported here was selected based on empirical observation of stable convergence and validation performance. However, without a systematic ablation study, our claims regarding the superiority of these choices remain tentative. We therefore acknowledge that these results should be interpreted as indicative rather than definitive. A more rigorous ablation across hyperparameter dimensions, architectures, and fine-tuning strategies represents an important direction for future work to establish the robustness and generalizability of the observed trends. Finally, although our experiments focused on the CRAFT corpus—which remains the largest human-curated dataset for GO annotation [45]—future work should explore generalizability across additional biomedical corpora to better assess model robustness in diverse annotation settings. Conclusions This study demonstrates that large language models (LLMs) have potential for improving ontology annotation in scientific literature, particularly in the biomedical domain. By fine-tuning multiple LLMs, including MPT-7B, Phi, BiomedLM, and Meditron, on the CRAFT dataset, we observed meaningful improvements in the models’ ability to accurately tag Gene Ontology (GO) concepts within complex biomedical texts. The fine-tuning and parameter-efficient tuning techniques, such as QLoRA, allowed certain models to achieve performance on par with traditional RNN-based architectures like Bi-GRU while also maintaining resource efficiency. Performance metrics, including F1 score and semantic similarity, confirmed that LLMs can achieve comparable performance to traditional RNN models in capturing semantic nuances and handling indirect concept references, which are prevalent in biomedical literature. However, we also observed trade-offs in terms of computational demands, as larger models required substantially more memory and processing time during fine-tuning and inference. The findings suggest that while LLMs can match or slightly increase annotation accuracy, the decision to implement them in real-world applications should consider both performance gains and resource requirements. However, challenges remain, especially in balancing model accuracy with computational efficiency. Techniques such as parameter-efficient fine-tuning (PEFT) specifically low-rank adaptation (LoRA) via qLoRA—show promise in optimizing these trade-offs, though further exploration is needed to fully assess their effectiveness. Our results also highlight important trade-offs between LLMs and traditional neural architectures like Bi-GRU in the context of gene ontology annotation. While some LLMs show modest improvements in F1 scores or semantic similarity, these gains come with significant increases in computational cost, memory demands, and inference latency. In contrast, the boosted Bi-GRU model demonstrates comparable annotation accuracy with vastly lower resource requirements, making it better suited for large-scale or real-time applications. The justification for using LLMs therefore depends on the application context. In use cases where semantic ambiguity, indirect term references, or domain generalization are especially challenging—and where computational resources are sufficient—LLMs may provide tangible benefits. However, in environments where speed, efficiency, and cost-effectiveness are paramount, traditional models remain a strong and interpretable alternative. Our study contributes not by claiming superiority of one class of model over another, but by articulating the conditions under which each approach is optimal. In summary, this work offers a practical framework for evaluating the trade-offs of using LLMs for biomedical ontology annotation. It provides valuable empirical insights that can guide practitioners in selecting models that balance semantic depth with resource constraints, and lays the groundwork for future explorations of hybrid or progressive annotation pipelines that combine the strengths of lightweight and large-scale models. Finally, our reliance on the CRAFT corpus as the sole evaluation dataset is a limitation, as it may not fully capture the diversity of biomedical writing styles or annotation challenges encountered in broader literature and clinical notes. Future work should evaluate model generalizability across additional corpora to strengthen the robustness of these findings. Acknowledgements Not Applicable. Author contributions P.D conducted the data analysis and prepared the first draft of the manuscript. S.M and P.M supervised the research and conceived the experiments. P.M prepared the submitted draft of the manuscript. All authors reviewed the manuscript. Funding This work is funded by a CAREER award (#1942727) from the Division of Biological Infrastructure at the National Science Foundation, USA. Data availability No datasets were generated or analysed during the current study. Declarations Ethics approval and consent to participate Not applicable. Competing interests The authors declare no competing interests. Footnotes Publisher’s Note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. References Associated Data Data Availability Statement No datasets were generated or analysed during the current study.