A Semi-Automated Annotation Workflow for Paediatric Histopathology Reports Using Small Language Models

Avish Vijayaraghavan, Jaskaran Singh Kawatra, Sebin Sabu, Jonny Sheldon, Will Poulett, Alex Eze, Daniel Key, John Booth, Shiren Patel, Jonny Pearson, Dan Schofield, Jonathan Hope, Pavithra Rajendran, Neil Sebire · 2026

Read A Semi-Automated Annotation Workflow for Paediatric Histopathology Reports Using Small Language Models by Avish Vijayaraghavan, Jaskaran Singh Kawatra, Sebin Sabu, Jonny Sheldon, Will Poulett, Alex Eze, Daniel Key, John Booth, Shiren Patel, Jonny Pearson, Dan Schofield, Jonathan Hope, Pavithra Rajendran, Neil Sebire (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-04168.txt.

Introduction

Avish Vijayaraghavan and team built a semi-automated workflow using small language models to pull facts from paediatric histopathology reports in patient records. Large models need heavy local compute, and sending sensitive clinical data to cloud services raises privacy worries even when deidentified. On paediatric renal biopsy reports from Great Ormond Street Hospital, with 400 hand-annotated as gold standard, Gemma 2 2B hit 84.3 percent accuracy, beating spaCy, BioBERT-SQuAD, RoBERTa-SQuAD, and GLiNER. Entity guidelines added 7 to 19 percent over a zero-shot baseline, and few-shot examples added 6 to 38 percent.

Catalog record

CREATORAvish Vijayaraghavan, Jaskaran Singh Kawatra, Sebin Sabu, Jonny Sheldon, Will Poulett, Alex Eze, Daniel Key, John Booth, Shiren Patel, Jonny Pearson, Dan Schofield, Jonathan Hope, Pavithra Rajendran, Neil Sebire
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE70836
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.04168

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

A Semi-Automated Annotation Workflow for Paediatric Histopathology Reports Using Small Language Models Avish Vijayaraghavan, Jaskaran Singh Kawatra, Sebin Sabu, Jonny Sheldon, Will Poulett, Alex Eze, Daniel Key, John Booth, Shiren Patel, Jonny Pearson, Dan Schofield, Jonathan Hope, Pavithra Rajendran, Neil Sebire Abstract Electronic Patient Record (EPR) systems contain valuable clinical information, but much of it is trapped in unstructured text, limiting its use for research and decision-making. Large language models can extract such information but require substantial computational resources to run locally, and sending sensitive clinical data to cloud-based services, even when deidentified, raises significant patient privacy concerns. In this study, we develop a resource-efficient semi-automated annotation workflow using small language models (SLMs) to extract structured information from unstructured EPR data, focusing on paediatric histopathology reports. As a proof-of-concept, we apply the workflow to paediatric renal biopsy reports, a domain chosen for its constrained diagnostic scope and well-defined underlying biology. We develop the workflow iteratively with clinical oversight across three meetings, manually annotating 400 reports from a dataset of 2,111 at Great Ormond Street Hospital as a gold standard, while developing an automated information extraction approach using SLMs. We frame extraction as a Question-Answering task grounded by clinician-guided entity guidelines and few-shot examples, evaluating five instruction-tuned SLMs with a disagreement modelling framework to prioritise reports for clinical review. Gemma 2 2B achieves the highest accuracy at 84.3%, outperforming off-the-shelf models including spaCy (74.3%), BioBERT-SQuAD (62.3%), RoBERTa-SQuAD (59.7%), and GLiNER (60.2%). Entity guidelines improved performance by 7-19% over the zero-shot baseline, and few-shot examples by 6-38%, though their benefits do not compound when combined. These results demonstrate that SLMs can extract structured information from specialised clinical domains on CPU-only infrastructure with minimal clinician involvement. Our code is available at https://github.com/gosh-dre/nlp_renal_biopsy . keywords small language models, information extraction, named entity recognition, question-answering, medical reports, paediatrics, renal biopsy 1 Introduction Data-driven clinical decision-making enables healthcare professionals to deliver personalised care by drawing on the diverse digitised information stored within Electronic Patient Record (EPR) systems Lyu 2025 . However, much of this pertinent information remains locked within unstructured data. Without conversion to a standardised structured format, it cannot be used easily for quantitative analysis or secondary research. Histopathology, the microscopic examination of tissue samples, is fundamental to disease diagnosis and treatment planning. A simplified process is shown in Figure 1 : a biopsy is performed, tissue is stained, pathologists examine the slides and document their findings in detailed reports that inform clinical decision-making. However, these reports exemplify the challenge of unstructured clinical text, combining clinical history, detailed microscopic and macroscopic observations, and diagnostic conclusions in free-form narratives that make it difficult to aggregate findings. Automating the annotation of histopathology reports could enhance patient care in multiple ways: standardising the documentation process, reducing clinician workload, enabling scalable insights for research, and supporting quality audits. Figure 1 : The general histopathology workflow. Our focus is on extracting key text spans from the clinician’s report that are relevant for diagnosis. A fundamental challenge in working with histopathology reports lies in the nature of medical free text. These reports contain domain-specific terminology, abbreviations, informal language, and occasionally noisy or misspelled text. A comprehensive study by the King’s Fund Trust Bean et al. 2023 , analysing over one million clinical documents, confirmed that a significant portion of clinical communication occurs in free text format, highlighting the importance of addressing this domain-specific language variability. Additionally, these reports often include ambiguities and shorthand notations that, while intuitive for human clinicians, pose significant challenges for traditional natural language processing (NLP) models. Dictionary or ontology-based methods (e.g., SNOMED-CT) work well for simple diagnosis prediction, but our focus is on capturing detailed contextual information for accurate report summarisation. Token-based named entity recognition (NER) methods can achieve high performance on clearly-defined entities, but struggle with variable phrasing, longer and often discontinuous text spans, and implicit entity relations. Additionally, histopathology reports often contain meta-annotations like negations (e.g., “there is no evidence of rejection”) and uncertainties (e.g., “there are features suggestive of chronic tubulointerstitial nephritis”). Standard medical NER models like medspaCy Eyre et al. 2022 do not support meta-annotations, and while MedCAT Kraljevic et al. 2021 does, neither adequately captures the variability of entity descriptions found in histopathology reports. The emergence of large language models (LLMs) presents an opportunity to address these challenges. Unlike token-classification-based methods, autoregressive LLMs perform well on language tasks that do not require precise localisation or categorisation of entities, making them better suited to capturing variable and context-dependent information. LLMs also offer greater task flexibility because their inputs (“prompts”) can be customised in natural language and easily adapted to specific biomedical domains. This flexibility makes them straightforward to integrate into existing clinical workflows. However, most approaches require substantial computational resources and external API access, making them unsuitable for sensitive paediatric data that must remain on local infrastructure. In this study, we propose an automated workflow for annotating unstructured text within medical reports using a Question-Answering (QA) framework. We describe its development using real-world data from Great Ormond Street Hospital, a context that introduces several constraints: (1) the sensitive nature of the data restricts access to secure on-premises servers, (2) computational infrastructure is limited to CPU processing only, (3) clinician time for annotation tasks is scarce, and (4) large volumes of data remain unlabelled. These challenges are compounded by the inherent complexity of paediatric medicine, where patient presentations vary considerably, and by the hospital’s national referral pathways, which bring patients from diverse geographic and environmental backgrounds that may influence disease presentation. This is particularly relevant given that few language models have been trained on or applied to paediatric text Jabarulla et al. 2024 ; Barile et al. 2024 ; Andrew et al. 2024 ; Yang et al. 2024 , despite - or perhaps due to - the known high variability in paediatric clinical reporting. In order to overcome these constraints, our approach is focussed on using LLMs in the order of 1B to 5B parameters, also referred to as Small Language Models (SLMs). To address the scarcity of labelled data, our workflow includes a Streamlit-based annotation interface for efficient labelling. A subsequent disagreement modelling stage prioritises data scientist-annotated reports for secondary clinical review. We have investigated different quantised generative models and evaluated their performance using modified prompts for in-context learning. This approach offers a pragmatic balance between computational efficiency and the ability to capture complex medical narratives, alongside easier interfacing with non-computational practitioners. We demonstrate our approach on renal biopsies, a relatively constrained and well-understood domain within histopathology. Renal biopsies involve a narrower range of pathological triggers and diagnostic outcomes, making them ideal for testing the robustness of our proposed machine learning pipeline alongside human evaluation. Since histopathology as a whole is a well-established diagnostic modality, renal biopsy reports offer a reliable and informative starting point, acting as a proof-of-concept for more complex and less standardised pathology subfields. 2 Related Work 2.1 LLMs for Biomedical NER into QA The emergence of word embeddings marked a shift towards deep learning-based biomedical NER. scispaCy Neumann et al. 2019 , for instance, integrated UMLS linkages and offered pre-trained embeddings tailored to biomedical texts. Biomedical takes on the transformer architecture BERT (e.g., BioBERT Lee et al. 2019 ,

Read the complete text