Cancer Type, Stage and Prognosis Assessment from Pathology Reports using LLMs
Rachit Saluja, Jacob Rosenthal, Yoav Artzi, David J. Pisapia, Benjamin L. Liechty, Mert R. Sabuncu · 2025
Read Cancer Type, Stage and Prognosis Assessment from Pathology Reports using LLMs by Rachit Saluja, Jacob Rosenthal, Yoav Artzi, David J. Pisapia, Benjamin L. Liechty, Mert R. Sabuncu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-01194.txt.
Introduction
Pathology reports hold the facts that decide cancer care, but they are written as free text. Rachit Saluja and co-authors test whether large language models can read them reliably. The tasks are naming the cancer type, assigning the standard stage, and judging the outlook.
The team tries several leading models, including the GPT family, Mistral, and the open Llama models, with no task-specific training at first. Based on the results, they fine-tune two models on report data. The tuned models beat the others at all three tasks in fresh tests. The study measures a skill that earlier work had left unquantified. It also shows that modest, focused training can turn a general model into a strong reader of pathology text. Such tools could speed up registry work and case review.
Catalog record
| CREATOR | Rachit Saluja, Jacob Rosenthal, Yoav Artzi, David J. Pisapia, Benjamin L. Liechty, Mert R. Sabuncu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 53 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-01194
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Cancer Type, Stage and Prognosis Assessment from Pathology Reports using LLMs Rachit Saluja, Jacob Rosenthal, Yoav Artzi, David J. Pisapia, Benjamin L. Liechty, Mert R. Sabuncu Abstract Large Language Models (LLMs) have shown significant promise across various natural language processing tasks. However, their application in the field of pathology, particularly for extracting meaningful insights from unstructured medical texts such as pathology reports, remains underexplored and not well quantified. In this project, we leverage state-of-the-art language models, including the GPT family, Mistral models, and the open-source Llama models, to evaluate their performance in comprehensively analyzing pathology reports. Specifically, we assess their performance in cancer type identification, AJCC stage determination, and prognosis assessment, encompassing both information extraction and higher-order reasoning tasks. Based on a detailed analysis of their performance metrics in a zero-shot setting, we developed two instruction-tuned models: Path-llama3.1-8B and Path-GPT-4o-mini-FT . These models demonstrated superior performance in zero-shot cancer type identification, staging, and prognosis assessment compared to the other models evaluated. 1 Introduction Cancer remains one of the leading causes of death globally [ 1 ] . The diagnosis and prognosis of cancer require the collaboration of numerous healthcare professionals, who invest considerable time and expertise to optimize patient outcomes. Despite this effort, current practices lack a cohesive, data-driven approach that could significantly enhance diagnosis and prognosis by leveraging the vast amounts of clinical information available in unstructured text within the medical record, such as pathology reports. Due to time and budget constraints, healthcare professionals are often unable to analyze and extract meaningful insights from the millions of pathology reports generated, which limits their ability to conduct comprehensive analyses for improving patient care. Pathology reports, in particular, are complex and unstructured medical documents characterized by inconsistent formatting and a high degree of variability [ 2 , 3 ] . These reports often include extensive sections of information that are clinically irrelevant, making them less efficient for clinicians [ 3 ] . However, these reports also contain carefully curated information from highly trained domain-expert human pathologists in the form of diagnoses and descriptions of morphology. Extracting relevant data from pathology reports can provide valuable insights, revealing disease patterns, treatment outcomes, and overall patient health [ 4 ] . Utilizing this information can lead to more accurate prognosis predictions and the creation of personalized treatment plans [ 5 , 6 ] . Beyond clinical use, this data is also essential for researchers constructing large-scale datasets and conducting medical studies. Recent studies have demonstrated that large language models (LLMs) exhibit exceptional capabilities in leveraging their vast knowledge for tasks such as information extraction and certain forms of reasoning [ 7 , 8 ] . In the clinical domain, LLMs have shown remarkable potential, excelling at extracting data from medical texts without requiring extensive pretraining [ 9 ] . They have also proven highly effective in tasks like diagnosing complex medical cases [ 10 , 11 , 12 ] , mining radiology reports for key insights [ 13 , 14 , 15 , 16 ] , and extracting information from electronic health records [ 17 , 18 , 19 ] . Despite these advances, the application of LLMs in natural language processing (NLP) tasks specific to pathology remains underexplored. A recent study employed a BERT-based model [ 20 ] for named entity recognition, focusing on extracting diagnostic elements from breast pathology reports. However, this approach was limited to a single specimen class and lacked generalizability to other organ systems and disease processes [ 21 ] . More recently, large-scale LLMs like GPT have shown superior performance compared to supervised-trained models in various information extraction tasks [ 22 ] . Notably, GPT-4 was found to outperform supervised models in a zero-shot inference setting, without requiring additional training, although this study was also limited to breast cancer pathology reports. In another study [ 23 ] , LLMs were used to estimate the TNM stage of cancers from pathology reports, a critical task for determining prognosis and guiding treatment decisions. However, cancer staging is not always systematically included in pathology reports, and previous methods for extracting TNM attributes relied on regular expressions or traditional NLP techniques. This study introduced a dedicated TNM staging LLM specifically designed for this task. Despite its utility, it does not provide comprehensive information about the anatomic extent of the cancer beyond tumor size, nodal involvement, and metastasis. As a result, relying solely on TNM staging may not fully capture the complexity of disease progression. Moreover, these models are typically designed to handle a single task. Although they can be deployed locally, this setup may become expensive, as separate models are required to determine the tumor, nodes, and metastasis stages, respectively. An evaluation of the same task using open-source LLMs, combined with empirical tweaks to prompting techniques, assessed their performance in TNM staging [ 24 ] . However, a key limitation of these studies is that they tested their methodologies on extremely large models that require multiple GPUs with substantial VRAM, which is impractical for large-scale data mining, especially in resource-constrained settings. Furthermore, token generation with these large models is notably slow, further hindering their widespread application. To address the issues mentioned above, we first evaluate the capabilities of both large-scale closed-source LLMs and open-source LLMs in performing information extraction tasks, such as identifying cancer types. Next, we assess their ability to holistically deduce the severity of cancer from pathology reports and predict the AJCC cancer stage. This staging system provides a more comprehensive assessment than the TNM stage, as it focuses beyond the anatomic extent of the cancer. Finally, we benchmark the performance of these LLMs in predicting patient prognosis from pathology reports alone, as a measure of performance on a complex reasoning task which requires both concrete information extraction and high-level reasoning. After evaluating the performance of existing models, we developed a methodology to instruction-tune two new models that perform equally well, if not better, than both large-scale closed-source and open-source LLMs across all three tasks. The primary motivation for instruction tuning was to create a smaller, more efficient model capable of delivering high performance while being deployable in resource-constrained environments. We instruction-tuned GPT-4o-mini using a black-box fine-tuning approach provided by the OpenAI Platform, and Llama3.1-instruct-8B, which can be trained in an open-source environment. Notably, the Llama3.1-instruct-8B model, which we refer to as Path-llama3.1-8B , is a low-resource LLM capable of handling all three tasks efficiently and can be deployed in resource-constrained settings. The key contributions of our study are as follows: 1. We conduct a comprehensive evaluation of both open-source (Llama3-Instruct 8B and Llama3-Instruct 70B) [ 25 ] and closed-source LLMs (GPT-4o, GPT-4o-mini, Mistral-Large, Mistral-Medium) [ 26 , 27 ] in zero-shot settings, focusing on information extraction tasks such as cancer type identification, as well as reasoning tasks like estimating the AJCC stage and predicting patient prognosis from pathology reports. 2. To enhance the reliability and accuracy of predictions, we developed our own instruction-tuned models that perform as well as, or better than, existing models. Notably, one of these is an open-source model that can deliver accurate predictions even in low-resource environments. 3. We open-source all code and trained models, making them easily deployable. Additionally, we provide the code required