Knowledge Elicitation with Large Language Models for Interpretable Cancer Stage Identification from Pathology Reports

Yeawon Lee, Christopher C. Yang, Chia-Hsuan Chang, Grace Lu-Yao · 2025

Read Knowledge Elicitation with Large Language Models for Interpretable Cancer Stage Identification from Pathology Reports by Yeawon Lee, Christopher C. Yang, Chia-Hsuan Chang, Grace Lu-Yao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-01052.txt.

Introduction

Yeawon Lee, Christopher Yang, Chia-Hsuan Chang, and Grace Lu-Yao use large language models to read cancer stage information out of free text pathology reports. Staging drives prognosis and treatment, but extracting the pathologic TNM stage from unstructured reports usually needs large hand labeled data sets, which limits how far older methods scale.

They test two knowledge elicitation methods. One prompts the model repeatedly to induce staging rules from unlabeled reports, keeping them in long term memory. The other extracts rules from guidelines in one pass, then applies them without repeated retrieval. On breast cancer reports from the TCGA collection, the memory based method beat the retrieval based one under zero shot chain of thought prompting, and both aim at interpretable, rule based staging.

Catalog record

CREATORYeawon Lee, Christopher C. Yang, Chia-Hsuan Chang, Grace Lu-Yao
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE61 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-01052

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Knowledge Elicitation with Large Language Models for Interpretable Cancer Stage Identification from Pathology Reports Yeawon Lee, Christopher C. Yang, Chia-Hsuan Chang, Grace Lu-Yao Abstract Cancer staging is critical for patient prognosis and treatment planning, yet extracting pathologic TNM staging from unstructured pathology reports poses a persistent challenge. Existing natural language processing (NLP) and machine learning (ML) strategies often depend on large annotated datasets, limiting their scalability and adaptability. In this study, we introduce two Knowledge Elicitation methods designed to overcome these limitations by enabling large language models (LLMs) to induce and apply domain-specific rules for cancer staging. The first, Knowledge Elicitation with Long-Term Memory (KEwLTM), uses an iterative prompting strategy to derive staging rules directly from unannotated pathology reports, without requiring ground-truth labels. The second, Knowledge Elicitation with Retrieval-Augmented Generation (KEwRAG), employs a variation of RAG where rules are pre-extracted from relevant guidelines in a single step and then applied, enhancing interpretability and avoiding repeated retrieval overhead. We leverage the ability of LLMs to apply broad knowledge learned during pre-training to new tasks. Using breast cancer pathology reports from the TCGA dataset, we evaluate their performance in identifying T and N stages, comparing them against various baseline approaches on two open-source LLMs. Our results indicate that KEwLTM outperforms KEwRAG when Zero-Shot Chain-of-Thought (ZSCOT) inference is effective, whereas KEwRAG achieves better performance when ZSCOT inference is less effective. Both methods offer transparent, interpretable interfaces by making the induced rules explicit. These findings highlight the promise of our Knowledge Elicitation methods as scalable, high-performing solutions for automated cancer staging with enhanced interpretability, particularly in clinical settings with limited annotated data. keywords Large Language Models, Cancer Stage, Prompting, Knowledge Elicitation, Pathology Reports 1 Introduction Cancer staging is critical for determining the prognosis, treatment plan, and overall clinical management for cancer patients. The American Joint Committee on Cancer (AJCC) TNM classification system is widely used to stage cancer based on three key factors: Tumor size (T), regional lymph Node involvement (N), and the presence or absence of distant Metastasis (M). Obtaining pathologic TNM staging information usually requires manually parsing and extracting the relevant details from pathology reports, which are typically provided in free-text format. This unstructured format makes it difficult for healthcare providers to quickly access stage information, creating a need for natural language processing (NLP) solutions. In recent years, NLP techniques Odisho et al. 2020 ; De Angeli et al. 2021 ; Gao et al. 2017 ; Gao et al. 2019 ; Wu et al. 2020 have been used to automate the extraction of cancer stage information from free-text pathology reports. However, traditional NLP, machine learning (ML), and deep learning methods typically require large, annotated datasets for training, which can be expensive and time-consuming to create. Furthermore, these methods often struggle to perform well when applied to reports from different hospitals or institutions due to variations in reporting styles and terminology without retraining or adaptation. Therefore, there is a critical need for a scalable solution for cancer stage classification that does not rely heavily on labor-intensive labeled datasets, especially in clinical settings. The emergence of large language models (LLMs) such as Llama, Mixtral, and GPT has introduced a new era in NLP. LLMs have demonstrated the ability to apply their knowledge across diverse tasks, often without requiring explicit training on each new dataset. This capability stems from their extensive pre-training on vast corpora. During this pre-training, LLMs can learn general medical knowledge, including the fundamental rules of cancer staging, from publicly available resources like medical textbooks and guidelines Vaswani et al. 2017 ; Carlini et al. 2023 . Previous studies Chang et al. 2024a ; Chang et al. 2024b have shown the effectiveness of using LLMs for cancer stage classification tasks through prompting. However, prompting approaches that rely solely on the LLM’s existing general knowledge may achieve sub-optimal performance when addressing context-specific clinical details. Actual patient pathology reports, which contain such details, are typically part of the electronic health record and protected under privacy regulations. Consequently, LLMs cannot access them during pre-training. This lack of exposure to real-world clinical narratives, with their inherent variability and specialized terminology, can limit the LLM’s ability to accurately apply its general knowledge in these specific contexts. To address these limitations, while building upon established techniques like iterative prompting and retrieval-augmented generation (RAG), we propose two Knowledge Elicitation approaches with specific adaptations for clinical utility: Knowledge Elicitation with Long-Term Memory (KEwLTM) and Knowledge Elicitation with Retrieval-Augmented Generation (KEwRAG). KEwLTM enables LLMs to derive domain-specific knowledge directly from a small number of unannotated pathology reports. A key aspect of KEwLTM is its label-free induction process. It does not rely on ground-truth labels or human annotations. Instead, the model iteratively induces and refines high-level staging rules from the content of the instance reports themselves, storing these rules in a persistent long-term memory. This approach is particularly valuable in clinical contexts where large annotated datasets are scarce or restricted. The explicit rules also enhance interpretability. In contrast, KEwRAG adapts the standard RAG framework by front-loading the rule extraction. Instead of retrieving and appending raw text chunks to the LLM for each query, KEwRAG first retrieves relevant information from an external source (e.g., clinical guidelines) once. It then prompts the LLM to synthesize these retrieved texts into a concise, structured set of rules. This stable set of rules is then used for subsequent inferences, eliminating repeated retrieval overhead and providing a more coherent, auditable knowledge base that clinicians can easily review and validate. Both methods are designed for practical integration in limited-data clinical settings, aiming to be usable locally and generate domain-specific knowledge in a compact, reusable, and interpretable form. Our findings suggest these adapted methods can improve cancer stage identification accuracy while providing clinicians with transparent insights into how predictions are made, thereby promoting effective human-AI collaboration in healthcare. 2 Related Work In recent years, natural language processing (NLP) and machine learning (ML) have been instrumental in developing automated systems for extracting cancer stage information from free-text pathology reports. Odisho et al. Odisho et al. 2020 utilized contextual token embeddings with logistic regression, AdaBoost, and random forests for pathologic stage classification. Angeli et al. De Angeli et al. 2021 applied active learning to select training samples and then trained a convolutional neural network for stage identification. Gao et al. Gao et al. 2017 ; Gao et al. 2019 developed a hierarchical network to learn representations from words to complete reports, demonstrating its effectiveness in tumor grade classification using SEER data. Wu et al. Wu et al. 2020 leveraged attention-based graph convolution networks, using multi-source knowledge graphs for improved TNM stage identification on TCGA data. However, the adaptability of these ML and deep learning models remains limited, particularly because they rely on labeled datasets, which are often expensive to produce and constrained in scope. This limitation makes it difficult for these models to adapt to data that significantly differ from their training datasets, such as the diverse formats used in pathology reports across different medical facilities and the varying rules for different cancer types. Foundation models, such as pre-trained language models, are designed to be more adaptable by training on vast amounts of unstructured data Vaswani et al. 2017 . This large-scale,

Read the complete text