Natural language processing models for patient-centered summaries of prostatectomy pathology reports Authors: Parsa Iranmahboub, Priya Dave, Michael Hung, Joe Pelt, Hasan O. Ali, Kyle Blum, Ashwin Ramaswamy, B. Malik Wahba, Amalia Korniyenko, Aaron Huang, Juan Angulo-Lozano, Helen Xu, Evan Suzman, Lina Posada Calderon, Leonardo D. Borregales, Douglas S. Scherr Published: 2026 ABSTRACT Many patients now view their radical prostatectomy (RP) pathology before provider discussion, increasing anxiety and administrative burden. With expanding utilization of AI-assisted medical workflows, it is important to implement strategies to improve the interpretation of patient-facing information involving complex medical language. We compare a rules-based natural language processing (NLP) model and a large language model (LLM) using zero-shot prompting to identify the optimal framework for developing accurate, patient-facing RP pathology summaries. Models were assessed for accuracy in extracting key pathology features, calculating recurrence-free probabilities at varying intervals, and providing clinical recommendations from pathology reports at a single institution. Error-free summaries were generated in 92% of rules-based NLP and 97% of LLM reports (p = 0.18). In an external test set of differently formatted RP pathology reports from a separate institution, the LLM maintained high accuracy without additional training, while the rules-based NLP model achieved similarly high accuracy after minimal refinement. These findings suggest that both approaches can effectively support patient-facing pathology summaries, allowing practices and hospital systems to adopt the framework best suited to their technical resources, financial considerations, governance infrastructure, and institutional priorities.Supplementary InformationThe online version contains supplementary material available at 10.1038/s41598-026-55613-7. Introduction With the expanding use of electronic patient-provider communication and increasing patient access to health information through the electronic medical record (EMR), there is a growing need to facilitate timely and effective delivery of postprocedural clinical information and recommendations. The 21st Century Cures Act, implemented in April 2021, mandates immediate patient access to clinical notes, radiology reports, laboratory results, and surgical pathology reports via the EMR.1 Consequently, patients often view complex medical results before healthcare providers have an opportunity to interpret findings, contextualize prognostic implications, and provide appropriate counseling.2,3 Although timely access may enhance patient engagement, the release of abnormal or complicated results without clinical interpretation can increase patient anxiety.4 Medical jargon coupled with limited interpretive guidance may further contribute to poor understanding, misinterpretation, and unnecessary distress.5–7 Immediate result release also places additional demands on providers and health systems. Patients often expect rapid outreach following abnormal findings. In one survey, approximately 75% of patients felt clinicians should contact them within 24 hours of abnormal result release, compared with only 9% of clinicians who agreed.8 The resultant increase in patient portal messaging following the Cures Act further exacerbates clinical workflow burden.3,9 Natural language processing (NLP) models offer a potential strategy to bridge this communication gap by analyzing patterns in human language to extract key features from medical reports and generate patient-facing summaries. Large language models (LLMs), a generative NLP tool capable of producing human-like text, may be particularly well-suited for applications in prostate cancer. As postoperative management following radical prostatectomy often follows physician-directed algorithmic pathways once critical pathologic features and recurrence probabilities are identified, this setting is amenable to structured summarization and automated recommendation generation. While LLMs can be adapted through fine-tuning on task-specific datasets, this approach requires curated training data and substantial development resources.10 Alternatively, zero-shot prompting relies on instructional input without additional training but may be more susceptible to generating factually inconsistent or overly confident outputs, raising concerns about potential patient harm.11 A complementary approach is the use of rules-based NLP frameworks, which identify predefined pathology features through structured pattern matching and generate standardized responses using hard-coded logic, thereby limiting uncontrolled generative behavior. However, rules-based approaches can be technically demanding to develop and remain vulnerable to variation in predefined patterns. As NLP-based tools are increasingly explored to improve efficiency and patient communication–and have demonstrated performance comparable to or exceeding that of human experts in certain clinical text summarizations12–identifying methods that reliably produce accurate, patient-appropriate interpretations of urologic pathology reports suitable for EMR integration remains a critical challenge. In this study, we evaluate various NLP methodologies for generating patient-centered summaries of radical prostatectomy pathology reports that clearly convey key findings, recurrence risk estimates, and concise clinical recommendations. Results 399 pathology reports from patients undergoing radical prostatectomy at Weill Cornell Medicine between January 2017 to December 2021 met inclusion criteria. The reports were split into a validation set (40) and a test set (100), with the remaining reports available for training. The distribution of reports by stage is provided (Table 1). Table 1Pathological stage distribution of training, validation, and test sets. pNx = regional lymph nodes not assessed.Pathological stageTraining set(n = 259)Validation set(n = 40)Test set(n = 100)pT2135 (52%)21 (53%)52 (52%)pT3a91 (35%)14 (35%)35 (35%)pT3b33 (13%)5 (13%)13 (13%)pN110 (4%)1 (2.5%)4 (4%)pN0241 (93%)38 (95%)93 (93%)pNx8 (3%)1 (2.5%)3 (3%) Table 1Pathological stage distribution of training, validation, and test sets. pNx = regional lymph nodes not assessed.Pathological stageTraining set(n = 259)Validation set(n = 40)Test set(n = 100)pT2135 (52%)21 (53%)52 (52%)pT3a91 (35%)14 (35%)35 (35%)pT3b33 (13%)5 (13%)13 (13%)pN110 (4%)1 (2.5%)4 (4%)pN0241 (93%)38 (95%)93 (93%)pNx8 (3%)1 (2.5%)3 (3%) Pathological stage distribution of training, validation, and test sets. pNx = regional lymph nodes not assessed. Using the final configuration, the rule-based spaCy model achieved a mean accuracy of 99.2% (95% CI, 98.7–99.7%) for extracting pathology features across the test set. Performance across individual pathology subcategories is summarized in Table 2. The spaCy model correctly predicted recurrence-free probabilities at 2, 5, and 10 years in 99% of cases and provided an appropriate clinical recommendation for all reports. For the LLM, the GPT-5 model with zero-shot prompting achieved a mean accuracy of 99.7% (95% CI, 99.4–100%; p = 0.10) for pathology extraction across the test set (Table 2). The GPT-5 model correctly predicted recurrence-free probabilities in 98% of cases and also provided an appropriate clinical recommendation for all reports. Comparing overall performance, the rules-based spaCy model generated error-free outputs for 92/100 test reports, whereas the GPT-5 model with zero-shot prompting produced error-free outputs for 97/100 test reports. This difference was not statistically significant (p = 0.18) (Fig. 1a). Table 2Test set analysis for rules-based NLP (spaCy) model and LLM (GPT-5 with zero-shot prompting) with Weill Cornell pathology reports. Overall extraction accuracy is defined by average proportion of variables extracted correctly. CI = 95% confidence interval.spaCy modelGPT-5(n = 100)(n = 100)Overall Extraction Accuracy, mean 99.2%(CI 98.7–99.7)99.7%(CI 99.4–100) Correct Pathological Stage, n (%)100 (100%)100 (100%) Correct Gleason Score and Grade, n (%)100 (100%)100 (100%) Correct Marginal Status, n (%)99 (99%)98 (98%) Correct Extraprostatic Extension, n (%)99 (99%)100 (100%) Correct Seminal Vesicle Invasion, n (%)100 (100%)100 (100%) Correct Bladder Neck Invasion, n (%)100 (100%)99 (99%) Correct Perineural Invasion, n (%)96 (96%)100 (100%) Correct Angiolymphatic Invasion, n (%)98 (98%)100 (100%) Correct Pelvic Lymph Nodes Examined, n (%)100 (100%)100 (100%) Correct Pelvic Lymph Nodes Involved, n (%)100 (100%)100 (100%)Correct Recurrence-Free Probability, n (%)99 (99%)98 (98%)Correct Clinical Recommendation, n (%)100 (100%)100 (100%)Complete Error-Free Summaries, n (%)92 (92%)97 (97%) Table 2Test set analysis for rules-based NLP (spaCy) model and LLM (GPT-5 with zero-shot prompting) with Weill Cornell pathology reports. Overall extraction accuracy is defined by average proportion of variables extracted correctly. CI = 95% confidence interval.spaCy modelGPT-5(n = 100)(n = 100)Overall Extraction Accuracy, mean 99.2%(CI 98.7–99.7)99.7%(CI 99.4–100) Correct Pathological Stage, n (%)100 (100%)100 (100%) Correct Gleason Score and Grade, n (%)100 (100%)100 (100%) Correct Marginal Status, n (%)99 (99%)98 (98%) Correct Extraprostatic Extension, n (%)99 (99%)100 (100%) Correct Seminal Vesicle Invasion, n (%)100 (100%)100 (100%) Correct Bladder Neck Invasion, n (%)100 (100%)99 (99%) Correct Perineural Invasion, n (%)96 (96%)100 (100%) Correct Angiolymphatic Invasion, n (%)98 (98%)100 (100%) Correct Pelvic Lymph Nodes Examined, n (%)100 (100%)100 (100%) Correct Pelvic Lymph Nodes Involved, n (%)100 (100%)100 (100%)Correct Recurrence-Free Probability, n (%)99 (99%)98 (98%)Correct Clinical Recommendation, n (%)100 (100%)100 (100%)Complete Error-Free Summaries, n (%)92 (92%)97 (97%) Test set analysis for rules-based NLP (spaCy) model and LLM (GPT-5 with zero-shot prompting) with Weill Cornell pathology reports. Overall extraction accuracy is defined by average proportion of variables extracted correctly. CI = 95% confidence interval. 99.2% (CI 98.7–99.7) 99.7% (CI 99.4–100) Fig. 1Contingency tables comparing error-free counts of pathology report summaries between the rules-based NLP (spaCy) and LLM (GPT-5 with zero-shot prompting). (a) Pathology reports from Weill Cornell Medicine. (b) Pathology reports from Stony Brook University Hospital, with the spaCy configuration prior to refinement with external training set. (c) Pathology reports from Stony Brook University Hospital, with the spaCy configuration after refinement with external training set. Contingency tables comparing error-free counts of pathology report summaries between the rules-based NLP (spaCy) and LLM (GPT-5 with zero-shot prompting). (a) Pathology reports from Weill Cornell Medicine. (b) Pathology reports from Stony Brook University Hospital, with the spaCy configuration prior to refinement with external training set. (c) Pathology reports from Stony Brook University Hospital, with the spaCy configuration after refinement with external training set. The average time to generate a model output for GPT-5 with zero-shot prompting was 95 seconds (IQR 55–120) (Supplementary Fig. 1), with an average token input of 2293 and an average token output of 6218. No prompt deviations or extraneous content were observed in the GPT-5 responses. To examine consistency between LLM responses for the same pathology report, 20 reports from the test set were randomly selected to be independently summarized again, one month after the initial evaluation. No differences in pathology extraction, recurrence-free probabilities, or clinical recommendations were observed between GPT-5 responses for the same pathology report. With prior GPT-4.1 configurations of the LLM, a limited analysis of the first 20 test reports revealed 0% accuracy in calculating the recurrence-free probabilities. This was true for both the zero-shot and fine-tuned methods (Supplementary Table 1). The average deviation between the GPT-4.1 configurations and MSKCC’s predicted recurrence-free probabilities are shown in Supplementary Table 2. External testing with Stony Brook University Hospital pathology reports revealed 100% accuracy for extracting pathology features with the GPT-5 zero-shot prompting model. This compared to an average accuracy of 89.8% (95% CI, 88.9–90.7%, p < 0.01 ) for spaCy before training with the external training set and 99.8% (99.4–100%), p = 1.0) after training (Table 3). Notably, the pre-trained spaCy model could extract the pathological tissue stage for only 4% of Stony Brook reports. Table 3Test set analysis for the LLM (GPT-5 with zero-shot prompting) and rules-based NLP (spaCy) with Stony Brook pathology reports. The spaCy configurations are divided by before and after refinement of the model with the external training set. Overall extraction accuracy is defined by average proportion of variables extracted correctly. CI = 95% confidence interval.GPT-5spaCy (before training)spaCy (after training)(n = 50)(n = 50)(n = 50)Overall Extraction Accuracy, mean 100%89.8%(CI 88.9–90.7)99.8%(CI 99.4–100) Correct Pathological Stage, n (%)50 (100%)2 (4%)50 (100%) Correct Gleason Score and Grade, n (%)50 (100%)50 (100%)50 (100%) Correct Marginal Status, n (%)50 (100%)50 (100%)50 (100%) Correct Extraprostatic Extension, n (%)50 (100%)50 (100%)50 (100%) Correct Seminal Vesicle Invasion, n (%)50 (100%)49 (98%)49 (98%) Correct Bladder Neck Invasion, n (%)50 (100%)50 (100%)50 (100%) Correct Perineural Invasion, n (%)50 (100%)50 (100%)50 (100%) Correct Angiolymphatic Invasion, n (%)50 (100%)48 (96%)50 (100%) Correct Pelvic Lymph Nodes Examined, n (%)50 (100%)50 (100%)50 (100%) Correct Pelvic Lymph Nodes Involved, n (%)50 (100%)50 (100%)50 (100%)Correct Recurrence-Free Probability, n (%)50 (100%)49 (98%)49 (98%)Correct Clinical Recommendation, n (%)50 (100%)50 (100%)50 (100%)Complete Error-Free Summaries, n (%)50 (100%)2 (4%)49 (98%) Table 3Test set analysis for the LLM (GPT-5 with zero-shot prompting) and rules-based NLP (spaCy) with Stony Brook pathology reports. The spaCy configurations are divided by before and after refinement of the model with the external training set. Overall extraction accuracy is defined by average proportion of variables extracted correctly. CI = 95% confidence interval.GPT-5spaCy (before training)spaCy (after training)(n = 50)(n = 50)(n = 50)Overall Extraction Accuracy, mean 100%89.8%(CI 88.9–90.7)99.8%(CI 99.4–100) Correct Pathological Stage, n (%)50 (100%)2 (4%)50 (100%) Correct Gleason Score and Grade, n (%)50 (100%)50 (100%)50 (100%) Correct Marginal Status, n (%)50 (100%)50 (100%)50 (100%) Correct Extraprostatic Extension, n (%)50 (100%)50 (100%)50 (100%) Correct Seminal Vesicle Invasion, n (%)50 (100%)49 (98%)49 (98%) Correct Bladder Neck Invasion, n (%)50 (100%)50 (100%)50 (100%) Correct Perineural Invasion, n (%)50 (100%)50 (100%)50 (100%) Correct Angiolymphatic Invasion, n (%)50 (100%)48 (96%)50 (100%) Correct Pelvic Lymph Nodes Examined, n (%)50 (100%)50 (100%)50 (100%) Correct Pelvic Lymph Nodes Involved, n (%)50 (100%)50 (100%)50 (100%)Correct Recurrence-Free Probability, n (%)50 (100%)49 (98%)49 (98%)Correct Clinical Recommendation, n (%)50 (100%)50 (100%)50 (100%)Complete Error-Free Summaries, n (%)50 (100%)2 (4%)49 (98%) Test set analysis for the LLM (GPT-5 with zero-shot prompting) and rules-based NLP (spaCy) with Stony Brook pathology reports. The spaCy configurations are divided by before and after refinement of the model with the external training set. Overall extraction accuracy is defined by average proportion of variables extracted correctly. CI = 95% confidence interval. 89.8% (CI 88.9–90.7) 99.8% (CI 99.4–100) For calculating recurrence-free probabilities with the Stony Brook test set, the GPT-5 model was correct for 100% of reports, compared to 98% with both the pre- and post-trained spaCy models. All 3 models provided the correct clinical recommendation for the entire external test set. Regarding overall performance with externally formatted pathology reports, the GPT-5 model generated error-free responses for 50/50 reports. This compared to 2/50 reports (p < 0.01) (Fig. 1b) for the pre-trained spaCy configuration and 49/50 reports (p = 1.0) (Fig. 1c) for the post-trained spaCy configuration. Discussion In this study, we demonstrate that multiple NLP frameworks can be successfully implemented to generate accurate and reliable patient-centered summaries of radical prostatectomy pathology reports. Importantly, we define ‘patient-centered’ not only as correct extraction and explanation of medical terminology but also as delivery of information that directly informs patient decision-making, such as recurrence risk, prognostics, and clear next steps in care. From an implementation perspective, these findings highlight the feasibility of automated, patient-facing interpretation tools into existing clinical workflows to support timely, accurate communication following pathology result release. Using a rules-based NLP approach with spaCy, we achieved high overall accuracy in pathology feature extraction, recurrence-free probability estimation, and generation of guideline-concordant clinical recommendations. Most extraction errors involved perineural and angiolymphatic invasion, features typically reported in less standardized language and weighted less critically into recurrence nomograms, resulting in minimal downstream clinical impact. The single error that meaningfully altered recurrence estimation stemmed from the misspelling of “identified” in describing extraprostatic extension, underscoring both the strengths and limitations of rules-based NLP. It performs reliably with semi-structured notes and ensures reproducibility with deterministic responses.14,15 However, it remains sensitive to misspellings, syntax discrepancies, and documentation quality.16 GPT-5 with zero-shot prompting similarly achieved very high accuracy across all three evaluation metrics. The few observed errors represented challenges in ambiguity and inference uncommon in typical reporting, such as incomplete interpretation of ‘abuts’ for marginal status and failure to infer bladder neck invasion from margin involvement when not explicitly stated. Notably, the strong performance of zero-shot prompting suggests that fine-tuning is not required to achieve clinically meaningful summaries. This is particularly relevant for health systems with limited data science infrastructure, as fine-tuning requires large and clean datasets to prevent incorrect data from corrupting the model.10 Additionally, if nomograms, clinical algorithms, or input variables change, a fine-tuned model will need to be tediously re-trained, an important consideration for long-term maintainability.17 Our findings align with prior work demonstrating that LLM models can achieve abstraction accuracy comparable to human reviewers for prostatectomy pathology reports. Kaufmann et al. reported a zero-shot LLM accuracy of 94.2%, which was non-inferior to human abstractors at 96.3%.18 The refined accuracy in our model (99.7%) likely reflects improvements in multi-step reasoning and instruction-following capabilities of newer foundation models. Importantly, we also observed meaningful performance differences across foundation models, with earlier-generation models (GPT-4.1) failing to reliably compute recurrence risk despite explicit prompting. These findings emphasize that model selection is not interchangeable for the development of a successful patient-centered summary, and clinical deployment requires rigorous, model-specific validation rather than reliance on general LLM capability claims. Concerns regarding hallucination and generation of plausible yet factually incorrect statements are frequently cited barriers to clinical adoption of LLMs, particularly in patient-facing applications.19,20 Reassuringly, our summaries with GPT-5 and zero-shot prompting were factually accurate and reproducible, adhering to the requested format without introducing misleading information or extraneous information (e.g. prostate volume). Instead, our success with zero-shot prompting emphasizes the importance of utilizing structured prompts that encourage stepwise output generation.21 Evaluation with an external test set revealed differences in performance across frameworks. GPT-5 with zero-shot prompting outperformed the rules-based spaCy model without additional training, with the latter frequently missing the pathological stage due to differences in the labeling of a key identifying word. However, after minimal refinement using only 3 specific Stony Brook reports, the rules-based spaCy model achieved similarly high performance. Although cross-institution compatibility may be less critical for many urology practices, pathology report formats can change within an institution over time. In that setting, LLM-based workflows may be more tolerant of formatting variability, whereas rules-based systems require validation and targeted pattern updates to maintain performance. Although GPT-5 with zero-shot prompting demonstrated stronger absolute performance, there was no statistical difference across frameworks, reinforcing that implementation decisions should be guided by technical resources, practice infrastructure, and institutional priorities. An LLM with zero-shot prompting avoids the technical need for coding or model training and is more accurate and adaptable without the need to tediously re-train in the setting of syntactical irregularities or periodic formatting changes. However, an LLM can introduce additional patient data safety and intellectual property concerns compared to a rules-based NLP (spaCy). A spaCy model provides greater end-to-end visibility, as the model is designed within the institution and can be entirely executed on a local, secure, and HIPAA-controlled server. In contrast, LLMs typically rely on third-party dependence, requiring a Business Associate Agreement for HIPAA-compliance and navigation of protected health information (PHI), as well as contractual assurance to mitigate data security risks, potential data leakage, and downstream reuse of prompts and model-generated outputs for future model training.22–25 LLM-based workflows also carry higher operational costs, involving additional per-input and -output token costs or provisioned monthly costs beyond the EMR integration expenses common to any NLP system.26 Finally, while an LLM may require sizeable time to generate a summary, the delay may be trivial if the model is embedded as an EMR message rather than used as a real-time tool. While this study significantly enhances understanding of NLP framework utilization to generate patient-centered pathology summaries, implementation-based limitations exist. Pathology reports at our institution are semi-structured with the inclusion of a synoptic diagnosis. Performance may vary in settings relying solely on unstructured reports without a synopsis. Notably for LLMs, the available foundation models may also evolve over time, as these models are updated or deprecated with technological advancement. Although patient age carries little weight in the MSKCC nomogram (i.e. the magnitude of the age coefficient is incredibly small) and recurrence-free estimates in this study were generated from individualized pathology vignettes incorporating specific pathologic features and preoperative PSA, future prospective studies should incorporate patient-specific age. Further, none of our evaluated workflows are currently suited for full autonomous clinical use. A ‘human in the loop’ remained essential to manually extract the de-identified elements of a pathology report, incorporate additional clinical variables such as preoperative PSA, and monitor quality control to ensure appropriate clinical communication. In the long term, we envision an EMR-integrated workflow in which a model-generated patient-facing summary is automatically released alongside the pathology report, serving not as a replacement for physician counseling, but to bridge the communication gap between report availability and the patient’s subsequent follow-up visit. Yet, meaningful integration into the EMR, particularly with automated data ingestion, administrative oversight, and clear escalation pathways, is necessary before these tools can be safely deployed at scale without a human in the loop. Nonetheless, this proof-of-concept work establishes a critical foundation by informing design requirements and feasibility for implementation. Future work will expand external generalizability across pathology reports from multiple institutions and prospectively evaluate model-generated reports in clinical practice, including their impact on patient comprehension, anxiety, and satisfaction as well as provider efficiency. In summary, both rules-based NLP and LLM with zero-shot prompting are viable frameworks for generating patient-centered summaries of prostatectomy pathology reports and delivering clinically relevant and actionable information. Successful implementation is dependent upon balancing accuracy, adaptability, and data ownership. Practices and hospital systems should select the framework that best aligns with their technical capabilities, financial considerations, governance infrastructure, and institutional priorities. Methods The study was reviewed by Weill Cornell Medicine’s Institutional Review Board (IRB) and determined to be IRB exempt. Radical prostatectomy pathology reports from January 2017 to December 2021 at Weill Cornell Medicine were extracted from the EMR. The reports were de-identified to include only the ‘synoptic diagnosis’ and ‘diagnosis’ sections, along with the patient’s preoperative PSA. Reports were excluded if they involved cystoprostatectomy, documented pre-surgical treatment (e.g. radiation or hormonal therapy), or contained conflicting pathology information (e.g. mention of pN0 when an involved node had been identified). Eligible reports were stratified by pathological staging based on tissue examination (pT) and regional node involvement (pN). Stratified sampling was then used to divide the reports into training, validation, and test sets, representing 65%, 10%, and 25% of the total cohort, respectively (Fig. 2). Fig. 2Inclusion criteria schematic. Among the eligible radical prostatectomy pathology reports, proportional stratified sampling was conducted to separate the reports into a training, validation, and test set by pathological stage. Inclusion criteria schematic. Among the eligible radical prostatectomy pathology reports, proportional stratified sampling was conducted to separate the reports into a training, validation, and test set by pathological stage. Model inputs consisted of the de-identified diagnostic portions of the pathology report, along with sentences stating the patient’s age and preoperative PSA. The preoperative PSA was specific to each patient, but every patient was treated as 65 years old to further de-identify the model input. To generate patient-facing summaries, the model outputs were designed with three components: a concise summary and explanation of notable pathology features; an estimate of recurrence-free probabilities at 2, 5, and 10 years; and a clinical recommendation outlining next steps in care. The model was tasked with extracting 10 pathology variables: pathological stage, Gleason score (and Grade Group), extraprostatic extension, seminal vesicle invasion, surgical margin status, bladder neck invasion, perineural invasion, angiolymphatic invasion, number of nodes examined, and number of nodes involved. Extraction accuracy was defined as the average percentage of variables extracted correctly per report. Features not mentioned in a report were treated as negative, whereas those explicitly labeled as “indeterminate” were distinctly categorized as such. Recurrence-free probabilities for each report were calculated by referencing the Memorial Sloan Kettering Cancer Center’s (MSKCC) ‘Post-Radical Prostatectomy’ nomogram.13 Probabilities were rounded to the nearest whole number, and accuracy required a match within 1% across all three time intervals (2, 5, and 10 years). Clinical recommendation was binary: patients with a 2-year recurrence-free probability below 80% or with pathological pelvic lymph node metastasis (pN1) received one variation, while all other patients received an alternative message. Mean accuracies between different models were compared with paired t-test. For binary comparison of error-free outputs between different models, we used an exact McNemar’s test. Example model-generated summaries are shown in Supplementary Fig. 2. Rules-based NLP design The rules-based NLP was designed using spaCy, an open-source Python library. For initial training, five pathology reports were randomly selected to cover diverse scenarios: pT2 stage with negative margins, pT3b stage with positive margins, pN1 disease, regional nodes not assessed (pNx), and extraprostatic extension not assessed. At least one report included perineural invasion and angiolymphatic invasion to ensure model exposure to these scenarios. Syntax from these cases was used to refine parsing and entity-recognition patterns. The preliminary spaCy model was evaluated against the validation set, with iterative adjustments applied before final examination with the test set. LLM design LLMs were developed using OpenAI’s generative pre-trained transformer (GPT) foundation models. For the primary comparison study, we used a GPT-5 model (released August 2025) with zero-shot prompting. The model was provided with detailed and stepwise instructions that specified all required pathology features to be extracted, suggested phrasing, equations and updated variables for recurrence-free probability calculations, and clinical recommendation criteria. Model hyperparameters were optimized using the validation set. The final GPT-5 configuration was evaluated on the test set using a medium ‘reasoning effort’ setting to reduce reasoning token use without compromising accuracy. Prior LLM iterations used OpenAI’s ‘GPT-4.1-2025-04-14’ model (released April 2025), evaluated with both zero-shot prompting and fine-tuned configurations. The zero-shot version employed the same explicit prompt used for GPT-5. For the fine-tuned version, we supplemented this prompt with 50 randomly stratified pathology reports from the training set, each paired with the ideal provider-designed output format. Both GPT-4.1 models were evaluated on the test set with a temperature of 0.20 to minimize response randomness. External validation To evaluate model performance on pathology reports formatted differently from those used for initial development, we obtained 53 de-identified radical prostatectomy reports from Stony Brook University Hospital dated between January 2023 and December 2024. Fifty reports were selected to create the external test set. The remaining three reports (representing pT2, pT3a, and pT3b with varying margin statuses and at least one case of regional nodes not assessed) were excluded from the test set and instead used as an external training set. The LLM was evaluated on the external test set without any prior exposure to the external training reports. In contrast, the rules-based spaCy model was assessed on the external test set both before and after refinement using the external training set. Supplementary Information Below is the link to the electronic supplementary material. Supplementary Material 1 Supplementary Material 1