RADIANT-PET: Reasoning-Augmented PET/CT Lesion Segmentation with Large Language Models and Reinforcement Learning

Jiasheng Wang, Tanun Jitwatcharakomol, Piyawadee Jongpradubgiat, Simeng Zhu · 2026

Read RADIANT-PET: Reasoning-Augmented PET/CT Lesion Segmentation with Large Language Models and Reinforcement Learning by Jiasheng Wang, Tanun Jitwatcharakomol, Piyawadee Jongpradubgiat, Simeng Zhu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-28392.txt.

Introduction

Jiasheng Wang, Tanun Jitwatcharakomol, Piyawadee Jongpradubgiat, and Simeng Zhu segment lesions in PET/CT scans, where healthy organs and imaging artifacts can glow like tumors and fool image-only models. Their RADIANT-PET system adds a reasoning layer that second-guesses each candidate spot before accepting it as a lesion.

Wang's team first segments generously, catching nearly everything suspicious, then converts each candidate into a structured text summary of intensity, shape, and anatomical context. A language model judges each candidate as true lesion or false positive, optionally reading the radiology report too. Reinforcement learning sharpens that judge, rewarding correct calls and correct anatomical placement. Across two cohorts the system beat strong image-only baselines, with the largest gains when reports were provided.

Catalog record

CREATORJiasheng Wang, Tanun Jitwatcharakomol, Piyawadee Jongpradubgiat, Simeng Zhu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE22 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.28392

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

RADIANT-PET: Reasoning-Augmented PET/CT Lesion Segmentation with Large Language Models and Reinforcement Learning Jiasheng Wang, Tanun Jitwatcharakomol, Piyawadee Jongpradubgiat, Simeng Zhu Abstract Accurate lesion segmentation in PET/CT is critical for oncology, yet remains challenging because physiologic tracer uptake and artifacts can mimic malignant signal. We present RADIANT-PET, a reasoning-augmented framework that couples a high-sensitivity voxel-level segmentation model with lesion-level large language model (LLM) adjudication. Candidate uptake regions are generated with a deliberately permissive segmentation stage, then converted into structured textual descriptions that summarize uptake intensity, morphology, and regional and global anatomical context. An LLM classifies each candidate as true lesion vs. false positive, optionally leveraging the radiology report as additional clinical context. To strengthen lesion-level reasoning, we further optimize a local LLM via reinforcement learning using Group Relative Policy Optimization, rewarding correct lesion classification and anatomically concordant site assignment. Across AutoPET and an OSU test cohort, RADIANT-PET consistently outperforms strong image-only baselines, with the largest improvements observed when radiology reports are provided. Overall, these results demonstrate that LLM-based lesion-level reasoning adds a novel reasoning layer beyond conventional segmentation, suppressing physiologic false positives and aligning voxel-level predictions with clinical interpretation. The project repository is available at: https://github.com/jwang-580/RADIANT-PET . 1 Introduction Positron emission tomography combined with computed tomography (PET/CT) is a cornerstone of oncologic imaging, supporting diagnosis, staging, treatment planning, response assessment, and prognostication [ 7 ] . Its clinical utility derives from the preferential uptake of radiotracers such as 2-deoxy-2-[ 18 F]fluoro-D-glucose (FDG) in metabolically active lesions: PET depicts tracer distribution, while CT provides anatomical context. Accurate baseline PET/CT tumor-burden measurement is particularly important in lymphoma because metabolic tumor volume informs prognostic modeling and outcome prediction after therapies such as chimeric antigen receptor (CAR) T-cell therapy. However, physiological uptake, inflammation or post-treatment change, infection, and imaging artifacts can be mistaken for tumor and materially distort these measurements. Image-only approaches, including nnUNet-based models, have struggled with this task and have not achieved Dice similarity coefficient (DSC) values above 0.8 [ 6 , 9 ] . In clinical practice, radiologists rely not only on uptake patterns but also on medical reasoning informed by knowledge of physiologic activity, differentiation of inflammation or infection, recognition of technical artifacts, and integration of clinical context. We hypothesize that the performance of purely image-based segmentation models is fundamentally limited by the absence of explicit reasoning capabilities [ 8 ] . Incorporating the reasoning abilities of large language models (LLMs) offers a principled way to improve discrimination between malignant uptake and physiologic or artifactual activity and thereby improve the reliability of tumor-burden quantification. Figure 1: Overview of the RADIANT-PET framework. (A) A high-sensitivity segmenter (SUV thresholding or a modified nnU-Net) proposes a permissive uptake mask. (B) Watershed splitting separates the mask into candidate regions. Each region is then converted into a structured JSON description combining radiomics and organ relationships by incorporating TotalSegmentator masks. (C) A local LLM classifies each candidate as lesion vs. physiological uptake; GRPO reinforcement learning improves this task-specific reasoning. (D) The final segmentation is obtained by removing candidates classified as false positives. We propose RADIANT-PET (Reasoning-Augmented Description–Inference Network for PET), a framework that transforms each candidate lesion region into a structured text description including information on tracer uptake characteristics and spatial relationships to surrounding organs. These region-level descriptions are then evaluated by an LLM to classify each uptake as true or false positive. We further enhance the LLM using reinforcement learning with Group Relative Policy Optimization (GRPO), directly optimizing for lesion-level classification accuracy. The framework also natively integrates radiology reports as additional context, enabling joint reasoning over imaging-derived structured descriptions and clinical narrative (Figure 1 ). By coupling voxel-level segmentation with lesion-level reasoning, RADIANT-PET achieves strong performance for PET/CT lesion segmentation. Our main contributions are: 1. Reasoning-augmented PET/CT segmentation framework. We introduce a hybrid framework that couples conventional segmentation methods for candidate lesion proposal with an LLM-based reasoning stage that adjudicates each candidate. 2. Reinforcement learning for medical reasoning. We improve LLM lesion adjudication using GRPO-based reinforcement learning with lesion-level supervision. 3. Native integration of radiology reports as physician knowledge. We directly incorporate free-text radiology reports into the reasoning step, enabling the LLM to natively leverage physician interpretations and clinical context. 2 Methods We adopt a two-stage framework: first, a permissive candidate-generation method (either intensity thresholding or a high-sensitivity nnUNet-based segmenter) proposes all putative uptake regions; second, a reasoning-based large language model adjudicates each candidate using clinical context and anatomical plausibility to retain only true lesions. 2.1 High-Sensitivity image segmentation model (HS-UNet) We train a high-sensitivity segmentation network to generate a deliberately permissive set of candidate lesions, prioritizing recall so that most true lesions are captured at the cost of additional false positives. Our candidate-generation stage follows the model-centric winner of autoPET III (nnUNet-v2), which ranked first on the FDG sub-track [ 9 , 6 ] . This model outperformed the earlier autoPET I/II winners based on older nnUNet variants. We consequently use nnUNet-v2 as the state-of-the-art fully automated image-only baseline. It combines large-scale pretraining with organ-supervision–based multitask learning to improve robustness across anatomically diverse uptake patterns. To prioritize high sensitivity in candidate generation, we modify the training objective. Specifically, the total loss ( L_{\mathrm{total}} , Eq. 1 ) is defined as the sum of a Tversky loss ( L_{\mathrm{Tversky}} ) and an asymmetric cross-entropy loss ( L_{\mathrm{ACE}} ). The Tversky loss [ 10 ] (Eq. 2 ) uses \alpha=1 and \beta=0 to reduce the penalty for false positives, encouraging permissive candidate masks. The ACE term (Eq. 3 ), motivated by prior work [ 11 ] , further emphasizes false negatives over false positives by setting \lambda_{\mathrm{FN}}=8.0 and \lambda_{\mathrm{FP}}=0.125 . \displaystyle L_{\mathrm{total}} \displaystyle=L_{\mathrm{Tversky}}+L_{\mathrm{ACE}}, (1) \displaystyle L_{\mathrm{Tversky}} \displaystyle=1-\frac{\mathrm{TP}+\epsilon}{\mathrm{TP}+\alpha\cdot\mathrm{FN}+\beta\cdot\mathrm{FP}+\epsilon}, (2) \displaystyle L_{\mathrm{ACE}} \displaystyle=\frac{1}{N}\sum_{i}w_{i}\left[-\log\frac{e^{z_{i,y_{i}}}}{\sum_{c}e^{z_{i,c}}}\right], (3) \displaystyle w_{i} \displaystyle=\begin{cases}\lambda_{\mathrm{FN}}&\text{if }y_{i}\neq 0\text{ and }\hat{y}_{i}=0,\\ \lambda_{\mathrm{FP}}&\text{if }y_{i}=0\text{ and }\hat{y}_{i}\neq 0,\\ 1&\text{otherwise.}\end{cases} During inference, the positive-class decision threshold was lowered from 0.5 to 0.1 to increase sensitivity and generate additional lesion candidates. Aside from the modified loss functions and inference procedure, all other training settings followed the implementation released by Rokuss et al. [ 9 ] . We refer to this high-sensitivity nnUNet variant as HS-UNet. The final model is publicly available. 2.2 Threshold-Based Segmentation and Post-Processing with Watershed Splitting We used SUV thresholding as an alternative, model-free approach to identify FDG-avid regions on lymphoma PET. Fixed-SUV thresholding is a clinical-standard approach for lymphoma lesion delineation in international consensus guidance and supports reproducible multicenter measurement [ 2 ] . Although several rules are used in practice, including SUV \geq 2.5 , SUV \geq 4.0 , and 40% of SUVmax, the 2.5 cutoff is validated as a reproducible threshold for lymphoma and is widely used to compute metabolic tumor volume (MTV) [ 5 ] . Thresholding is therefore a clinical-standard candidate-generation pipeline. At this stage, the main limitation is not the voxel-level delineation of each FDG-avid region but distinguishing tumor from physiologic and inflammatory uptake at the lesion level. To separate confluent uptake into individual candidate regions, we applied a watershed-based splitting procedure to the resulting masks, for both HS-UNet- and threshold-based segmentations. Stage 1 (distance-based watershed). For each connected component, we computed the Euclidean distance transform and applied Gaussian smoothing ( \sigma=2.0 voxels). Local maxima in the smoothed distance map were used as seed

Read the complete text