Patient-specific vs Multi-Patient Vision Transformer for Markerless Tumor Motion Forecasting
Gauthier Rotsart de Hertaing, Dani Manjah, Benoit Macq · 2025
Read Patient-specific vs Multi-Patient Vision Transformer for Markerless Tumor Motion Forecasting by Gauthier Rotsart de Hertaing, Dani Manjah, Benoit Macq (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-07811.txt.
Introduction
Gauthier Rotsart de Hertaing, Dani Manjah, and Benoit Macq compare two ways to train a Vision Transformer for lung tumor motion forecasting in a 2025 paper. Good forecasts help deliver proton therapy doses with care. Their markerless method needs no implanted markers.
One model was trained for a single patient, the other on data from many patients. Tests used radiographs built from planning scans. The single-patient models were more exact on planning data. The multi-patient model held up better when anatomy changed between sessions and worked without retraining. The authors frame the choice as precision against ready use in busy clinics.
Catalog record
| CREATOR | Gauthier Rotsart de Hertaing, Dani Manjah, Benoit Macq |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 28 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-07811
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Patient-specific vs Multi-Patient Vision Transformer for Markerless Tumor Motion Forecasting Gauthier Rotsart de Hertaing, Dani Manjah, Benoit Macq Abstract Background: Accurate forecasting of lung tumor motion is essential for precise dose delivery in proton therapy. While current state-of-the-art markerless methods mostly rely on deep learning models, the use of transformer-based architectures remains unexplored in this domain, despite their proven ability in trajectory forecasting tasks. Purpose: This work introduces a novel markerless forecasting approach for lung tumor motion using Vision Transformers (ViT). Two training strategies are compared under clinically realistic constraints: a patient-specific (PS) approach that learns individualized motion patterns, and a multi-patient (MP) model designed for generalization. The comparison explicitly accounts for the limited number of images that can be generated between planning and treatment sessions, a key constraint in clinical implementation. Methods: Digitally reconstructed radiographs (DRR) derived from planning four-dimensional computed tomography scans (4DCT) of 31 patients were used to train the MP model; a 32nd patient was reserved for evaluation. PS models were trained using only the target patient’s planning data. Both models used 16 DRRs per input and predicted tumor motion over a 1-second horizon. Performance was assessed using Average Displacement Error (ADE) and Final Displacement Error (FDE), on both planning (T1) and treatment (T2) data. Results: On T1 data, PS models outperformed MP models across all training set sizes, especially when trained on larger datasets (up to 25,000 DRRs, p < 0.05 p<0.05 ). However, MP models demonstrated stronger robustness to inter-fractional anatomical variability and achieved comparable performance on T2 data, without requiring retraining. Conclusions: To the best of our knowledge, this study is the first to apply ViT architectures to markerless tumor motion forecasting. It demonstrates that while patient-specific training can achieve higher precision, multi-patient models offer robust out-of-the-box performance, which is particularly suitable for time-constrained clinical settings. Future work may investigate fine-tuning MP models on each new patient, with the goal of enabling rapid and accurate model deployment. Keywords tumor motion forecasting, vision transformers, multi-patient models, patient-specific models, radiotherapy 1 Introduction Proton therapy aims to target cancerous cells with protons while minimizing exposure to surrounding healthy tissues and critical structures such as the heart and spine. Its success depends on locating tumors precisely, which is particularly challenging for thoracic and abdominal tumors, which move with the patient’s breathing. This movement introduces uncertainties in tumor positioning, compelling oncologists to apply safety margins to ensure proper dose delivery and leading in turn to the exposure of healthy tissues and vital structures Dieterich and Suh 2007 . Various techniques have been explored to improve tumor localization and minimize damage to healthy tissues, including breath-holding and abdominal compression Murphy et al. 2002 ; Nelson et al. 2005 ; Nissen and Appelt 2013 ; Lin et al. 2017 , as well as respiratory gating Shirato et al. 2000 ; Berbeco et al. 2005 ; Keall et al. 2006 . Nevertheless, breath-holding is not feasible for all patients, and respiratory gating tends to increase treatment time. To overcome these limitations, this paper focuses on a novel real-time tumor tracking (RTTT) method that aims to enable dynamic beam adjustments to the tumor’s position during treatment. RTTT methods rely on extracting tumor position from X-ray fluoroscopy imaging using either marker-based or markerless approaches. However, due to the low soft-tissue contrast of fluoroscopy images, marker-based tracking methods are often employed to enhance localization accuracy Shirato et al. 2003 ; Dieterich and Suh 2007 . These methods have been extensively researched and are known for delivering state-of-the-art results in tumor motion forecasting Putra et al. 2006 ; Jöhl et al. 2020 ; Krauss et al. 2011 ; Jiang et al. 2019 ; Teo et al. 2018 ; Wang et al. 2018 ; Lin et al. 2019 ; Pohl et al. 2022 . However, they are invasive, requiring surgical procedures that can lead to complications such as pneumothorax and infections Nuyttens et al. 2006 ; Prévost et al. 2008 ; Abdefg et al. 2002 ; Li et al. 2009 ; Prévost et al. 2009 . They also rely on real-time extraction of marker positions and perfect correlation with the tumor centroid, which requires constant updating due to inter-fraction changes. Conversely, markerless methods based on fluoroscopic images are being explored, offering new perspectives for tumor motion management in proton therapy. One approach to markerless tracking leverages the correlation between the tumor and the diaphragm Cerviño et al. 2009 ; Cerviño et al. 2010 ; Petkov et al. 2013 ; Petkov et al. 2014 ; Hirai et al. 2020 . Another involves template matching between fluoroscopic images and four-dimensional computed tomography scans (4DCT) Li et al. 2009 ; de Bruin et al. 2021 ; de Vries et al. 2021 ; Fu et al. 2024 . Recent advancements in deep learning have shown promising results in providing information about tumor positions that are typically invisible without fiducials. Despite these advancements, addressing anatomical changes between treatment sessions remains a key focus Hirai et al. 2019 ; Sakata et al. 2020 ; Zhao et al. 2019 ; Terunuma et al. 2023 ; Wei et al. 2019 ; Wei et al. 2020 . Moreover, the aforementioned markerless methods neglected system latency due to their primary focus on extracting tumor positions from fluoroscopic images. Nevertheless, latency can reach up to 500 ms Poulsen et al. 2010 , which impacts dosimetry and has been shown to reduce clinical target volume coverage Bedford et al. 2015 . As the field of deep learning continues to evolve to address the challenges of tracking mobile tumors, innovative architectures are being explored. Among them, transformers — a particular class of deep neural networks Vaswani et al. 2017 ; Dosovitskiy et al. 2020 — are emerging as promising tools in tasks that involve sequential data, especially in trajectory forecasting Giuliari et al. 2021 ; Franco et al. 2023 ; Rotsart de Hertaing et al. 2024 . To the best of our knowledge, this work is the first to investigate the ability of transformer networks to track and forecast the trajectories of mobile tumors. We contribute a novel end-to-end markerless method that focuses on forecasting tumor motion. The model is trained on digitally reconstructed radiograph (DRR) images, simulating fluoroscopic image acquisition with a realistic frame rate. To ensure clinical relevance, the model is trained on images from the treatment planning phase and evaluated on images acquired later in the treatment process. In addition, we explore training strategies, comparing patient-specific and multi-patient approaches, considering the limited time available between the planning and treatment sessions. Our findings showed that our multi-patient transformer network provides an accurate markerless method that is robust to system latency and suitable for clinical settings . Additionally, in line with fostering medical research reproducibility, we provide a GitHub repository available at https://github.com/GauthierRotsart/Tracking_tumor , advancing markerless tracking methods in clinical settings. 2 Materials and Methods Sections 2.1.1 and 2.1.2 describe the datasets used in this work. They include the patients, the treatment analyzed and the breathing amplitude computed in the x,y, and z directions with Euclidian norm and the volume of the tumor. The generation of the DRRs is explained in Section 2.2.1 . 2.1 Patients selection Data from thirty-two patients with lung tumors were collected in two