Simple Radiology VLLM Test-time Scaling with Thought Graph Traversal Journal: Pattern Recognition

Yue Yao * , Zelin Wen * , Yan Tong * , Xinyu Tian, Xuqing Li, Xiao Ma, Dongliang Xu † {\dagger} , Tom Gedeon · 2025

Read Simple Radiology VLLM Test-time Scaling with Thought Graph Traversal Journal: Pattern Recognition by Yue Yao * , Zelin Wen * , Yan Tong * , Xinyu Tian, Xuqing Li, Xiao Ma, Dongliang Xu † {\dagger} , Tom Gedeon (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-11989.txt.

Introduction

Yue Yao and seven co-authors present a method that helps an AI model write better chest X-ray reports, in a 2025 paper. Their approach needs no new training. Instead, at the moment of use, the model is guided to reason through the chest organ by organ in a medically sensible order, a scheme the team calls thought graph traversal.

A second trick adjusts how long the model keeps reasoning before it answers, which gives it room to catch and fix its own errors. The frozen model, guided this way, beat standard prompting methods on common benchmarks for report quality. Its step-by-step paths can also be read back, which exposed biases hidden in the test data. The team released its code and prompts so others can check and reuse the method.

Catalog record

CREATORYue Yao * , Zelin Wen * , Yan Tong * , Xinyu Tian, Xuqing Li, Xiao Ma, Dongliang Xu † {\dagger} , Tom Gedeon
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE32 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-11989

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Simple Radiology VLLM Test-time Scaling with Thought Graph Traversal Journal: Pattern Recognition Yue Yao * , Zelin Wen * , Yan Tong * , Xinyu Tian, Xuqing Li, Xiao Ma, Dongliang Xu † {\dagger} , Tom Gedeon Abstract Test-time scaling offers a promising way to improve the reasoning performance of vision-language large models (VLLMs) without additional training. In this paper, we explore a simple but effective approach for applying test-time scaling to radiology report generation. Specifically, we introduce a lightweight Thought Graph Traversal (TGT) framework that guides the model to reason through organ-specific findings in a medically coherent order. This framework integrates structured medical priors into the prompt, enabling deeper and more logical analysis with no changes to the underlying model. To further enhance reasoning depth, we apply a reasoning budget forcing strategy that adjusts the model’s inference depth at test time by dynamically extending its generation process. This simple yet powerful combination allows a frozen radiology VLLM to self-correct and generate more accurate, consistent chest X-ray reports. Our method outperforms baseline prompting approaches on standard benchmarks, and also reveals dataset biases through traceable reasoning paths. Code and prompts are open-sourced for reproducibility at https://github.com/glerium/Thought-Graph-Traversal . 1 Introduction Figure 1: Test-time scaling with Thought Graph Traversal. We apply test-time scaling to radiology report generation by introducing a Thought Graph Traversal framework, which enables structured, multi-step reasoning under varying test-time compute budgets. Our method improves report quality when reasoning budget increase (measured by the length of reasoning tokens). Notably, model accuracy shows a positive correlation with the number of reasoning tokens used during inference, until it reaches a saturation point beyond which additional reasoning yields diminishing returns. Performance gains in vision-language large models in recent years have predominantly stemmed from increasing training-time computational resources, particularly through the collection of large-scale datasets for self-supervised pretraining [ 1 ] . These advancements have enabled a shift toward a new paradigm: test-time scaling , which focuses on enhancing model performance by increasing computation during inference [ 2 , 3 , 4 ] . A growing body of work like Snell et al . has explored this concept [ 5 ] , and its effectiveness was recently confirmed by OpenAI’s o1 [ 6 ] , which achieved notable reasoning improvements by scaling test-time compute. OpenAI attributes this to large-scale reinforcement learning (RL), which implies extensive use of data [ 6 ] . In response, various groups have attempted to replicate the success of o1 using strategies like Monte Carlo Tree Search [ 7 ] , multi-agent collaboration [ 8 ] , and other techniques like multi-stage reinforcement learning in DeepSeek R1 [ 9 ] . Nevertheless, a task specific implementation of test-time scaling in a specific area ( i.e . , medical report generation) has yet widely studied. Our approach is also motivated by recent research in medical report generation. Many existing methods like Chen et al . employ encoder-decoder architectures [ 10 ] . These methods aim to produce complete radiology reports from input images. However, several critical shortcomings remain unaddressed. Structural Deficiency: many generated radiology reports lack standardized structure, reducing readability and making it harder for clinicians to identify essential findings [ 11 ] . This issue is exacerbated by training on datasets composed of inconsistent and heterogeneous report formats. As a result, uniformity and clarity in generated outputs are compromised. Lack of Interpretability and Interactivity: Current systems often fail to provide transparent reasoning, relying instead on attention maps that offer limited insights. Furthermore, they lack mechanisms for clinicians to guide or modify the report generation process in accordance with specific patient contexts. This lack of transparency and flexibility limits their usefulness in real-world settings [ 12 ] . Addressing these limitations, we propose a Thought Graph Traversal (TGT) that explicitly manages the depth of reasoning within VLLMs during inference. By integrating a dedicated reasoning mechanism, our model dynamically controls the extent of logical reasoning at test time, enabling deeper and more clinically relevant analyses. Additionally, Thought Graph Traversal allows structured prior medical knowledge, systematically guiding the model towards logically coherent and medically accurate outputs. We conduct extensive ablation experiments targeting the structured reasoning of VLLMs’ results. An additional benefit of this module is its capacity to expose underlying biases present in chest X-ray datasets, which is crucial for improving model generalizability and fairness. Through extensive experimentation, our method demonstrates superior performance compared to baseline models across established benchmark datasets ( e.g . , MIMIC-CXR). Shown in Figure 1 , by implementing reasoning budget forcing, we successfully scale the model’s inference capabilities, resulting in significant improvements in diagnostic accuracy of generated reports. To foster transparency, reproducibility, and community-driven advancements, our developed model, along with datasets and source code, are publicly available. 2 Related work Vision Large Language Model. Recent advances in Transformer and computing have enabled very large language models (LLMs) like ChatGPT [ 13 ] with enhanced performance on text generation and translation. Architectures like CLIP [ 14 ] achieve multimodal understanding via image-text pretraining. Additionally, recent work guides LLMs via prompts rather than explicit training. For example, VisualChatGPT [ 15 ] connects ChatGPT with vision models for image-inclusive conversations, exemplifying this more flexible prompt-based approach to unleashing LLM potential. The idea of guiding LLMs through prompts has inspired our research. Guiding an LLM through anatomical region prompts results in structured, interpretable reports. Furthermore, clinical context prompts facilitate physician interactivity, enhancing the practical utility of the generated reports. Radiology VLLMs. Recent frameworks like Wang et al . build on medical image captioning to incorporate expert or external knowledge for richer contextual understanding in generated radiology reports [ 16 ] . Recent approaches like the MET [ 16 ] , Kiut [ 16 ] , and KBMA [ 17 ] incorporate specialized knowledge through fusion of expert tokens, multimodal injection, and trainable knowledge bases to enhance contextual understanding in radiology report generation. The work [ 18 ] uses anatomical regions for the first time to generate reports. To further increase interpretability, some efforts like Qin et al . introduce memory networks [ 19 ] to store knowledge and learn implicit image-text links. Unlike these methods focusing on model training, we focus on model test-time scaling only. Knowledge graphs have been widely used to incorporate domain knowledge for assisting radiology report generation. For example, [ 20 ] proposed integrating pre-constructed graphs that capture relationships between diseases and organs via graph neural networks, enabling dedicated abnormality-aware feature learning. [ 21 ] later extended this idea by dynamically updating the graph with new knowledge during training. More recently, [ 22 ] designed a knowledge distillation module that fuses information from a symptom graph into the final decoding stage, which shares conceptual similarities with the dynamic decision prompting (DDP) in our work. However, all these approaches rely on modifying the model through training or finetuning. In contrast, our method performs test-time scaling only, leaving the VLLM architecture and weights untouched. We use a structured prompting mechanism based on a Thought Graph Traversal, which guides the model’s reasoning process in a lightweight and plug-and-play fashion. Despite its simplicity, our method achieves stronger clinical efficacy without requiring additional training or parameter updates. Prompt engineering and test time scaling. Prompt

Read the complete text