ThermoAct: Thermal-Aware Vision-Language-Action Models for Robotic Perception and Decision-Making
Young-Chae Son, Dae-Kwan Ko, Yoon-Ji Choi, Soo-Chul Lim*Dongguk University · 2026
Read ThermoAct: Thermal-Aware Vision-Language-Action Models for Robotic Perception and Decision-Making by Young-Chae Son, Dae-Kwan Ko, Yoon-Ji Choi, Soo-Chul Lim*Dongguk University (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-25044.txt.
Introduction
Son and co-authors give robots a sense of heat with ThermoAct, a vision-language-action framework. Most robot models plan from sight and words alone, yet heat data can flag risk and reveal traits of objects. ThermoAct feeds thermal images into the plan. A large vision-language model reads the command, splits it into small steps, and steers the arm. This setup also makes data gathering for training more efficient. Real-world task trials show the approach works, with hints of better success and safety than sight-only systems.
Catalog record
| CREATOR | Young-Chae Son, Dae-Kwan Ko, Yoon-Ji Choi, Soo-Chul Lim*Dongguk University |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 33941 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.25044
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
ThermoAct: Thermal-Aware Vision-Language-Action Models for Robotic Perception and Decision-Making Young-Chae Son, Dae-Kwan Ko, Yoon-Ji Choi, Soo-Chul Lim*Dongguk University Abstract In recent human-robot collaboration environments, there is a growing focus on integrating diverse sensor data beyond visual information to enable safer and more intelligent task execution. Although thermal data can be crucial for enhancing robot safety and operational efficiency, its integration has been relatively overlooked in prior research. This paper proposes a novel Vision-Language-Action (VLA) framework that incorporates thermal information for robot task execution. The proposed system leverages a Vision-Language Model (VLM) as a high-level planner to interpret complex natural language commands and decompose them into simpler sub-tasks. This approach facilitates efficient data collection and robust reasoning for complex operations. Unlike conventional methods that rely solely on visual data, our approach integrates thermal information, enabling the robot to perceive physical properties and proactively ensure environmental safety. Experimental results from real-world task scenarios validate the feasibility of our proposed framework, suggesting its potential to enhance task success rates and safety compared to existing vision-based systems. I Introduction Recently, Vision-Language-Action (VLA) models have rapidly emerged as a prominent framework for realizing this integrated capability, demonstrating their potential to enable robots to comprehend language commands grounded in visual information and execute a diverse array of real-world tasks [ 4 , 27 , 16 , 24 , 20 ] . Early VLA studies predominantly utilized images acquired from RGB cameras as the primary visual input [ 5 , 16 ] .However, in real-world environments, relying solely on 2D image information is often insufficient for executing a wide range of tasks. More intelligent and safer interactions become possible only when a robot can integrate and process diverse sensory information, such as the position and distance of objects, tactile feedback, and sound. Consequently, recent VLA research has been expanding to incorporate multimodal information from various sensory modalities, including depth, tactile, and audio [ 14 , 12 , 32 ] . Contemporary VLA models have demonstrated the ability to perform a variety of complex tasks, such as folding clothes and clearing cluttered tables. Despite these advancements, the physical property of temperature remains largely overlooked. Consequently, current systems are unable to execute a seemingly simple command like ”pick up the coldest coke,” as they can visually recognize the object but cannot perceive its thermal state to make the correct selection. For instance, in an environment with a hot hair straightener or an active induction cooktop, a robot with thermal perception can proactively detect and mitigate potential fire hazards. In contrast, a robot lacking this sense would merely identify the object’s presence, failing to comprehend the situational risk. We propose ThermoAct , a VLA-based approach aimed at handling these challenges. ThermoAct is designed to integrate thermal information obtained from a thermal camera into the task planning system, enabling the robot to perform temperature-based decision-making. This work demonstrates that by integrating thermal imagery with a VLA, a robot can make more intelligent and safer choices. In particular, tasks such as selecting the ’coldest’ object or identifying ’hot and hazardous’ situations require a level of abstract reasoning that extends beyond simple image-to-action mapping. Training an end-to-end VLA model directly for such complex reasoning is inefficient, especially given the current scarcity of large-scale thermal datasets. Therefore, this study shows that a hierarchical architecture, which uses a Vision-Language Model (VLM) as a high-level planner, can be a viable approach to overcome this problem. This structure effectively decomposes a complex main task into simpler sub-tasks that the VLA can readily learn, enabling stable execution even in long-horizon sequences. In this paper, we apply this hierarchical framework to real-world scenarios, experimentally analyzing and verifying the impact of thermal information on the robot’s execution performance, stability, and safety. II Related Works II-A Integrative Approaches: VLMs and VLAs Vision-Language Models (VLMs), trained on large-scale multimodal data, possess extensive prior knowledge and remarkable reasoning capabilities. Previous studies have shown that such knowledge enables robots to recognize objects and perform tasks without additional training [ 18 , 26 ] . Moreover, VLM reasoning has been leveraged for robot task planning, improving intelligent action generation and situational adaptability [ 8 , 15 ] . An early example of applying large-scale data-driven learning to robotics is RT-1, the first large-scale robotics transformer model, which demonstrated the ability to perform diverse tasks with a high success rate [ 5 ] . Subsequent research has focused on developing capabilities that are not confined to specific environments and can handle novel tasks and situations, leading to the advancement of Vision-Language-Action (VLA) models [ 27 , 16 ] . VLAs excel at generating robot actions by comprehensively understanding natural language instructions and visual information. Recently proposed models like \pi_{0} [ 3 ] and \pi_{0}.5 [ 23 ] have shown that they can perform various tasks with only simple fine-tuning. While VLAs have the advantage of learning natural behaviors from small amounts of data [ 16 , 3 ] , they exhibit performance degradation in complex or long-horizon tasks [ 11 ] . To overcome this limitation, research is being conducted to integrate the high-level reasoning of Vision-Language Models (VLMs) with the low-level actions of VLAs by adding a VLM to the top of the framework, thereby achieving deep reasoning and natural actions [ 28 , 25 , 11 ] . ViLa [ 11 ] overcomes these limitations by directly integrating the visual understanding capabilities of GPT-4V into the planning process, showing that this approach can particularly reduce comprehension errors. Inspired by the structure of the human brain, Agentic Robot [ 28 ] proposes a framework that unifies the active reasoning and actions of VLM-based robots. II-B Combining Multimodal dataset and VLAs It is well-established that using visual information or integrating additional sensor data aids robot manipulation [ 17 , 19 ] . However, existing VLA models have predominantly focused on the visual modality and have been applied to relatively simple tasks. To address this, recent studies have proposed integrated frameworks that incorporate additional multimodal information, enabling the execution of tasks that are difficult for conventional VLA models. For example, [ 33 ] developed an LLM with strong 3D perception capabilities that directly processes 3D point clouds instead of 2D images, demonstrating an understanding of 3D space. ECoT [ 30 ] utilizes depth information to improve CoT-based reasoning for real-world manipulation tasks by leveraging an understanding of 3D spatial structures. ForceVLA [ 29 ] integrates 6D force data from the robot’s end-effector, showing that directly measuring interaction forces can improve success rates in contact-rich tasks. TLA [ 9 ] was trained using a tactile sensor and demonstrated strong performance in contact-rich tasks. Furthermore, VTLA [ 31 ] combined images from a wrist-mounted camera with the GelStereo sensor, training on a dataset of Vision-Tactile-Action-Prompt pairs, and achieved higher success rates than the original TLA. Despite these advancements in incorporating force and sound, research on understanding physical interactions related to thermal properties remains scarce. This study investigates methods for performing tasks involving temperature by integrating thermal information into a Vision-Language-based model. II-C Thermal Sensing in Robotic Perception and Planning Thermal information is valuable for enabling robotic systems to perceive their surroundings with greater precision and to assess the state