Visual-RRT: Finding Paths toward Visual-Goals via Differentiable Rendering
Sebin Lee Jumin Lee Taeyeon Kim Youngju Na Woobin Im Sung-Eui Yoon · 2026
Read Visual-RRT: Finding Paths toward Visual-Goals via Differentiable Rendering by Sebin Lee Jumin Lee Taeyeon Kim Youngju Na Woobin Im Sung-Eui Yoon (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-16388.txt.
Introduction
Lee and colleagues give robot planners a goal they can see rather than a set of joint angles they must be told. Visual-RRT merges gradient-driven refinement from differentiable robot rendering with the broad sampling of rapidly-exploring random trees. A frontier strategy steers the search toward visually promising regions, while inherited optimization states keep gradient steps consistent across branches. Experiments with Franka, UR5e and Fetch arms in simulation and the real world showed the planner reaching image-specified goals, bridging sampling-based planning and vision-led robotics.
Catalog record
| CREATOR | Sebin Lee Jumin Lee Taeyeon Kim Youngju Na Woobin Im Sung-Eui Yoon |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 64955 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.16388
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Visual-RRT: Finding Paths toward Visual-Goals via Differentiable Rendering Sebin Lee Jumin Lee Taeyeon Kim Youngju Na Woobin Im Sung-Eui Yoon Abstract Rapidly-exploring random trees (RRTs) have been widely adopted for robot motion planning due to their robustness and theoretical guarantees. However, existing RRT-based planners require explicit goal configurations specified as numerical joint angles, while many practical applications provide goal specifications through visual observations such as images or demonstration videos where precise goal configurations are unavailable. In this paper, we propose visual-RRT (vRRT), a motion planner that enables visual-goal planning by unifying gradient-based exploitation from differentiable robot rendering with sampling-based exploration from RRTs. We further introduce (i) a frontier-based exploration-exploitation strategy that adaptively prioritizes visually promising search regions, and (ii) inertial gradient tree expansion that inherits optimization states across tree branches for momentum-consistent gradient exploitation. Extensive experiments across various robot manipulators including Franka, UR5e, and Fetch demonstrate that vRRT achieves effective visual-goal planning in both simulated and real-world settings, bridging the gap between sampling-based planning and vision-centric robot applications. Our code is available at https://sgvr.kaist.ac.kr/Visual-RRT . 1 Introduction Figure 1 : Motion planning toward visual goals. (a) RRT planners efficiently explore the C-space from the start configuration q_{\text{start}} yet require explicit goal configurations q_{\text{goal}} , limiting their direct use with image-specified goals. (b) Visual gradient-based methods minimize the rendering loss with respect to the goal image I_{\text{goal}} , but often struggle to reach the desired configuration. (c) Our vRRT integrates sampling-based exploration and visual-gradient exploitation, enabling the planner to efficiently discover a path toward I_{\text{goal}} . Gray regions in (a–c) indicate unreachable parts of the C-space. (d) Planning progression over time: while gradient-based optimization stagnates in local minima (top), our method continues to explore and successfully reaches the visual goal (bottom). In each step of our visualization, multiple robot poses represent parallel exploration and exploitation directions sampled during a single expansion step. Motion planning is a fundamental capability for autonomous robot manipulators, enabling them to find motion paths from a start to a goal configuration 19 ; 17 ; 41 ; 43 . Among these approaches, the rapidly-exploring random tree (RRT) 25 and its variants 18 ; 22 ; 9 have become cornerstone methods in robotics owing to their robustness and theoretical guarantees ( e.g. , probabilistic completeness or asymptotic optimality 44 ). To find motion paths, RRTs incrementally expand a search tree toward randomly sampled configurations (Fig. 1 a), efficiently exploring the configuration space (C-space) and bypassing local minima. Building on this, various works have improved path quality 18 ; 44 , convergence speed 22 ; 16 , and sampling efficiency 9 ; 39 ; 49 of RRTs. Despite their advances, RRT-based motion planners assume that explicit goal configurations are given, typically as numerical joint angles. In contrast, several emerging applications define robot goals through visual observations such as images or demonstration videos where the desired goal configuration is not explicitly known 30 ; 42 . However, existing RRT variants do not directly accommodate visually defined goals, limiting their applicability in vision-centric robotics where only goal images are provided. To bridge this gap, we extend RRTs to directly handle visual goals without explicit goal configurations. However, this extension is non-trivial since RRTs rely not only on exploration but also on exploitation where explicit goal configurations are crucial to guide goal-directed tree growth ( e.g. , goal biasing 25 , bi-directional search 22 ; 11 ; 51 , and potential field construction 39 ; 8 ). Differentiable robot rendering 30 ; 42 has shown that visual gradients of rendering loss enable such exploitation (Fig. 1 b), offering a promising foundation for visual-goal planning. In this paper, we propose visual-RRT (vRRT), a visual-goal motion planner that unifies gradient-based exploitation from differentiable robot rendering 30 with sampling-based exploration from RRTs 18 (Fig. 1 c). While differentiable rendering provides visual gradients, their single-path nature is prone to local minima (Fig. 1 d), requiring strategic integration with RRT’s multi-branch tree structure for effective goal-directed planning. Our key idea is to guide tree expansion toward promising configurations using visual gradients while maintaining stochastic sampling for global coverage. To further improve this combination, we introduce a frontier-based exploration-exploitation strategy that adaptively prioritizes visually promising search regions, leading to more effective planning. In addition, we propose inertial gradient tree expansion to formulate gradient-based exploitation, as existing RRT states ( e.g. , path cost) are not designed to capture gradient optimization history. Here, each expansion inherits optimization states from its parent for continuous optimization, enabling momentum-consistent search progress across tree branches while preserving RRT’s exploration capability. In experiments, we validate vRRT across various robot platforms with both real and synthetic datasets, demonstrating its effectiveness in visual-goal motion planning. Our contributions are summarized as follows: • We propose vRRT that unifies gradient-based exploitation from differentiable rendering with sampling-based exploration from RRTs, enabling visual-goal motion planning without explicit goal configurations. • We introduce (i) a frontier-based exploration-exploitation strategy that adaptively prioritizes visually promising search regions, along with (ii) inertial gradient tree expansion that inherits optimization states across tree branches for momentum-consistent gradient exploitation. • Extensive experiments demonstrate that vRRT consistently outperforms existing visual-goal planners in both simulated and real-world settings with various robot manipulators including Franka, UR5e, and Fetch. 2 Related Work Rapidly-exploring Random Tree (RRT) and its variants 18 ; 22 ; 9 have become widely adopted in robotic manipulation due to their robustness and theoretical guarantees ( e.g. , asymptotic optimality 44 ). Their core principle is to grow a search tree by sampling configurations while balancing exploration with goal-biased exploitation. Existing RRTs have enhanced path quality or planning efficiency through tree rewiring 18 ; 44 , bi-directional tree expansion 22 ; 51 ; 11 , and goal-directed biasing via potential fields 39 ; 8 ; 49 . Recently, learning-guided RRTs have also shown promising planning results by conducting planning in latent spaces 15 or by replacing heuristic components with neural samplers 40 ; 38 ; 47 ; 37 ; 14 . However, both classical and learning-guided RRTs often rely on explicit goal configurations for effective tree expansion. In this paper, we introduce a visual-RRT that redefines the planning objective of RRTs in the visual domain. Rather than exploiting known goal configurations, we propose to leverage differentiable robot rendering to guide tree expansion: visual-gradient steering provides exploitation signals by guiding branches toward poses whose renderings align with the goal image. This unification of gradient-based and sampling-based approaches enables effective visual-goal motion planning without explicit goal configurations. Differentiable Robot Rendering. Recent advances in differentiable rendering 20 ; 48 ; 13 ; 28 ; 50 have been applied to various robotics domains thanks to their photorealistic 3D scene representations: e.g. , world modeling 33 ; 32 , control 29 , and navigation 3 ; 1 ; 4 . Along with these advances, differentiable robot rendering 30 ; 42 fully leverages this differentiability to establish visual gradient-based connections between robot configurations and rendered images. Their core idea is to construct a self-model of the robot’s own body by combining Gaussian Splatting 20 , forward kinematics, and implicit linear blend skinning 31 , enabling rendering losses to be backpropagated to joint angles. While these methods support vision-centric robot applications ( e.g. , self-pose