See, Point, Fly: A Learning-Free VLM Framework for Universal Unmanned Aerial Navigation

Chih Yao Hu Yang-Sen Lin Yuna Lee Chih-Hai Su Jie-Ying Lee, Shr-Ruei Tsai Chin-Yang Lin Kuan-Wen Chen Tsung-Wei Ke Yu-Lun Liu, National Yang Ming Chiao Tung University, National Taiwan University https://spf-web.pages.dev · 2025

Read See, Point, Fly: A Learning-Free VLM Framework for Universal Unmanned Aerial Navigation by Chih Yao Hu Yang-Sen Lin Yuna Lee Chih-Hai Su Jie-Ying Lee, Shr-Ruei Tsai Chin-Yang Lin Kuan-Wen Chen Tsung-Wei Ke Yu-Lun Liu, National Yang Ming Chiao Tung University, National Taiwan University https://spf-web.pages.dev (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-22653.txt.

Introduction

Chih Yao Hu and 9 co-authors steer drones by pointing at pictures, not by predicting actions as text. Their framework, See Point Fly, is training-free and builds on vision-language models. It can follow any free-form instruction in any setting. The key shift is to treat action choice as a two-dimensional grounding task on the camera image.

The model breaks a vague command into steps, marking waypoints on the image each round. Waypoint plus travel distance becomes a three-dimensional move command. The system adjusts distance on the fly and runs in closed loop, so it can chase moving targets. It set a new record on a drone navigation benchmark and beat baselines in real-world tests.

Catalog record

CREATORChih Yao Hu Yang-Sen Lin Yuna Lee Chih-Hai Su Jie-Ying Lee, Shr-Ruei Tsai Chin-Yang Lin Kuan-Wen Chen Tsung-Wei Ke Yu-Lun Liu, National Yang Ming Chiao Tung University, National Taiwan University https://spf-web.pages.dev
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE41 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-22653

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

See, Point, Fly: A Learning-Free VLM Framework for Universal Unmanned Aerial Navigation Chih Yao Hu Yang-Sen Lin Yuna Lee Chih-Hai Su Jie-Ying Lee, Shr-Ruei Tsai Chin-Yang Lin Kuan-Wen Chen Tsung-Wei Ke Yu-Lun Liu, National Yang Ming Chiao Tung University, National Taiwan University https://spf-web.pages.dev Abstract We present See, Point, Fly (SPF), a training-free aerial vision-and-language navigation (AVLN) framework built atop vision-language models (VLMs). SPF is capable of navigating to any goal based on any type of free-form instructions in any kind of environment. In contrast to existing VLM-based approaches that treat action prediction as a text generation task, our key insight is to consider action prediction for AVLN as a 2D spatial grounding task. SPF harnesses VLMs to decompose vague language instructions into iterative annotation of 2D waypoints on the input image. Along with the predicted traveling distance, SPF transforms predicted 2D waypoints into 3D displacement vectors as action commands for UAVs. Moreover, SPF also adaptively adjusts the traveling distance to facilitate more efficient navigation. Notably, SPF performs navigation in a closed-loop control manner, enabling UAVs to follow dynamic targets in dynamic environments. SPF sets a new state of the art in DRL simulation benchmark, outperforming the previous best method by an absolute margin of 63%. In extensive real-world evaluations, SPF outperforms strong baselines by a large margin. We also conduct comprehensive ablation studies to highlight the effectiveness of our design choice. Lastly, SPF shows remarkable generalization to different VLMs. Project page: https://spf-web.pages.dev Keywords: Vision-Language Models, Zero-shot UAV Navigation, 2D-to-3D Waypoint Prompting 1 Introduction The rapid development of unmanned aerial vehicles (UAVs) has revolutionized applications from environmental monitoring to security patrol. However, autonomous UAV navigation remains challenging due to requirements for strong visual reasoning in unstructured environments, language understanding for user instructions, and high-level task planning with low-level action control [ 1 ] . These autonomous UAV navigation tasks are often framed as aerial vision-and-language (AVLN) tasks [ 2 , 3 ] . The autonomous UAV navigation tasks are commonly framed as aerial vision-and-language (AVLN) tasks [ 2 , 3 ] . Conventional methods primalily adopt end-to-end policy learning frameworks which consist of a text and vision encoder that maps language instructions and visual observations into latent representations, followed by a policy head that converts these representations into UAV actions [ 4 , 5 , 6 , 7 , 8 , 9 , 10 ] . The entire models are trained on a curated set of expert demonstrations [ 11 , 12 , 13 , 14 ] . However, due to the limited scale and diversity of the training data, these methods fail to generalize to unseen environments or task instructions. In contrast, recent works explore a training-free direction that directly converts Vison Large Language Models (VLM) into AVLN policies [ 15 , 16 , 17 , 18 ] . As VLMs are trained on large-scale internet data, these models have demonstrated not only rich common-sense knowledge of the world, strong capabilities in visual/language understanding, reasoning and planning, but also, strong generalization to novel environments and tasks [ 19 , 20 , 21 ] . How to repurpose VLMs that generate texts into embodied agents that generate physical actions has attracted increasing interest in robotics [ 22 , 23 , 24 , 25 ] , while the research direction is still underexplored in AVLN. Existing VLM-based approaches to AVLN build atop a direct solution, that considers action prediction simply as a text-generation task. VLMs are prompoted to output either continuous actions [ 16 ] or pre-defined skills [ 15 , 17 , 18 ] , in terms of texts. Despite the simplicity of these methods, they have two obvious problems: (1) embodied agents need to execute fine-grained actions, while texts are not suitable to represent high-precision floating numbers; (2) these VLMs have not been trained on aerial navigation data to predict 3D actions for navigation. In contrast, our key insight is to consider action prediction for AVLN as a 2D spatial grounding task. Instead of predicting 3D actions directly, we harnesses VLMs to annotate 2D waypoints [ 26 , 27 , 28 , 29 ] on the image, which do not require any domain knowledge of AVLN but general spatial understanding [ 30 , 31 ] . As these 2D waypoints are grounded in the visual scene, they inherently contain precise action information. These 2D waypoints can then be transformed into 3D actions using the camera information. Notably, we do not introduce the concept of predicting 2D waypoints for action selection—similar ideas have been explored in both robot manipulation and navigation [ 28 , 26 , 27 , 20 ] . For example, RT-Trajectory [ 26 ] leverages VLMs to directly label 2D waypoints on the image, which are then used by a separately trained policy network to predict corresponding actions. PIVOT [ 28 ] , in contrast, samples multiple candidate actions as 2D waypoints and employs a VLM to select the most appropriate one for execution. In this work, we build on this general idea and adapt it to the AVLN setting. Our method requires no additional neural network training, yet it significantly outperforms PIVOT, which is also a training-free approach. Figure 1: Zero-shot language-guided UAV control. (a) The UAV continually replans to keep pace with a moving person. (b) The UAV chains multiple goals across the hall. (c) The UAV locates the person on the ground and navigates around obstacles. Coloured 3D boxes mark successive camera viewpoints, revealing the UAV’s full flight trajectory over the reconstructed point cloud. All waypoints are generated directly by the vision-language model, with no task-specific training. We introduce See, Point, Fly (SPF), a novel VLM-based AVLN framework that navigates to any goal based on any free-form instructions in any environment. At the core of our method is a VLM [ 20 ] that conditions on the current scene and language instructions, and outputs the 2D waypoints in terms of pixel locations. These 2D waypoints are unprojected into unit-length 3D positions based on the camera parameters. These 3D positions denote the relative 3D actions to the current UAV location. To enhance the navigation speed, we propose an adaptive controller module that adjust the scale of the actions based on the distance between the UAV and the target. Since our method naturally enables closed-loop control of the UAV, as shown in Fig. 1 , UAVs are capable of following dynamic targets. Moreover, building atop VLMs, our method can easily tackle long-horizon and even ambiguous task instructions in a zero-shot manner. We test SPF on a simulation and a real-world benchmark. Our method outperforms prior state-of-the-art, TypeFly [ 15 ] by a large margin. We show that our method works well across a wide range of tasks, including long-horizon, abstract, and dynamic navigation tasks. We also conduct an extensive ablation study to validate the effectiveness of each design choice. In summary, our contributions are: (1) We propose a state-of-the-art AVLN framework that generalizes to novel scenes and free-form instructions; (2) We set a new state-of-the-art in the DRL simulator [ 32 ] simulation benchmark, outperforming prior SOTAs with a margin of 63\% in success rate; (3) We set a new state of the art in the real-world

Read the complete text