AutoTour: Automatic Photo Tour Guide with Smartphones and LLMs DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06

Huatao Xu, Zihe Liu, Zilin Zeng, Baichuan Li, Mo Li · 2026

Read AutoTour: Automatic Photo Tour Guide with Smartphones and LLMs DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06 by Huatao Xu, Zihe Liu, Zilin Zeng, Baichuan Li, Mo Li (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-06781.txt.

Introduction

Research by Huatao Xu and co-authors was posted on arXiv in 2026. It deals with AutoTour. The preprint joined the arXiv PAPERS collection in 2026. The authors present AutoTour , a system that enhances user exploration. By automatically generating fine-grained landmark annotations and descriptive narratives for photos captured by users. The key idea of AutoTour is to fuse visual features extracted from photos with nearby geospatial features queried from open matching databases. For specialists, the preprint offers a compact account of the problem and the authors' approach.

Catalog record

CREATORHuatao Xu, Zihe Liu, Zilin Zeng, Baichuan Li, Mo Li
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE53 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-06781

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

AutoTour: Automatic Photo Tour Guide with Smartphones and LLMs DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06 Huatao Xu, Zihe Liu, Zilin Zeng, Baichuan Li, Mo Li Abstract . We present AutoTour , a system that enhances user exploration by automatically generating fine-grained landmark annotations and descriptive narratives for photos captured by users. The key idea of AutoTour is to fuse visual features extracted from photos with nearby geospatial features queried from open matching databases. Unlike existing tour applications that rely on pre-defined content or proprietary datasets, AutoTour leverages open and extensible data sources to provide scalable and context-aware photo-based guidance. To achieve this, we design a training-free pipeline that first extracts and filters relevant geospatial features around the user’s GPS location. It then detects major landmarks in user photos through VLM-based feature detection and projects them into the horizontal spatial plane. A geometric matching algorithm aligns photo features with corresponding geospatial entities based on their estimated distance and direction. The matched features are subsequently grounded and annotated directly on the original photo, accompanied by large language model–generated textual and audio descriptions to provide an informative, tour-like experience. We demonstrate that AutoTour can deliver rich, interpretable annotations for both iconic and lesser-known landmarks, enabling a new form of interactive, context-aware exploration that bridges visual perception and geospatial understanding. 1. Introduction Tourism is an essential and irreplaceable aspect of modern life. Traditional tour and travel applications predominantly focus on delivering pre-curated tours, guided narratives, and static map-based exploration. For instance, popular applications such as SmartGuide ( 39 ) , GetYourGuide ( 12 ) , and Viator ( 43 ) provide location-based information and audio-guided experiences but lack real-time interactivity and personalized content. Meanwhile, platforms like Google Maps ( 17 ) and Mapy.com ( 24 ) offer offline maps and point-of-interest tagging, yet they require extensive navigation and manual interaction with the map interface. In this paper, as illustrated in Fig. 1 , we propose a novel interaction way to enhance user exploration through intelligent photo-based guidance. Users simply capture photographs using their smartphones, and the application automatically annotates key landmarks and natural features, such as buildings, lakes, and other landmarks, directly onto the images. In addition, it also presents the descriptions for those landmarks that can be translated to audio for people with needs. Those advancements not only personalize the travel experience but also deepen the user’s connection to their surroundings by offering instant, context-aware information. Figure 1 . Illustration of a new user interaction approach for touring. Although current deep learning models offer foundational capabilities for object recognition and scene understanding, they exhibit notable limitations in the context of personalized and fine-grained landmark detection. Traditional object detection models, such as YOLO ( 36 ) and Faster R-CNN ( 37 ) , are proficient at identifying general categories like "person," "car," or "tree," but they lack the semantic granularity required to distinguish specific landmarks or buildings by name. Vision-language models (VLM) like GPT-4o ( 30 ) and CLIP ( 35 ) have advanced the field by integrating visual and textual data, enabling them to recognize and describe well-known landmarks such as the Statue of Liberty or the Eiffel Tower. However, these models often struggle with less-famous or region-specific structures due to their reliance on large-scale, general-purpose datasets that may not encompass the diversity of global architecture. To this end, we propose AutoTour , a system that delivers fine-grained landmark annotations and detailed descriptions for photos captured by users. The key idea is to fuse visual information from photos with nearby objects retrieved from geospatial databases. However, to the best of our knowledge, there is currently no global dataset that contains comprehensive landmark names and structures, posing a significant barrier to training deep learning models for this large-scale task. To overcome this challenge, we design a training-free system that leverages the strong perception capabilities of existing VLMs and integrates them with open geo-spatial data sources (GDS) that maintain extensive databases of buildings and landmarks worldwide. Nevertheless, such integration introduces several challenges. Existing GDS platforms, such as Google Maps or AMap, are widely used for localization and navigation but are proprietary, offering only limited APIs that fail to provide access to comprehensive landmark-level information. To this end, we leverage OpenStreetMap (OSM) , an open-source geospatial platform that provides detailed data and supports flexible queries through the Overpass API. However, data retrieved from OSM—typically in JSON or XML format—is comprehensive yet challenging to process effectively. To address this, we design a dedicated preprocessing pipeline that extracts key geospatial features (e.g., buildings, roads, parks) from OSM data, filters them based on the user’s GPS location and camera orientation using on-device sensor data (e.g., GPS and IMU), and derives essential attributes such as feature type, distance, and direction to the user. Another major challenge arises from the discrepancy between spatial perspectives: photographs are captured from a vertical viewpoint, whereas geo-spatial data is represented in a horizontal plane. To address this issue, AutoTour adopts a divide-and-conquer strategy. We simplify the vision task by focusing on the detection of primary buildings or landmarks (termed photo features in this paper) and estimating their approximate distance, direction, and type — effectively projecting them from the vertical photo plane into the horizontal geospatial plane. The system then performs geometric overlap matching between photo features and geospatial features based on estimated distance and direction. Finally, matched features are visually annotated on the original photo using vision grounding models ( 5 ; 22 ) , and large language models generate descriptive narratives, collectively delivering an interactive and informative guided-tour experience. We implement a functional mobile application that allows users to freely take photographs and display descriptions and annotated features displayed on the map. To evaluate our system, we collect a dataset comprising 125 photos from multiple cities in China and the United States. Comprehensive user studies are conducted, where volunteers rate the quality of the generated annotations and descriptions. Our system achieves an average rating of 3.6 out of 4, demonstrating its effectiveness and user satisfaction. In summary, this work makes the following key contributions: • We explore a practical and user-friendly human-computer interaction approach to help users interpret their surroundings through photography. • We propose a novel framework that aligns geo-spatial and visual features by leveraging the powerful, training-free capabilities of VLMs and a geometric overlap matching algorithm. • We develop a working prototype of AutoTour and validate it through extensive experiments; the source code will be made publicly available to facilitate future research and development. The remainder of this paper is organized as follows. Section 2 reviews related work, and Section 3 introduces the detailed system design. Section 4 describes the implementation of the AutoTour mobile application, while Section 5 presents the evaluation results. Section 6 discusses limitations and future work, and Section 7 concludes the paper. 2. Related Works 2.1. Tour Applications While a wide range of tour-related applications exist, most exhibit notable limitations in supporting real-time, personalized exploration. For instance, SmartGuide ( 39 ) , GetYourGuide ( 12 ) , and Viator ( 43 ) primarily offer static, location-based content and curated experiences but

Read the complete text