LLM-VLM Fusion Framework for Autonomous Maritime Port Inspection using a Heterogeneous UAV-USV System

Muhayy Ud Din, Waseem Akram, Ahsan B. Bakht, Irfan Hussain · 2026

Read LLM-VLM Fusion Framework for Autonomous Maritime Port Inspection using a Heterogeneous UAV-USV System by Muhayy Ud Din, Waseem Akram, Ahsan B. Bakht, Irfan Hussain (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-13096.txt.

Introduction

Muhayy Ud Din, Waseem Akram, Ahsan B. Bakht, and Irfan Hussain study how to inspect ports with robots. Port checks keep trade safe and help sites meet rules. Old ways lean on hand work and plain image tools. Those ways scale poorly and miss the sense of a scene.

The team pairs two kinds of craft, a flying drone and a surface boat craft. They join a large language model with a vision language model to guide the pair. The joint brain can read scenes with more context than plain image tools. Their frame aims for checks that cover more ground with less hand work. Port teams may use it to spot risks and faults with more care.

Catalog record

CREATORMuhayy Ud Din, Waseem Akram, Ahsan B. Bakht, Irfan Hussain
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE74 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-13096

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

LLM-VLM Fusion Framework for Autonomous Maritime Port Inspection using a Heterogeneous UAV-USV System Thanks: 1 Khalifa University Center for Autonomous Robotic Systems (KUCARS), Khalifa University, United Arab Emirates. Thanks: ∗ This publication is based upon work supported by the Khalifa University of Science and Technology under Award No. RC1-2018-KUCARS. Corresponding Author, Email: [email protected] Muhayy Ud Din, Waseem Akram, Ahsan B. Bakht, Irfan Hussain Abstract Maritime port inspection plays a critical role in ensuring safety, regulatory compliance, and operational efficiency in complex maritime environments. However, existing inspection methods often rely on manual operations and conventional computer vision techniques that lack scalability and contextual understanding. This study introduces a novel integrated engineering framework that utilizes the synergy between Large Language Models (LLMs) and Vision Language Models (VLMs) to enable autonomous maritime port inspection using cooperative aerial and surface robotic platforms. The proposed framework replaces traditional state-machine mission planners with LLM-driven symbolic planning and improved perception pipelines through VLM-based semantic inspection, enabling context-aware and adaptive monitoring. The LLM module translates natural language mission instructions into executable symbolic plans with dependency graphs that encode operational constraints and ensure safe UAV-USV coordination. Meanwhile, the VLM module performs real-time semantic inspection and compliance assessment, generating structured reports with contextual reasoning. The framework was validated using the extended MBZIRC Maritime Simulator with realistic port infrastructure and further assessed through real-world robotic inspection trials. The lightweight on-board design ensures suitability for resource-constrained maritime platforms, advancing the development of intelligent, autonomous inspection systems. Project resources (code and videos) can be found here: https://github.com/Muhayyuddin/llm-vlm-fusion-port-inspection . I Introduction Seaports play a crucial role in international trade, as more than 80% of the volume of international trade in goods is carried by sea 41 . These maritime ports require a consistent evaluation of their infrastructure to ensure operational efficiency, safety, and compliance with environmental and security regulations 39 ; 28 . Traditional inspection techniques typically depend on manual visual inspections or human-operated boats, which are often time-consuming and expensive. It includes substantial safety risks to personnel operating in hazardous industrial port environments with heavy machinery and high-traffic vessel movements. Performing comprehensive inspections regularly across large port regions is a challenge. In addition, high operational costs are associated with the deployment of specialized inspection personnel. Recent works such as 13 ; 34 highlighted the crucial need for robust and automated inspection methods that can use emerging technologies such as autonomous unmanned systems to address the scalability and efficiency challenges inherent in the inspection of maritime infrastructure. Fig. 1: LLM-VLM framework for autonomous maritime inspection. Mission instructions are processed by a LLM to generate symbolic plans and dependency graphs for UAV-USV coordination. The VLM then analyzes sensor data from the inspection scene to produce structured inspection reports, enabling semantic understanding, anomaly detection, and compliance assessment in dynamic port environments. The heterogeneous UAV-USV systems show significant potential for maritime operations, providing essential capabilities that single-platform systems cannot achieve. These combined systems play a crucial role in the current maritime security, environmental monitoring, and inspection tasks, where the combination of UAV and USV platforms creates significant operational advantages 13 34 . Maritime environments present challenges that require the collaborative capabilities of heterogeneous systems. USVs provide stable platforms for extended operations and heavy payload deployment with autonomous navigation capabilities through busy waters, close-up inspection of above-waterline structures, and floating assets, while UAVs offer rapid deployment, high-altitude observation points, and access to areas that are not reachable by surface vessels 7 . Such unmanned systems have the potential to complete maritime tasks that are unattainable for manned vehicles, significantly reduce human exposure to risk, and provide cost-effective solutions for large-scale maritime operations 7 26 . The current state-of-the-art in heterogeneous UAV-USV systems focuses primarily on addressing fundamental technical challenges related to robust platform coordination, such as achieving reliable UAV landing on moving USV platforms 13 ; 40 ; 35 , developing advanced control systems for robust navigation in the presence of environmental disturbances, and advance localization strategies in the GNSS denied environment 16 ; 61 . For mission planning, UAV-USV heterogeneous systems primarily rely on fixed mission planning architectures based on predefined state machines and rigid operational frameworks, which limit their adaptability and autonomous decision-making capabilities. Traditional mission planning approaches utilize preconfigured plans that cannot cope with the entire complexity of maritime operations, and manually provided schedules that require extensive human intervention 6 . The execution of a mission follows predetermined sequences of actions without the ability to adapt to dynamic environmental changes or unexpected scenarios 37 . For inspection tasks, existing systems are based on classical computer vision pipelines for object detection, particularly traditional object detection algorithms such as YOLO-based approaches and conventional feature matching methods that focus primarily on detecting predefined categories 18 ; 44 . The reliance on traditional computer vision frameworks fundamentally constrains the inspection capabilities of heterogeneous systems, as they cannot provide semantic understanding of complex maritime scenes, they may not adapt to diverse operational contexts, and lack the contextual reasoning necessary for advanced autonomous inspection tasks 56 . To address these gaps, we propose a novel approach that uses Large Language Models (LLMs) for intelligent mission planning for unmanned heterogeneous systems and vision language models (VLMs) for advanced inspection capabilities. Our approach uses LLM such as GPT-4 and similar architectures, to enable dynamic mission planning through natural language instruction, allowing operators to specify complex maritime missions using intuitive text commands that are automatically translated into executable mission plan for USV and UAV 11 . For enhanced inspection capabilities, we employ VLMs such as Florence-2 that fuse visual and textual modalities, allowing inspection systems that provide sophisticated scene understanding and contextual analysis beyond traditional computer vision approaches, allowing the system to interpret complex maritime scenarios and make intelligent decisions based on multimodal environmental understanding. To our knowledge, this is the first study that applies LLM-VLM fusion approach for maritime port inspection using a heterogeneous UAV-USV system. The key contributions of this work is the LLM-VLM fusion framework for autonomous port inspection using the UAV-USV heterogeneous system. The main contribution is followed by the following contributions. • LLM-Driven Symbolic Mission Planning with Integrated Preconditions: Development of a novel framework that converts natural language inspection instructions into executable symbolic plans for heterogeneous UAV-USV systems. The LLM generates both the symbolic action sequences and their associated preconditions, incorporating maritime safety knowledge, operational constraints, and inter-platform dependencies within a unified planning output. • Dependency Graph-Based Execution Management: Design and implementation of a communication manager that constructs dependency graphs from LLM-generated symbolic plans and preconditions, enabling automatic resolution of task dependencies and safe parallel execution of UAV-USV subtasks under dynamic maritime conditions. This ensures that all preconditions are satisfied before task execution, while maximizing efficiency through intelligent scheduling. • VLM-Based Semantic Inspection System: Implementation of lightweight VLMs for real-time multimodal inspection, enabling context-aware anomaly detection, regulatory compliance checking, and structured report generation that goes beyond traditional computer vision pipelines through semantic reasoning and natural language understanding. • Comprehensive AI Model Benchmarking: Systematic evaluation of multiple LLM (GPT-4o, GPT-3.5-Turbo, GPT-4, Gemini, LLaMA) for mission planning accuracy and execution success, and lightweight VLM (SmolVLM, Florence-2, Qwen2-VL, Moondream2, GIT-base) for inspection accuracy, analysis of response time, computational efficiency, and deployment feasibility

Read the complete text