Weakly-supervised Latent Models for Task-specific Visual-Language Control
Xian Yeow Lee, Lasitha Vidyaratne, Gregory Sin, Ahmed Farahat, Chetan Gupta · 2025
Read Weakly-supervised Latent Models for Task-specific Visual-Language Control by Xian Yeow Lee, Lasitha Vidyaratne, Gregory Sin, Ahmed Farahat, Chetan Gupta (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-18319.txt.
Introduction
Xian Yeow Lee, Lasitha Vidyaratne, Gregory Sin, Ahmed Farahat, and Chetan Gupta build a compact world model for agents that inspect hazardous places. Such an agent follow a high-level goal and act precisely, for instance centring a found object in a drone's view. Language models alone manage the spatial grounding task only a little over half the time, and full world models that could help by simulating actions are heavy to train.
Their task-specific model learns, in a shared latent space, how each action shifts the state, using only supervision about goal states. Global action codes and paired training losses keep learning stable. The design aims to give small agents a cheap way to roll out candidate moves before committing, lifting control success without a giant model.
Catalog record
| CREATOR | Xian Yeow Lee, Lasitha Vidyaratne, Gregory Sin, Ahmed Farahat, Chetan Gupta |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 35 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-18319
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Weakly-supervised Latent Models for Task-specific Visual-Language Control Xian Yeow Lee, Lasitha Vidyaratne, Gregory Sin, Ahmed Farahat, Chetan Gupta Abstract Autonomous inspection in hazardous environments requires AI agents that can interpret high-level goals and execute precise control. A key capability for such agents is spatial grounding, for example when a drone must center a detected object in its camera view to enable reliable inspection. While large language models provide a natural interface for specifying goals, using them directly for visual control achieves only 58% success in this task. We envision that equipping agents with a world model as a tool would allow them to roll out candidate actions and perform better in spatially grounded settings, but conventional world models are data and compute intensive. To address this, we propose a task-specific latent dynamics model that learns state-specific action-induced shifts in a shared latent space using only goal-state supervision. The model leverages global action embeddings and complementary training losses to stabilize learning. In experiments, our approach achieves 71% success and generalizes to unseen images and instructions, highlighting the potential of compact, domain-specific latent dynamics models for spatial alignment in autonomous inspection. \workshoptitle Space in Vision, Language, and Embodied AI 1 Introduction Autonomous inspection systems offer deployment potential in industrial environments where hazards such as high-altitude structures, confined spaces, or toxic atmospheres make human operation unsafe [ 18 , 13 , 19 ] . These systems must integrate perception, task understanding, and precise control, capabilities well suited to AI agent architectures. World models improve planning and sample efficiency, while language models provide a natural interface for operators to specify high-level goals and convert them into executable actions [ 27 , 25 , 11 , 20 , 26 ] . A common use case involves autonomous or semi-autonomous inspection with a human in the loop. An operator might issue a command such as "Inspect for defects under the bridge," which the agent interprets, plans, optionally simulates using a world model, and executes the best action. This agentic approach reduces workload, speeds positioning, increases safety and enables more flexible and adaptive inspection strategies compared to rigid, predefined workflows. While full-fidelity world models could allow agents to simulate many possible action sequences, they are data and compute intensive and often unnecessary for targeted tasks [ 10 , 5 , 3 ] . We propose a task-specific latent dynamics model that captures only the action-conditioned spatial shifts in the latent space required for centering the object, enabling effective planning with limited supervision. We focus on a subproblem of visual inspection: the agent has detected the target object in its camera view, but it is off-center. The task is to generate motion commands that bring the object into the center of the images, testing spatially grounded planning and requiring precise visual-motor coupling. This workshop paper’s contributions are as follows: First, we identify a scenario where naive multimodal LLM-based planning achieves limited success, even with careful prompting and reasoning models. Second, we propose a latent dynamics model specialized for object centering. Thirdly, we show that the model can roll out effective actions using only goal-state supervision and demonstrate that compact, task-specific latent dynamics models trained with limited data can outperform larger foundational models for spatial alignment tasks. Figure 1 provides an overview of our LLM-powered AI agent using a latent dynamics model for this inspection task. Figure 1: Overview of an LLM-powered AI agent leveraging tools, including a latent dynamics model, for improved spatially grounded visual-language control in inspection tasks. 2 Related Works Research at the intersection of language, vision, and action has explored how agents interpret natural language instructions and interact with the environment. Early benchmarks such as ALFRED introduced household-scale instruction-following tasks [ 21 ] , while subsequent work integrated large language models with embodied agents, enabling robots to sequence skills from high-level commands [ 1 ] . More recent efforts, including CALVIN [ 16 ] and MineDojo [ 8 ] , provide long-horizon, multimodal benchmarks testing agents’ ability to ground instructions in visual and interactive contexts. Visual-language models (VLMs) have shown strong generalization for object identification, spatial queries, and zero-shot navigation, though most evaluations emphasize semantic rather than fine-grained control. Models such as LLaVA [ 15 ] and Flamingo [ 2 ] demonstrate multimodal reasoning in dialogue, but targeted studies reveal VLMs underperform on precise spatial reasoning tasks [ 4 , 23 ] . Model-based approaches using learned world models have improved planning, sample efficiency, and generalization in both simulated and real-world settings [ 12 ] . Methods like DreamerV3 leverage compact latent dynamics for scalable, data-efficient reinforcement learning, and latent predictive learning frameworks such as JEPA [ 14 ] show that abstract representations improve generalization in data-limited regimes. However, these approaches often face high computational cost, sparse supervision, and difficulty capturing precise action semantics for fine-grained control. In contrast, our work focuses on task-specific dynamics modeling , learning only the dynamics needed to achieve a target goal with limited supervision, using initial and goal states rather than full trajectories [ 9 , 7 , 17 ] . By operating in a latent space, we capture explicit action semantics without sequential action data, enabling data-efficient planning for centering objects. Alternative spatially grounded control methods include traditional vision-based pipelines that compute object displacements with camera calibration and bounding box detection [ 6 ] . While effective, these require engineering effort, careful calibration, and requires further integration with language models. Zero-shot multi-modal LLM planning offers a data-efficient alternative [ 24 ] , but our experiments (Section 4.1 ) show limited performance for precise, visually grounded tasks. Table 1 provides a qualitative comparison of zero-shot LLM planning, conventional world models, and our latent dynamics model, highlighting the importance of explicit action semantics for spatial alignment. Sequential Action Task Compute Method Data Representation Specific Cost Zero-shot LLM Planning No Implicit No Low Conventional World Model Yes Explicit (Observation) No High Latent Dynamics Model (this work) No Explicit (Latent) Yes Low Table 1: Comparison of approaches highlighting data requirements, how actions are represented, task specificity, and computational cost. 3 Problem Formulation and Method 3.1 Problem Definition We consider the problem of training a task-specific latent dynamics model for autonomous alignment in drone-based inspection since prior studies have shown that learning a dynamics model in a latent space can yield better generalization than direct action prediction, especially when data quality is limited [ 22 ] . An agent (a drone) receives a natural-language instruction (e.g., "center the object" ) and must select discrete movement actions to center a target object in the frame. We focus on this problem because centering is the minimal subproblem requiring vision–language–action grounding and precise control. Success here is a strong indicator for generalization to more complex inspection tasks. A state s is defined as the tuple (x_{\text{img}},x_{\text{instr}}) , where x_{\text{img}} is the current off-center image and x_{\text{instr}} is the textual instruction. The goal state s^{*} is any image where the object of interest (e.g., gauges, switches, or structural features) is centered. For practical applications, we assume a small set of goal images is available, which can be safely captured in advance. Our dataset consists of (s,a,s^{*}) triples, where a\in\mathcal{A} is the correct action vector that will lead