Vision-Language Embodiment for Monocular Depth Estimation
Jinchang Zhang, Guoyu Lu · 2025
Read Vision-Language Embodiment for Monocular Depth Estimation by Jinchang Zhang, Guoyu Lu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-16535.txt.
Introduction
Jinchang Zhang and Guoyu Lu estimated scene depth from a single camera in 2025 by building the camera's own physics into the learning model and letting words help. One flat image cannot by itself say how far things are, yet robots must guess depth from exactly that.
Their model computes an embodied depth from the camera's fixed properties as it moves through road scenes, with no extra sensors. That depth joins the image features for a fuller picture. Text notes about the scene, including rough depth hints, act as a prior that sharpens object sense. Two fuzzy sources, image and language, cover each other's gaps. Tests across scenes showed gains over image only methods.
Catalog record
| CREATOR | Jinchang Zhang, Guoyu Lu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 44 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-16535
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Vision-Language Embodiment for Monocular Depth Estimation Jinchang Zhang, Guoyu Lu Abstract Depth estimation is a core problem in robotic perception and vision tasks, but 3D reconstruction from a single image presents inherent uncertainties. Current depth estimation models primarily rely on inter-image relationships for supervised training, often overlooking the intrinsic information provided by the camera itself. We propose a method that embodies the camera model and its physical characteristics into a deep learning model, computing embodied scene depth through real-time interactions with road environments. The model can calculate embodied scene depth in real-time based on immediate environmental changes using only the intrinsic properties of the camera, without any additional equipment. By combining embodied scene depth with RGB image features, the model gains a comprehensive perspective on both geometric and visual details. Additionally, we incorporate text descriptions containing environmental content and depth information as priors for scene understanding, enriching the model’s perception of objects. This integration of image and language — two inherently ambiguous modalities — leverages their complementary strengths for monocular depth estimation. The real-time nature of the embodied language and depth prior model ensures that the model can continuously adjust its perception and behavior in dynamic environments. Experimental results show that the embodied depth estimation method enhances model performance across different scenes. 1 Introduction Monocular depth estimation is a critical task in robotic perception and computer vision, widely used in autonomous driving, augmented reality Tang et al. 2022 , and 3D reconstruction Newcombe et al. 2011 . Its objective is to assign a depth value to each pixel in an image. However,the mapping from three-dimensional (3D) to two-dimensional (2D) space is inherently ill-posed, depth estimation faces intrinsic ambiguity, requiring identification of the most plausible scene among multiple possibilities. In this context, we propose a depth estimation method based on language and camera model embodying, integrating the physical characteristics of the camera model into the deep learning system. We compute Embodied Road Depth in real-time according to immediate changes in the road environment and extend it to the entire scene, referred to as Embodied Scene Depth. This method combines the intrinsic properties of the camera and the actual scene, providing reliable depth priors without additional hardware. By using Embodied Scene Depth fused with RGB image features as input for depth estimation, the model can capture geometric priors and visual details of the scene, achieving more accurate depth estimation. Figure 1 : Overview of the framework. We utilize a plug-and-play pre-trained image segmentation model to obtain segmentation results from images and incorporate the camera model to calculate embodied scene depth. We extract the textual semantic description of the image and derive object depth descriptions based on semantic segmentation and embodied scene depth, merging them into a textual description. The text encoder is used to predict the mean and standard deviation of the latent distribution corresponding to the depth map of the textual description. We then sample \hat{z} from the distribution using the reparameterization trick, where \epsilon\sim N(0,1) , and decode it into a depth map for loss computation. In the feature fusion module, we extract features from the embodied scene depth and RGB image and use a cross-attention mechanism for feature fusion. Next, we optimize a conditional sampler by predicting patch-wise \tilde{\epsilon} from the fused features to sample \tilde{z} from the latent space, and output the depth through the depth decoder. The text and image depth decoders share weights and are updated in both alternating steps. Additionally, we embody environmental text descriptions as language priors for scene understanding. The object information embodied in the text descriptions helps constrain the typical sizes and layouts of objects in the scene. Although these descriptions may correspond to multiple compatible 3D scenes, they provide priors that enable the model to capture object scale more effectively. In monocular depth estimation, we leverage the complementary nature of images and text: images provide direct observations of the 3D scene, while text introduces strong priors on object scale and structure. By using an image caption generator (e.g., Hu et al. 2022 ), we generate scene descriptions, and based on semantic segmentation and Embodied Scene Depth, we can obtain depth descriptions and relative positions for each object. These descriptions are integrated as textual priors. We use a text variational autoencoder (VAE) composed of a CLIP text encoder and an MLP to encode the text into a latent distribution of scene layouts, represented by mean and standard deviation. We use reparameterization to sample from the latent distribution and generate depth maps through a depth decoder. To obtain depth maps corresponding to specific images, we use a conditional sampler to sample from the image, estimating the latent distribution from the text encoding. The resulting latent vector of the scene layout is used by the depth decoder to generate the most probable depth map within the distribution. In this paper, we detail our contributions, which revolve around the embodying of the camera model and vision-language integration. These contributions have significant, multifaceted impacts on the field of depth estimation based on vision-language models: 1: We propose a method that embodies the camera model and its physical characteristics into a deep learning model to compute embodied scene depth through real-time interactions with road environments, providing reliable priors for depth estimation. 2: We extract depth text descriptions based on embodied scene depth and semantic segmentation, integrating image semantic descriptions, and enable the text variational encoder to learn the distribution of possible corresponding 3D scenes as priors. 3: We introduce a conditional sampler based on multimodal features that fuses embodied scene depth and RGB image features. The conditional sampler samples from the fused features and uses text descriptions as conditional priors for modeling. 4: We propose an embodied depth estimation framework that embodies the camera, environment, and language, leveraging the complementary strengths of image and text—two inherently ambiguous modalities—for monocular depth estimation. Our framework is shown in the Fig. 1 . 2 Related Work 2.1 Monocular Depth Estimation In the domain of supervised monocular depth estimation using neural networks, the seminal work of Eigen et al. 2014 constitutes a pivotal and foundational contribution. Their research introduces a coarse-to-fine convolutional neural network. On the basis of his model, a range of methods have emerged. It can be categorized into two distinct directions: one that involves the monodepth estimation problem as a pixel-wise regression task Huynh et al. 2020 , and another that formulates it as a pixel-wise classification challenge Cao et al. 2017 . In the decoding stage, Lee et al. 2019 introduced a local planar guidance layer to infer the plane coefficients, which were used to recover the full depth of resolution of the map. Self-supervised depth estimation from monocular videos or stereo image pairs is emerging due to the mitigation of manual labeling efforts. In the realm of monocular depth, Zhou et al. 2017 pioneered a self-supervised framework. This was achieved by jointly training depth and pose networks anchored on an image reconstruction loss. Subsequently, multiple frameworks Godard et al. 2019 ; Lu 2023a ; Lu 2023b utilized a minimum re-projection loss and auto-masking loss. Based on sensor fusion method Newcombe et al. 2011 , several studies Guizilini et al. 2020 ; Chawla et al. 2021 ;