Extending Foundational Monocular Depth Estimators to Fisheye Cameras with Calibration Tokens

Suchisrit Gangopadhyay, Jung-Hee Kim, Xien Chen, Patrick Rim, Hyoungseob Park, Alex Wong, Yale University, Michigan State University {rit.gangopadhyay,xien.chen,patrick.rim,hyoungseob.park,alex.wong}@yale.edu, [email protected] · 2025

Read Extending Foundational Monocular Depth Estimators to Fisheye Cameras with Calibration Tokens by Suchisrit Gangopadhyay, Jung-Hee Kim, Xien Chen, Patrick Rim, Hyoungseob Park, Alex Wong, Yale University, Michigan State University {rit.gangopadhyay,xien.chen,patrick.rim,hyoungseob.park,alex.wong}@yale.edu, [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-04928.txt.

Introduction

Rit Gangopadhyay and co-authors teach, in a 2025 paper, depth-estimating AI models trained on normal photos to work with fisheye cameras, without retraining them. Fisheye lenses bend the image so strongly that models trained on ordinary pictures misjudge distances. The fix is a small set of calibration tokens, extra learned markers that nudge the model's inner picture of a fisheye scene toward its picture of a normal one. No fisheye training photos are needed.

Training instead warps ordinary photos into fake fisheye views and demands consistent depth answers. One token set then serves both indoor and outdoor tests, and several base models all improved past prior methods. By editing the model's latent space rather than the image, the method avoids the artifacts of geometric reprojection. The code is public.

Catalog record

CREATORSuchisrit Gangopadhyay, Jung-Hee Kim, Xien Chen, Patrick Rim, Hyoungseob Park, Alex Wong, Yale University, Michigan State University {rit.gangopadhyay,xien.chen,patrick.rim,hyoungseob.park,alex.wong}@yale.edu, [email protected]
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE44 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-04928

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Extending Foundational Monocular Depth Estimators to Fisheye Cameras with Calibration Tokens Suchisrit Gangopadhyay, Jung-Hee Kim, Xien Chen, Patrick Rim, Hyoungseob Park, Alex Wong, Yale University, Michigan State University {rit.gangopadhyay,xien.chen,patrick.rim,hyoungseob.park,alex.wong}@yale.edu, [email protected] Abstract We propose a method to extend foundational monocular depth estimators (FMDEs), trained on perspective images, to fisheye images. Despite being trained on tens of millions of images, FMDEs are susceptible to the covariate shift introduced by changes in camera calibration (intrinsic, distortion) parameters, leading to erroneous depth estimates. Our method aligns the distribution of latent embeddings encoding fisheye images to those of perspective images, enabling the reuse of FMDEs for fisheye cameras without retraining or finetuning. To this end, we introduce a set of Calibration Tokens as a light-weight adaptation mechanism that modulates the latent embeddings for alignment. By exploiting the already expressive latent space of FMDEs, we posit that modulating their embeddings avoids the negative impact of artifacts and loss introduced in conventional recalibration or map projection to a canonical reference frame in the image space. Our method is self-supervised and does not require fisheye images but leverages publicly available large-scale perspective image datasets. This is done by recalibrating perspective images to fisheye images, and enforcing consistency between their estimates during training. We evaluate our approach with several FMDEs, on both indoors and outdoors, where we consistently improve over state-of-the-art methods using a single set of tokens for both. Code available at: github.com/JungHeeKim29/calibration-token . 1 Introduction Three-dimensional (3D) reconstruction is a fundamental component in many spatial applications, including autonomous vehicles, extended reality (XR), robotic manipulation. Each of these applications has unique demands for the field of view (FOV), often wider than the standard (perspective) camera. To meet this need, these applications tend to be deployed on systems equipped with fisheye or other wide-angle cameras, which allows for wider coverage of the 3D environment. However, images captured by these cameras also come with substantial distortion, which arise from differences in projective geometry, where straight lines within the 3D environment or the 3D scene are preserved in perspective images but may appear curved in fisheye images. Foundational monocular depth estimators (FMDEs) [ 74 , 45 , 49 ] are trained on orders of tens of millions of images, enabling them to generalize across a wide range 3D scenes. However, their training data is comprised of internet images, which are predominantly captured using perspective cameras. Hence, despite being trained on large-scale datasets, FMDEs produce erroneous estimates when transferred to fisheye images (see Fig. 1 ). These errors stem from a covariate shift, which can be characterized by changes in camera calibration (intrinsic, distortion) parameters – leading to differences in object appearance and their perceived depth or distance from the camera. Figure 1 : Foundational monocular depth estimators fail on fisheye images. Despite being trained on large-scale datasets, foundational monocular depth estimators (FMDEs) models produces erroneous outputs. The inaccurate, blurry estimates are caused by a covariate shift that stem from fisheye distortion. To address fisheye distortion, one solution is to recalibrate and undistort images or perform a map projection to some canonical reference frame. In principle, if one has the correct calibration, it is possible to re-project a fisheye image into a perspective-like view (or vice versa). In practice, however, there are several problems: (1) The calibration process itself can be error-prone and sensitive to physical perturbations in the camera system. Minor bumps, focus changes, or lens replacements can degrade or invalidate previously computed intrinsic parameters. (2) Even when re-projection is performed accurately, the transformation introduces latency and spatial artifacts (e.g., stretching, cropping, aliasing, loss). When used as a preprocessing step for existing pretrained depth estimators, these artifacts still present a covariate shift and can degrade performance. Another solution is to train a separate monocular depth estimator specifically for fisheye images. However, publicly available image datasets for fisheye cameras are orders of tens to hundreds of times smaller than those for perspective cameras. Hence, it is difficult to assemble sufficient data to reach the large-scale training requirement of an FMDE. Nonetheless, one can adapt or finetune existing FMDEs for fisheye imagery. While this can improve performance on fisheye images, it introduces the risk of parameter drift, where the resulting FMDEs may lose their generalizability across 3D scenes. Moreover, the resulting finetuned model becomes specialized to fisheye cameras, limiting its applicability to other camera types, which adds operational overhead in applications involving mixed camera systems, e.g., autonomous vehicles or robotics. To address these challenges, we propose a novel approach termed Calibration Tokens . Our key insight is that existing FMDEs are already capable of estimating depth for perspective images, and that errors on fisheye images are caused by a covariate shift due to differences in camera calibration and distortion. Hence, rather than retraining or finetuning the entire model, we aim to “recalibrate” the fisheye latent embeddings such that they become more conducive to an FMDE originally trained on perspective images. Leveraging the fact that many FMDEs [ 45 , 74 , 50 , 49 ] follow a Transformer-based architecture [ 11 ] , we will exploit the (self- and cross-) attention mechanism to modulate the latent (token) embeddings by inserting Calibration Tokens as part of the input. Therefore, the existing FMDE will remain effectively unchanged, while Calibration Tokens serve to adapt their internal representations to mitigate the covariate shift by aligning the latent embeddings of fisheye images to the distribution of latent embeddings of perspective images. This design also allows us to preserve the original image content without performing any spatial re-projection, ensuring the process is lossless in terms of the raw pixels. Our hypothesis is that by adding a small set of trainable tokens to encode the fisheye camera calibration information and utilizing them to recalibrate the latent embeddings, we will be able to reuse existing FMDEs trained on perspective images and adapt them to fisheye images without sacrificing their generalizability across diverse 3D scenes. To train these Calibration Tokens, we propose a self-supervised objective that leverages inverse warping in the input and output spaces. FMDEs can infer high-fidelity depth maps for perspective images, so we use the perspective image depth estimates as our training target. We then induce artificial distortion on perspective images to create pairs of perspective and synthetic fisheye images with diverse fisheye distortions. However, rather than doing the same in the output space, we undistort the fisheye depth maps to the original perspective reference frame to compute a self-supervised loss between the undistorted fisheye and perspective depth maps. By minimizing this self-supervised loss, the Calibration Tokens learn to align fisheye image embeddings to those of perspective images in the latent space, without any labels. Additionally, computing the loss in the original perspective frame allows our method to preserve the supervision signal instead of introducing artifacts. Our approach allows us to bypass the need to compile large-scale fisheye datasets by exploiting the abundance of perspective image datasets. As our method operates in the reference frame of the input, we also avoid transformation artifacts at inference time, whether in the input or output space. Furthermore, our method preserves compatibility with perspective images: One simply needs to append or remove Calibration Tokens for FMDEs to be applied to fisheye or perspective

Read the complete text