DeFM: Learning Foundation Representations from Depth for Robotics
Manthan Patel, Jonas Frey, Mayank Mittal, Fan Yang, Alexander Hansson, Amir Bar, Cesar Cadena, and Marco Hutter · 2026
Read DeFM: Learning Foundation Representations from Depth for Robotics by Manthan Patel, Jonas Frey, Mayank Mittal, Fan Yang, Alexander Hansson, Amir Bar, Cesar Cadena, and Marco Hutter (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-18923.txt.
Introduction
Manthan Patel and seven co-authors tackle a gap in how robots learn from depth sensors. Depth cameras are now common on robots, and fake depth data made in sims can train skills that work in the real world. Yet learned depth features lag far behind those for color images, where big base models lead the field.
The authors build a base model for depth, called DeFM, meant to be reused across tasks. A shared depth model could cut the need to train from scratch for each new robot job. The work treats depth as a sense that earns its own large scale training, much like sight through color.
Catalog record
| CREATOR | Manthan Patel, Jonas Frey, Mayank Mittal, Fan Yang, Alexander Hansson, Amir Bar, Cesar Cadena, and Marco Hutter |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 69 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-18923
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
DeFM: Learning Foundation Representations from Depth for Robotics Manthan Patel, Jonas Frey, Mayank Mittal, Fan Yang, Alexander Hansson, Amir Bar, Cesar Cadena, and Marco Hutter Abstract Depth sensors are widely deployed across robotic platforms, and advances in fast, high-fidelity depth simulation have enabled robotic policies trained on depth observations to achieve robust sim-to-real transfer for a wide range of tasks. Despite this, representation learning for depth modality remains underexplored compared to RGB, where large-scale foundation models now define the state of the art. To address this gap, we present DeFM, a self-supervised foundation model trained entirely on depth images for robotic applications. Using a DINO-style self-distillation objective on a curated dataset of 60M depth images, DeFM learns geometric and semantic representations that generalize to diverse environments, tasks, and sensors. To retain metric awareness across multiple scales, we introduce a novel input normalization strategy. We further distill DeFM into compact models suitable for resource-constrained robotic systems. When evaluated on depth-based classification, segmentation, navigation, locomotion, and manipulation benchmarks, DeFM achieves state-of-the-art performance and demonstrates strong generalization from simulation to real-world environments. We release all our pretrained models, which can be adopted off-the-shelf for depth-based robotic learning without task-specific fine-tuning. Webpage: https://de-fm.github.io/ AI Artificial Intelligence LLM Large Language Model VLA Vision Language Action VLM Vision Language Model RT Real-Time GT Ground Truth GNSS Global Navigation Satellite System RTK Real-time Kinematic Positioning LiDAR Light Detection and Ranging IMU Inertial Measurement Unit RPE Relative Position Error RRE Relative Rotation Error APE Absolute Position Error ARE Relative Rotation Error RL Reinforcement Learning HF Holistic Fusion MAP Maximum a Posteriori SLAM Simultaneous Localization and Mapping SSL Self-Supervised Learning VFM Vision Foundation Models BiFPN Bi-directional Feature Pyramid Network PCA Principal Component Analysis ViT Vision Transformer I Introduction Foundation models pretrained on large-scale RGB datasets have transformed visual representation learning in recent years. Methods such as DINO [ 1 , 2 , 3 ] , and CLIP [ 4 ] have become the de facto standard for vision tasks, demonstrating remarkable zero-shot transfer to classification, detection, segmentation, and retrieval. Their success has extended into robotics, where pretrained RGB encoders serve as the backbone for various tasks such as manipulation, navigation, and scene understanding [ 5 , 6 , 7 ] . Specialized VFM ( VFM ) tailored to robotics [ 8 , 9 , 10 , 11 ] , further highlights the central role of large-scale self-supervised representation learning in enabling embodied agents. However, RGB-only observations are challenging to use for robotics, particularly in the context of sim-to-real transfer, which is a central paradigm in robot learning. This difficulty arises because simulated RGB rarely captures real-world visual complexity. In contrast, depth is invariant to lighting, texture, and color variation, establishing it as highly effective for generalization and sim-to-real transfer [ 12 ] . Depth sensors are now ubiquitous across robotic platforms, including manipulators, mobile robots, drones, and AR/VR devices, and have been leveraged for locomotion [ 13 , 14 , 15 ] , navigation [ 16 , 17 , 18 ] , and manipulation [ 19 , 20 , 21 ] . This widespread adoption highlights depth as an important modality for scalable representation learning in robotics, especially when robustness and generalization are required. Despite the benefits of depth modality, surprisingly, today, no pretrained general-purpose depth encoders exist. Current approaches either re-purpose RGB-pretrained encoders for depth by applying colormaps or channel stacking [ 22 ] , which introduces distribution mismatch and degrades geometric fidelity, or they train task-specific depth encoders from scratch [ 15 , 19 ] , leading to poor generalization across tasks, environments, and sensors. Self-supervised learning on depth remains underexplored, despite the fact that depth inherently captures critical cues for robotics, including metric geometry (e.g., object size and distance), free space (e.g., occupancy), and physical affordances, which are the action possibilities an object offers a robot, such as grasping. Given the availability and advantages of depth data, we raise the natural question: Can we build a universal pretrained depth encoder for robotics ? The simple answer is yes. In this work, we present DeFM, a dedicated self-supervised foundation model trained exclusively on depth images that demonstrates highly effective, transferable features for embodied agents. As a first qualitative result, we show that the feature space learned by DeFM is fundamentally sound: embeddings from the depth encoder reveal emergent semantic clustering when visualized via PCA (Fig. 5 ), mirroring the structure observed in effective RGB representations. This confirms that even without texture or color, semantic information can be extracted from depth, highlighting that the model captures information beyond pure geometry. DeFM leverages a DINO-style self-distillation objective applied to our curated dataset. We discuss key design choices for depth pretraining, including the importance of preserving metric depth, input normalization, and dataset mixtures for cross-domain generalization. To enable practical use in resource-constrained robotics settings, we distill our largest DeFM (ViT-L) into lightweight models ranging from 3\text{\,}\mathrm{M} to 30\text{\,}\mathrm{M} parameters, spanning both convolutional networks (ResNet, EfficientNet, RegNet) and compact transformers (ViT-S). Most critically, we demonstrate that DeFM can be applied frozen across a broad spectrum of robotics tasks. Evaluated on depth-based classification, segmentation, locomotion, navigation, and manipulation, DeFM consistently outperforms RGB-pretrained encoders repurposed for depth (Fig. ). This includes task-specific encoder baselines trained from scratch as well as state-of-the-art RGB foundation models adapted to depth. In summary, we make the following key contributions: • We present DeFM, the first large-scale self-supervised foundation model specifically for depth images with a focus on robotic tasks. • We curate a dataset of 60\text{\,}\mathrm{M} depth images and release distilled models ranging from 3\text{\,}\mathrm{M} to 30\text{\,}\mathrm{M} parameters, covering both CNNs and ViTs, to enable wide adoption in robotics research. • We introduce a novel input normalization strategy to retain metric awareness across multiple scales. DeFM achieves state-of-the-art performance across diverse perception tasks, including classification, segmentation, and various RL-based robotics tasks, including locomotion, navigation, and manipulation, with robust sim-to-real transfer. II Related Work II-A Vision Foundation Models Early visual representation learning relied on supervised pretraining on large-scale classification datasets. Convolutional networks [ 23 , 24 , 25 ] achieved strong performance when trained on datasets such as Imagenet-21k [ 26 ] . The introduction of ViT ( ViT ) [ 27 ] demonstrated that scaling model capacity and data, e.g., using proprietary datasets such as JFT-300M [ 28 ] , further improved performance across several benchmarks. However, features learned primarily for image classification often transfer sub-optimally to dense prediction tasks like segmentation or geometric prediction without extensive task-specific fine-tuning [ 29 ] . These limitations, along with the limited availability of labeled data, motivated a shift toward SSL ( SSL ), which leverages the structure of raw data to define supervisory signals without human annotation [ 30 , 29 , 31 ] . VFM refer to large, pretrained visual encoders designed to learn general-purpose representations that transfer effectively across a wide range of downstream tasks and domains without any finetuning. SSL has become a dominant pretraining paradigm for these models due to its scalability and domain flexibility. Below, we summarize the common SSL pretraining objectives. II-A 1 Contrastive Learning Contrastive methods such as SimCLR [ 32 ] , learn visual representations by maximizing the similarity between different