ProtoDepth: Unsupervised Continual Depth Completion with Prototypes

Patrick Rim Hyoungseob Park S. Gangopadhyay Ziyao Zeng Younjoon Chung Alex Wong · 2025

Read ProtoDepth: Unsupervised Continual Depth Completion with Prototypes by Patrick Rim Hyoungseob Park S. Gangopadhyay Ziyao Zeng Younjoon Chung Alex Wong (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-12745.txt.

Introduction

Patrick Rim and co-authors present ProtoDepth, a method that lets a depth completion model learn new settings without forgetting old ones, in a paper from 2025. The task is to build full depth maps from a color photo plus sparse range points. When such a model trains on a new place, it tends to lose what it knew, a failure called catastrophic forgetting.

ProtoDepth keeps the base model frozen and learns small sets of prototypes for each new domain. Those prototypes reshape the model's inner features just enough for the new setting. When the test domain is unknown, learned domain descriptors pick the right prototype set on the fly. Since the base weights never change, old skills stay intact. Tests on standard benchmarks check the idea.

Catalog record

CREATORPatrick Rim Hyoungseob Park S. Gangopadhyay Ziyao Zeng Younjoon Chung Alex Wong
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE76 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-12745

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ProtoDepth: Unsupervised Continual Depth Completion with Prototypes Patrick Rim Hyoungseob Park S. Gangopadhyay Ziyao Zeng Younjoon Chung Alex Wong Abstract We present ProtoDepth, a novel prototype-based approach for continual learning of unsupervised depth completion, the multimodal 3D reconstruction task of predicting dense depth maps from RGB images and sparse point clouds. The unsupervised learning paradigm is well-suited for continual learning, as ground truth is not needed. However, when training on new non-stationary distributions, depth completion models will catastrophically forget previously learned information. We address forgetting by learning prototype sets that adapt the latent features of a frozen pretrained model to new domains. Since the original weights are not modified, ProtoDepth does not forget when test-time domain identity is known. To extend ProtoDepth to the challenging setting where the test-time domain identity is withheld, we propose to learn domain descriptors that enable the model to select the appropriate prototype set for inference. We evaluate ProtoDepth on benchmark dataset sequences, where we reduce forgetting compared to baselines by 52.2% for indoor and 53.2% for outdoor to achieve the state of the art. Project Page: https://protodepth.github.io/ . 1 Introduction Reconstructing three-dimensional (3D) environments facilitates spatial applications such as autonomous navigation, robotic manipulation, and AR/VR. These applications typically utilize platforms equipped with heterogeneous sensors, such as cameras and sparse range sensors (e.g., LiDAR, ToF) which yield RGB images and point clouds. To reconstruct 3D scenes from an ego-centric point of view in the form of depth maps, the point clouds are typically projected onto the image plane. However, their projections tend to be sparse, which is unlike RGB images that comprise irradiance measured at each pixel. These complementary modalities can be used to predict dense depth maps; this multimodal 3D reconstruction task is called depth completion. Depth completion models can be trained in a supervised (using ground truth) or unsupervised (using Structure-from-Motion) manner. As ground truth is prohibitively expensive to acquire, we subscribe to the unsupervised learning paradigm, which enables one to learn without human intervention. While this suggests the potential to continuously learn, existing models are trained and evaluated on single datasets under the assumption of a stationary data distribution. However, sequences of multiple datasets exhibit non-stationary distributions and are captured by sensors with varying calibrations. Hence, fitting to new data samples inevitably causes the model to “catastrophically forget” French 1999 ; McCloskey and Cohen 1989 ; Thrun 1995 ; Ratcliff 1990 previously learned information, where the model performance degrades significantly on data from distributions that it had already observed. Figure 1 : Results on KITTI validation set during continual training on outdoor dataset sequence (KITTI \rightarrow Waymo \rightarrow VKITTI). To enable pretrained models to adapt to new environments or domains in an unsupervised manner, we consider continual learning, where training strategies aim to mitigate catastrophic forgetting of previously observed training distributions when learning from a continuous stream of non-stationary data. One school of thought is to constrain updates to parameters deemed “important” for previous domains based on a regularization term controlled by a weight chosen as a hyperparameter. Another is to incur memory costs by storing previously observed training data and incorporating them during learning of new domains, akin to rehearsals. The trade-off is in the sensitivity of optimization (and forgetting) to the choice of hyperparameter and in the size of storage and data privacy. Both schools perform updates to the full set of model parameters, which is computationally expensive. Unlike them, we opt to freeze the pretrained model to ensure zero forgetting and learn a small set of parameters, which we term “prototypes,” that modulate the latent features for each new domain encountered. We model the change in distribution as a domain-specific bias to be learned by global multiplicative and local additive prototypes that transform the latent features to fit the new distribution. In the same vein are learnable prompts or tokens Wang et al. 2022d used in continual learning for vision transformers, where the model itself is similarly frozen. Yet, there is no natural scale at which to discretize images, unlike discrete text tokens. In contrast, our proposed prototypes lift the requirements of tokenized inputs and can thus be applied to convolutional neural networks, which are prevalently used in unsupervised depth completion, as well as transformers. To this end, we propose ProtoDepth, a novel prototype-based method for unsupervised continual depth completion. Instead of prompts in input space, we deploy lightweight prototypes to a frozen pretrained model to encode prototypical information of each domain. These prototypes model global and local biases, where global prototypes learn a transformation from the latent pretrained data distribution to that of the new domain, and local prototypes capture fine-grained features that can be selectively queried depending on the input. Naturally, when the test-time domain identity is known, i.e., domain-incremental, ProtoDepth exhibits no forgetting and learns the new data distribution with high fidelity. We further encode each domain as a descriptor to enable inference when test-time domain identity is withheld, i.e., domain-agnostic, where the prototype set corresponding to the highest affinity domain descriptor for a given sample is chosen. Our contributions : We propose (1) a novel prototype-based paradigm for unsupervised continual depth completion that incurs no forgetting in the domain-incremental setting, and (2) a prototype set selection mechanism that extends the prototype paradigm to domain-agnostic settings with minimal forgetting. This is facilitated by (3) a novel training objective that learns descriptors for each domain, which can be used to determine the prototype set suitable for inference without knowledge of domain identity. (4) Our method, ProtoDepth, reduces forgetting over baselines by over 50% across six datasets; to the best of our knowledge, this is the first unsupervised continual depth completion method. 2 Related Work Continual learning is the process of incrementally adapting the weights of a parameterized model to perform new tasks involving non-stationary distributions, while preserving information learned from previous tasks. Regularization-based methods aim to mitigate forgetting by restricting the plasticity of model parameters that are important for previously learned tasks. Kirkpatrick et al. 2017 ; Zenke et al. 2017 ; Aljundi et al. 2018 ; Chaudhry et al. 2018 introduce regularization terms to the loss function to identify and penalize changes to important weights while new tasks are learned. Rios and Itti 2019 ; Dhar et al. 2019 ; Lee et al. 2019 ; Zhai et al. 2019 ; Douillard et al. 2020 ; Hou et al. 2019 ; Li and Hoiem 2017 attempt to preserve output behavior on previous tasks using knowledge distillation Hinton 2015 ; Romero et al. 2014 . Rannen et al. 2017 ; Nguyen et al. 2018 ; Titsias et al. 2020 ; Pan et al. 2020 regularize the space of learned functions to encourage similarity between outputs of task-specific heads. However, while they perform well in simpler continual learning settings, regularization-based methods can struggle with more challenging tasks Mai et al. 2022 and larger domain shifts between datasets Rebuffi et al. 2017 ; Wu et al. 2019 . Rehearsal-based methods use a memory buffer to store a limited amount of data from previous tasks, allowing the model to periodically re-train on this data during continual learning. Chaudhry et al. 2018 ; Rolnick et al. 2019 ; Hayes et al. 2019

Read the complete text