Object-Centric Pretraining via Target Encoder Bootstrapping

Nikola Đukić Tim Lebailly Tinne Tuytelaars · 2025

Read Object-Centric Pretraining via Target Encoder Bootstrapping by Nikola Đukić Tim Lebailly Tinne Tuytelaars (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-15141.txt.

Introduction

Nikola Đukić, Tim Lebailly, and Tinne Tuytelaars train object-centric vision models from scratch with a moving target encoder. Object-centric models split a scene into slots for single objects, and they have leaned on frozen foundation models for training targets. A frozen target caps how good the model can get, yet naive updates make training collapse.

Their 2025 paper, named OCEBO, updates the target encoder as a slow average of the object-centric model itself. Slot attention feeds its bias into the target, and a cross-view patch filter keeps training focused on patches that carry real signal. The setup trains on real-world data without a pre-made teacher, a first for this family of models.

Catalog record

CREATORNikola Đukić Tim Lebailly Tinne Tuytelaars
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE45 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-15141

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Object-Centric Pretraining via Target Encoder Bootstrapping Nikola Đukić Tim Lebailly Tinne Tuytelaars Abstract Object-centric representation learning has recently been successfully applied to real-world datasets. This success can be attributed to pretrained non-object-centric foundation models, whose features serve as reconstruction targets for slot attention. However, targets must remain frozen throughout the training, which sets an upper bound on the performance object-centric models can attain. Attempts to update the target encoder by bootstrapping result in large performance drops, which can be attributed to its lack of object-centric inductive biases, causing the object-centric model’s encoder to drift away from representations useful as reconstruction targets. To address these limitations, we propose O bject- CE ntric Pretraining by Target Encoder BO otstrapping, a self-distillation setup for training object-centric models from scratch, on real-world data, for the first time ever. In OCEBO, the target encoder is updated as an exponential moving average of the object-centric model, thus explicitly being enriched with object-centric inductive biases introduced by slot attention while removing the upper bound on performance present in other models. We mitigate the slot collapse caused by random initialization of the target encoder by introducing a novel cross-view patch filtering approach that limits the supervision to sufficiently informative patches. When pretrained on 241k images from COCO, OCEBO achieves unsupervised object discovery performance comparable to that of object-centric models with frozen non-object-centric target encoders pretrained on hundreds of millions of images. The code and pretrained models are publicly available at https://github.com/djukicn/ocebo . 1 Introduction In recent years, large-scale foundation models have become ubiquitous across many domains of deep learning. This is mainly thanks to the development of self-supervised learning (SSL) techniques that enable pretraining on massive amounts of unlabeled data. In computer vision, several families of SSL techniques have emerged, focusing on (i) imposing cross-view invariance at the global ( Grill et al., 2020 ; Caron et al., 2021 ) , patch ( Zhou et al., 2022 ; Stegmüller et al., 2023 ) or cluster ( Wen et al., 2022 ; Lebailly et al., 2024 ) level, or (ii) reconstruction in the pixel ( He et al., 2021 ) or latent space ( Zhou et al., 2022 ; Assran et al., 2023 ) . On the other hand, research in cognitive psychology ( Benjamin Peters, 2021 ) suggests that human visual perception reasons about visual inputs by decomposing scenes into sets of objects with corresponding representations. In the past years, efforts have been made to mimic this behavior ( Locatello et al., 2020 ; Seitzer et al., 2022 ; Kakogeorgiou et al., 2024 ; Didolkar et al., 2024 ) and move past global and patch representations towards object-centric representations. Object-centric models are not only more biologically plausible ( Benjamin Peters, 2021 ) , but can provably achieve compositional generalization ( Wiedemer et al., 2024 ) and have already proven useful on several downstream tasks, such as robotic control ( Zadaianchuk et al., 2021 ; Haramati et al., 2024 ) , compositional generation ( Singh et al., 2022 ; Wu et al., 2023 ; Jiang et al., 2023 ; Sajjadi et al., 2022 ) , and visual question answering ( Xu et al., 2024 ) . While incredibly promising, object-centric models have not yet been successfully pretrained on a large-scale real-world dataset. A slot attention-based object-centric model consists of an image encoder, slot attention encoder and slot decoder. Initially, these were trained end-to-end with a reconstruction objective in the pixel-space ( Locatello et al., 2020 ) , resulting in good scene decomposition on synthetic datasets but slot collapse 1 1 1 Slot collapse refers to a scenario where slots learn to attach to positionally coherent blocks rather than actual objects. on real-world data. Seitzer et al. (2022) introduce a reconstruction objective in the latent space of a pretrained encoder (e.g., DINO ( Caron et al., 2021 ) , DINOv2 ( Oquab et al., 2023 ) , etc.), showing that the presence of informative reconstruction targets is crucial for avoiding slot collapse. As this encoder provides reconstruction targets, we refer to it as a target encoder . Moreover, Seitzer et al. (2022) initialize the encoder of their object-centric model with the same pretrained encoder and keep it frozen alongside the target encoder, effectively training only the slot attention encoder and decoder. While the framework with the frozen target encoder achieves unprecedented performance on real-world datasets ( Seitzer et al., 2022 ) , it has a significant limitation. Namely, the target encoder cannot be updated, meaning that there exists an upper bound on what the object-centric model can learn from it. In fact, despite the target encoder being pretrained on millions or even hundreds of millions of images ( Caron et al., 2021 ; Oquab et al., 2023 ) , the object-centric model can efficiently consume all its knowledge and reach the upper limit. This is demonstrated in an experiment by Didolkar et al. (2024) , where the performance plateaus with \sim 16k images from MS COCO ( Lin et al., 2015 ) and does not further increase, even with an order of magnitude more data. This clashes with the scalability trends observed in other (self-supervised) representation learning families. A question arises — how to update the target encoder, thus removing the performance upper limit and enabling large-scale pretraining of object-centric models? Kakogeorgiou et al. (2024) show for the first time that unfreezing a few final layers of the object-centric model’s encoder is possible and further improves the performance. Didolkar et al. (2024) further extend this finding by fine-tuning the full encoder. A straightforward approach would be updating the target encoder as an exponential moving average (EMA) of the object-centric model’s encoder. However, this has been shown to result in huge performance drops ( Didolkar et al., 2024 ) . We hypothesize that the reason for this is the lack of object-centric inductive biases in the target encoder. Most often trained for cross-view consistency, SSL models organize their representations in terms of semantics, rather than object instances. For example, semantically similar parts of different objects will be close in the representation space and farther from less similar parts of the same object. When the object-centric model tries to reconstruct such features, its inductive bias focuses on assigning these semantically similar regions to different objects (i.e., instances), thus updating its encoder to focus slightly less on semantics and more on positions in the image. EMA update of the target encoder then results in informative knowledge slowly leaking at the cost of more positional information, which ultimately degrades the performance. In this work, we argue that a way to overcome the current limitations of object-centric learning and unlock its full potential is by large-scale pretraining from scratch. We pose object-centric learning as a self-distillation bootstrapping problem ( Caron et al., 2021 ) , in which the object-centric model is distilled from the target encoder that is in turn updated as an EMA of the object-centric model’s encoder. We propose the object-centric self-distillation loss , a patch-level loss that acts as a reconstruction objective. We apply it in a cross-view consistency fashion to enrich slot attention with augmentation invariance. To prevent slot collapse stemming from the lack of informative reconstruction targets due to random

Read the complete text