Rethinking Temporal Consistency in Video Object-Centric Learning: From Prediction to Correspondence

Zhiyuan Li, Rongzhen Zhao, Wenyan Yang, Wenshuai Zhao, Pekka Marttinen, Joni Pajarinen · 2026

Read Rethinking Temporal Consistency in Video Object-Centric Learning: From Prediction to Correspondence by Zhiyuan Li, Rongzhen Zhao, Wenyan Yang, Wenshuai Zhao, Pekka Marttinen, Joni Pajarinen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-03650.txt.

Introduction

Zhiyuan Li and five co-authors rethink how video object-centric learning keeps track of objects over time. The standard approach trains dynamics modules to predict future object representations, called slots. The authors argue these predictors are costly approximations of a simpler discrete matching problem: matching objects between frames.

Modern self-supervised vision backbones already encode features that tell instances apart, so prediction can be replaced by matching. Their Grounded Correspondence framework starts slots from salient regions in frozen backbone features and keeps identity through Hungarian matching, with no learnable parameters for time modeling. It still achieves competitive performance on the MOVi-D, MOVi-E, and YouTube-VIS benchmarks.

Catalog record

CREATORZhiyuan Li, Rongzhen Zhao, Wenyan Yang, Wenshuai Zhao, Pekka Marttinen, Joni Pajarinen
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE44 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.03650

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Rethinking Temporal Consistency in Video Object-Centric Learning: From Prediction to Correspondence Zhiyuan Li, Rongzhen Zhao, Wenyan Yang, Wenshuai Zhao, Pekka Marttinen, Joni Pajarinen Abstract The de facto approach in video object-centric learning maintains temporal consistency through learned dynamics modules that predict future object representations, called slots. We demonstrate that these predictors function as expensive approximations of discrete correspondence problems. Modern self-supervised vision backbones already encode instance-discriminative features that distinguish objects reliably. Exploiting these features eliminates the need for learned temporal prediction. We introduce Grounded Correspondence, a framework that replaces learned transition functions with deterministic bipartite matching. Slots initialize from salient regions in frozen backbone features. Frame-to-frame identity is maintained through Hungarian matching on slot representations. The approach requires zero learnable parameters for temporal modeling yet achieves competitive performance on MOVi-D, MOVi-E, and YouTube-VIS. Project page: https://magenta-sherbet-85b101.netlify.app/ Figure 1 : From prediction to correspondence: rethinking temporal consistency in video object-centric learning. Top: The predictive paradigm uses content-blind initialization without spatial priors. Temporal consistency relies on learned dynamics modules that forecast future slot states through high-capacity Transformers ( 25 ) or recurrent networks ( 2 ) . Bottom: Grounded Correspondence eliminates learned temporal parameters entirely. Slots initialize from saliency peaks in frozen backbone features. Frame-to-frame identity is maintained through parameter-free Hungarian matching ( 11 ) on slot representations. This paradigm shift achieves competitive performance with zero learnable parameters for temporal modeling. 1 Introduction Unsupervised object-centric learning decomposes visual scenes into discrete slot representations ( 18 ) . The field has evolved from synthetic benchmarks toward unconstrained real-world video sequences, where maintaining object identity across temporal sequences has become a central problem. The dominant solution treats this challenge through predictive dynamics modeling: methods deploy transitioners that forecast future slot states from historical context ( 8 ; 5 ; 29 ; 1 ) . Current implementations share two design choices. Slot initialization remains content-blind: methods either sample queries from a shared Gaussian distribution or learn fixed parameters applied uniformly across datasets ( 18 ; 8 ; 19 ; 23 ) . Temporal consistency then relies on learned dynamics modules, ranging from recurrent networks to temporal Transformers and physics-informed architectures ( 10 ; 24 ; 26 ; 21 ; 13 ; 27 ; 12 ) . While these methods achieve competitive benchmark performance, they assume that maintaining object identity requires explicit simulation of physical dynamics. This assumption carries substantial computational cost. We challenge the necessity of learned dynamics modules. These high-capacity predictors often reduce to identity permutations, functioning as computationally expensive solutions to what is fundamentally an alignment problem. Modern self-supervised vision backbones already encode object-binding signals within their feature maps ( 14 ) . Patches belonging to the same instance exhibit elevated feature similarity, effectively creating unsupervised saliency maps where high-confidence regions correspond to object centers. When the encoder already provides instance-aware features, temporal consistency becomes a problem of maintaining stable slot-to-object correspondences rather than simulating forward dynamics. Temporal consistency is a correspondence problem, not a prediction problem. Slot Attention exhibits permutation equivariance ( 18 ) , meaning that semantic content is independent of index order. Maintaining object identity across frames requires only that we recover the correct permutation between consecutive timesteps. A parameter-free bipartite matching solver suffices for this task. Combined with initialization grounded in backbone saliency, this eliminates the need for learned dynamics modules. We introduce Grounded Correspondence, a framework that replaces parameterized temporal transitioners with deterministic discrete optimization. Slot queries are initialized from emergent saliency peaks in the frozen backbone. Frame-to-frame identity is maintained through Hungarian matching ( 11 ) on slot features. The approach achieves competitive performance on MOVi-D, MOVi-E, and YouTube-VIS while requiring zero learnable parameters for temporal modeling. We review related work in Section 2 , analyze current methods to motivate our design in Sections 3 to 4 , present the framework in Section 5 , and provide experimental validation in Section 6 (preliminaries in Appendix A ). Our contributions are as follows: • We show that exploiting instance-aware signals from self-supervised backbones DINOv2 ( 22 ) accelerates slot convergence compared to content-blind initialization, reducing the computational cost of the iterative grouping process. • We demonstrate empirically that learned temporal transitioners in current methods approximate solutions to bipartite matching problems, suggesting architectural redundancy. • We introduce Grounded Correspondence, a framework combining grounded initialization with parameter-free Hungarian matching that requires zero learnable parameters for temporal modeling. • Through extensive experiments on MOVi-D, MOVi-E, and YouTube-VIS, we demonstrate strong improvements on synthetic benchmarks (+15.7 ARI on MOVi-D, +6.8 ARI on MOVi-E) and competitive performance on real-world sequences. Figure 2 : Slot Attention convergence with different initialization strategies. Four examples from YouTube-VIS. For each example, left shows content-blind initialization across 3 iterations, right shows grounded initialization with 1 iteration. Content-blind initialization requires multiple iterations to stabilize object boundaries, while grounded initialization achieves stable segmentation in a single iteration. 2 Related Work Video Object-Centric Learning. Unsupervised scene decomposition extracts slot representations from visual data. Early methods reconstructed pixels on synthetic sequences ( 8 ) , while subsequent work targeted feature-level reconstruction on naturalistic videos ( 24 ; 5 ; 29 ) . Applications now include datasets with unconstrained visual variability ( 1 ) and downstream tasks including 3D novel-view synthesis ( 17 ) , language-controllable editing ( 3 ) , and multiple-object tracking ( 31 ) . Existing methods enforce object-centric structure through specialized architectural components. We examine whether modern vision backbones already encode sufficient structural priors to reduce this architectural complexity. Slot Initialization Strategy. Slot Attention requires initializing query vectors that determine the resulting decomposition quality. Standard methods sample from a shared Gaussian distribution ( 8 ) or use fixed learnable parameters ( 19 ) . MetaSlot ( 16 ) introduces a prototype codebook to handle variable slot counts, while SMTC ( 23 ) employs semantic-aware Gaussian means. These approaches remain agnostic to image content at initialization. Recent work demonstrates that self-supervised Vision Transformers encode object-binding signals within their feature maps: patches from the same instance exhibit elevated similarity ( 14 ) . SlotContrast ( 19 ) briefly explores k-means clustering for initialization. We exploit these emergent instance-aware properties directly, seeding slots at high-saliency feature locations rather than learning content-agnostic priors. Temporal Consistency. Existing methods treat temporal consistency as a prediction task. Models deploy recurrent transitioners ( 10 ; 15 ) or temporal Transformers ( 26 ; 12 ; 24 ; 8 ) to forecast future slot states. Variants incorporate conditional autoregressive priors ( 21 ) , physics-informed Hamiltonian constraints ( 13 ) , or latent diffusion for slot-conditioned denoising ( 27 ) . Recent work refines query prediction mechanisms ( 30 ) or distills representations into efficient student architectures ( 20 ; 7 ) . These approaches assume that maintaining identity requires modeling object dynamics. We argue that temporal consistency is a correspondence problem rather than a prediction problem. When the visual encoder provides instance-discriminative features, discrete matching between consecutive frames suffices to maintain object identity. We replace learned temporal predictors with deterministic bipartite matching. 3 Rethinking Slot Initialization: From Content-Blind to Grounded Slot Attention partitions visual tokens through iterative competitive grouping ( 18 ) . In video object-centric learning, slots for frame t are refined through multiple attention iterations

Read the complete text