Predicting Video Slot Attention Q ueries from Rand om S lot- F eature Pairs

Rongzhen Zhao, Jian Li, Juho Kannala, Joni Pajarinen · 2025

Read Predicting Video Slot Attention Q ueries from Rand om S lot- F eature Pairs by Rongzhen Zhao, Jian Li, Juho Kannala, Joni Pajarinen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-01345.txt.

Introduction

Object-centric learning tries to split a video into separate objects, much as people see a scene. Rongzhen Zhao and co-authors note that current video methods have two blind spots: they ignore the next frame when preparing to track objects, and they never really learn how objects move. Their method, RandSF.Q, feeds the predictor both the current object slots and the frame features, so it has more to work with. It also trains on randomly sampled slot-feature pairs drawn from past steps, which forces it to learn real motion patterns.

In tests on scene representation, the method beat earlier video approaches by a wide margin, gaining up to 10 points on object discovery. The gains carried over to downstream scene understanding tasks. The authors argue that better query prediction is the lever that lifts the whole pipeline.

Catalog record

CREATORRongzhen Zhao, Jian Li, Juho Kannala, Joni Pajarinen
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-01345

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Predicting Video Slot Attention Q ueries from Rand om S lot- F eature Pairs Rongzhen Zhao, Jian Li, Juho Kannala, Joni Pajarinen Abstract Unsupervised video Object-Centric Learning (OCL) is promising as it enables object-level scene representation and dynamics modeling as we humans do. Mainstream video OCL methods adopt a recurrent architecture: An aggregator aggregates current video frame into object features, termed slots, under some queries; A transitioner transits current slots to queries for the next frame. This is an effective architecture but all existing implementations both ( i1 ) neglect to incorporate next frame features, the most informative source for query prediction, and ( i2 ) fail to learn transition dynamics, the knowledge essential for query prediction. To address these issues, we propose Random Slot-Feature pair for learning Query prediction (RandSF.Q): ( t1 ) We design a new transitioner to incorporate both slots and features, which provides more information for query prediction; ( t2 ) We train the transitioner to predict queries from slot-feature pairs randomly sampled from available recurrences, which drives it to learn transition dynamics. Experiments on scene representation demonstrate that our method surpass existing video OCL methods significantly, e.g., up to 10 points on object discovery, setting new state-of-the-art. Such superiority also benefits downstream tasks like dynamics modeling. Our core source code and training logs are available as the supplement. Figure 1: ( a ) Mainstream video OCL adopts a recurrent architecture, where a transitioner transits current slots into the query for next video frame, and an aggregator aggregates the next frame feature into slots under the query. ( b ) Our intuitive observation: To predict next query using the transitioner, next frame feature is already available and very informative thus should also be utilized. ( c ) Our empirical observation: By removing the transitioner and using current slots directly as next query, the algorithm works even better – Existing transitioner is not effectively learned. Introduction By video Object-Centric Learning (OCL) ( Singh, Deng, and Ahn 2022 ; Zadaianchuk, Seitzer, and Martius 2024 ) , objects in the video can be discovered and represented as respective single feature vectors, termed slots, as well as tracked across frames. This is achieved under self-supervision, by simply forcing the slots to reconstruct the input video frames in some format. Representing a visual scene described by a video as object-level features is not only cognitively plausible as we humans perceive the world similarly ( Palmeri and Gauthier 2004 ; Cavanagh 2011 ) , but also practically feasible as a visual scene evolves upon dynamics among objects. This is why OCL is gaining popularity in applications like scene representing ( Locatello et al. 2020 ) , dynamics modeling ( Wu et al. 2023a ) , planning and decision-making ( Palmeri and Gauthier 2004 ; Ferraro et al. 2025 ) lately. Mainstream video OCL methods ( Singh, Wu, and Ahn 2022 ; Kipf et al. 2022 ) adopt a recurrent architecture, as shown in Figure 1 ( a ). Given some query vectors, the aggregator , a Slot Attention ( Locatello et al. 2020 ) module, aggregates current video frame into slots, where each of such feature vectors represents an object. For next frame aggregation, the transitioner , typically a Transformer encoder block ( Vaswani et al. 2017 ) , transits current slots into the next query to provide temporally consistent representations for objects in dynamically evolving scenes, ensuring coherent tracking and identity preservation across time. Despite its overwhelmingly wide adoptation, does the query prediction via the transitioner between frames really learn the knowledge to effectively power OCL on videos? To the best of our knowledge, all existing implementations of this recurrent architecture have not acknowledged or explored the following two issues: i1 Although visual features of the next frame are already available and are much more informative, existing transitioners predict the next query only based on current (maybe along with previous) slots without leveraging available future information, as shown in Figure 1 ( b ); i2 Despite the primacy of transition dynamics knowledge in query prediction, existing transitioners lack the inductive bias to learn it, making themselves not only ineffective but even detrimental, as shown in Figure 1 ( c ). To address the above issues, we propose RandSF.Q, which uses Random Slot-Feature pairs as transition input to enhance the learning of Query prediction. Our method tackles these limitations through two key techniques: t1 To provide more information for query prediction, we design a new transitioner, which for the first time incorporates both current slots and features of the next video frame as its input, as shown in Figure 2 ( a ); t2 To enforce the learning of transition dynamics, we train the transitioner to predict query for the next frame from slot-feature pairs that are randomly sampled from available recurrences, as shown in Figure 2 ( b ). Besides two novel techniques mentioned above, our contributions also include: • New state-of-the-art on video object discovery tasks, as well as consistent performance boosts on downstream tasks like object recognition and video prediction; • Straightforward quantification on our transitioner’s query prediction capability. Figure 2: Our model architecture. ( left ) Our method is built upon SlotContrast ( Manasyan et al. 2025 ) . A frozen DINO2 ( Oquab et al. 2023 ) model encodes current video frame \boldsymbol{I}_{t} into current feature \boldsymbol{F}_{t} ; A Slot Attention ( Locatello et al. 2020 ) module aggregates \boldsymbol{F}_{t} into current object-level vectors, slots \boldsymbol{S}_{t} , under current query \boldsymbol{Q}_{t} ; A Transformer decoder block ( Vaswani et al. 2017 ) transits \boldsymbol{S}_{t} conditioned on next feature \boldsymbol{F}_{t+1} to next query \boldsymbol{Q}_{t+1} ; A random Transformer decoder ( Zhao et al. 2025d ) decodes \boldsymbol{S}_{t} into current reconstruction \boldsymbol{F}^{\prime}_{t} . The objective is minimizing difference between \boldsymbol{F}_{t} and \boldsymbol{F}^{\prime}_{t} . ( right ) How our transitioner works . To effectively learn transition dynamics, our transitioner explores slots \boldsymbol{S}_{t_{1}} and feature \boldsymbol{F}_{t_{2}} at any past time point within window size \mathit{\Delta} to predict next query \boldsymbol{Q}_{t+1} during training . Relative time embeddings are added to \boldsymbol{S}_{t_{1}} and \boldsymbol{F}_{t_{2}} to indicate their offset from t+1 . To maximize prediction accuracy, our trainsitioner exploits only the most recent slots \boldsymbol{S}_{t} and feature \boldsymbol{F}_{t+1} to predict \boldsymbol{Q}_{t+1} during evaluation . The left sub-figure is adapted from ( Zhao et al. 2025c ) . Related Work In this section, we briefly review existing advances of OCL on video, compare with self-supervised object tracking, and focus on existing query prediction techniques. Video object-centric learning . Given that Slot Attention has been central to video OCL since its inception, we classify existing methods as classical ones ( Jiang et al. 2019 ; Kosiorek et al. 2018 ; van Steenkiste et al. 2018 ; Veerapaneni et al. 2020 ; Burgess et al. 2019 ) and modern ones ( Kabra et al. 2021 ; Aydemir, Xie, and Guney 2023 ; Kipf et al. 2022 ; Elsayed et al. 2022 ; Safadoust and Güney 2023 ; Singh, Wu, and Ahn 2022 ; Traub et al. 2022 ; Zadaianchuk, Seitzer, and Martius 2024 ; Zoran et al. 2021 ; Qian et al. 2023 ) . Modern methods’ performance has improved a lot even on real-world videos under

Read the complete text