Smoothing Slot Attention Iterations and Recurrences

Rongzhen Zhao, Wenyan Yang, Juho Kannala, Joni Pajarinen · 2025

Read Smoothing Slot Attention Iterations and Recurrences by Rongzhen Zhao, Wenyan Yang, Juho Kannala, Joni Pajarinen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-05417.txt.

Introduction

Rongzhen Zhao, Wenyan Yang, Juho Kannala, and Joni Pajarinen present a 2025 method, SmoothSA, for teaching machines to see a scene as a set of objects. The core tool, slot attention, gathers image traits into object slots through rounds of refining. On the first image or frame, the slots start blank, with no hint of what to gather, which blurs the result.

For video, later frames reuse the prior slots, a unlike task that one shared routine handles poorly. SmoothSA warms the blank slots with a dose of the input's own traits, taught by a tiny self-trained module. It also splits the routine: full rounds for the first frame, a single round for later ones. Tests on object finding, naming, and visual reasoning backed the changes.

Catalog record

CREATORRongzhen Zhao, Wenyan Yang, Juho Kannala, Joni Pajarinen
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-05417

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Smoothing Slot Attention Iterations and Recurrences Rongzhen Zhao, Wenyan Yang, Juho Kannala, Joni Pajarinen Abstract Slot Attention (SA) and its variants lie at the heart of mainstream Object-Centric Learning (OCL). Objects in an image can be aggregated into respective slot vectors, by iteratively refining cold-start query vectors, typically three times, via SA on image features. For video, such aggregation is recurrently shared across frames, with queries cold-started on the first frame while transitioned from the previous frame’s slots on non-first frames. However, the cold-start queries lack sample-specific cues thus hinder precise aggregation on the image or video’s first frame; Also, non-first frames’ queries are already sample-specific thus require transforms different from the first frame’s aggregation. We address these issues for the first time with our SmoothSA : (1) To smooth SA iterations on the image or video’s first frame, we preheat the cold-start queries with rich information of input features, via a tiny module self-distilled inside OCL; (2) To smooth SA recurrences across all video frames, we differentiate the homogeneous transforms on the first and non-first frames, by using full and single iterations respectively. Comprehensive experiments on object discovery, recognition and downstream benchmarks validate our method’s effectiveness. Further analyses intuitively illuminate how our method smooths SA iterations and recurrences. Our code is available in the supplement. Figure 1: Image Object-Centric Learning (OCL) is essentially realized via Slot Attention (SA) iterations on the image ( upper ), while video OCL is via SA recurrences across the video’s frames ( whole ). The query cold-start issue in Slot Attention (SA) iterations on the image or video’s first frame: The cold-start queries lack sample-specific cues thus hinder precise aggregation. The transform homogeneity issue in SA recurrences on the video’s first and non-first frames: Non-first frames’ queries are already sample-specific thus require transforms different from the first frame’s aggregation. Introduction Object-Centric Learning (OCL) ( Locatello et al. 2020 ) aims to represent objects in a visual scene as distinct vectors, with the background as another vector. Ideally, this yields a structured compact representation that outperforms popular dense feature maps in advanced vision tasks. In dynamics modeling, evolving these object-level slots over time captures more accurate object interactions ( Villar-Corrales and Behnke 2025 ) . For visual reasoning, their concise form allows more explicit object relationship modeling, slashing the search space and computation load ( Ding et al. 2021 ) . In visual prediction, disentangling objects facilitates more compositional generation of future frames ( Villar-Corrales, Wahdan, and Behnke 2023 ) . Powered by Slot Attention (SA) ( Locatello et al. 2020 ) , modern OCL methods have significantly improved and can now scale to real-world complex images and videos. SA is essentially a form of iterative cross attention, where query vectors compete to aggregate their corresponding object information, discovering objects as segmentation masks and representing them as slot vectors ( Locatello et al. 2020 ) . The model is trained by minimizing reconstruction loss based on the slots, requiring no external supervision. Specifically, for image, the queries are usually cold-start and sampled from multiple Gaussian distributions fitted to the entire dataset ( Jia, Liu, and Huang 2023 ) . Such queries contain no information about any specific sample, thus to obtain slots by refining queries using SA on image features, typically three iterations are necessary. For video, such aggregation occurs recurrently across all frames in a shared way, where queries for the first frame are the same as in the image case while queries for non-first frames are transitioned from the previous frame’s slots ( Singh, Wu, and Ahn 2022 ) . Unlike the first frame’s queries, non-first frames’ queries are already quite sample-specific, yet the aggregation transforms are identical or homogeneous across all frames. To the best of our knowledge, all works on SA and its variants confront these facts but have not acknowledged the implied issues, as shown in Figure 1 : ( i1 ) Query cold-start in SA iterations. For an image or video’s first frame, the cold-start queries lack scene-specific information. Although three SA iterations can gradually refine these uninformative queries into useful slots, such aggregation would not work as good as that with informative queries. ( i2 ) Transform homogeneity in SA recurrences. For video frames, the first frame’s queries are cold-start while non-first frames’ are much more informative. These differing conditions impose different requirements on the aggregation transforms, thus such homogeneous transforms would not work as good as those adapted to informative-different queries. Our solutions are straightforward. We propose SmoothSA , which smooths SA iterations on the image or video’s first frame by preheating the queries, and smooths SA recurrences across video’s first and non-first frames by differentiating the transforms: ( s1 ) A tiny module preheats the cold-start queries using rich information from input features. It is trained by predicting current slots through self-distillation within the OCL model. ( s2 ) Different aggregation transforms handle video’s first and non-first frames respectively. This is realized by simply employing three SA iterations on the first frame while only one on each non-first frame. Briefly, our contributions are: ( c1 ) for the first time addressing the query cold-start issue in SA iterations on the image and video’s first frame; ( c2 ) for the first time addressing the transform homogeneity issue in SA recurrences across the video’ first and non-first frames; ( c3 ) new state-of-the-art on image and video OCL benchmarks; ( c4 ) consistent performance boosts on downstream advanced vision tasks. Related Work As SA is a kind of cross attention that depends on queries to aggregate information from visual features, we review works from perspectives of aggregation and queries. Slot Attention on images and videos . The seminal work on the aggregation module SA ( Locatello et al. 2020 ) proposes refining the initial randomly initialized queries into object-centric slots via typically three iterations of the same SA module on image features. Then, all image OCL methods including ( Singh, Deng, and Ahn 2022 ; Seitzer et al. 2023 ; Wu et al. 2023b ; Jiang et al. 2023 ; Kakogeorgiou et al. 2024 ; Zhao et al. 2025b ; Zhao et al. 2025c ; Zhao et al. 2025d ; Zhao et al. 2025e ) adopt this iterative design. The pioneering work STEVE ( Singh, Wu, and Ahn 2022 ) extends SA to videos by conducting standard image OCL on each frame, using randomly initialized queries for the first frame while using recurrently predicted queries from previous slots for non-first frames. After, all video OCL methods including SAVi ( Kipf et al. 2022 ) , SAVi++ ( Elsayed et al. 2022 ) , SOLV ( Aydemir, Xie, and Guney 2023 ) , VideoSAUR ( Zadaianchuk, Seitzer, and Martius 2024 ) , SlotContrast ( Manasyan et al. 2025 ) , STATM ( Li et al. 2025b ) , SlotPi ( Li et al. 2025a ) and RandSF.Q ( Zhao et al. 2025a ) adopt such recurrent design. Now that SA is the core module of mainstream OCL methods for images or videos, all methods face but never acknowledge two issues described in Section Introduction. Our method is the first to address these issues directly. Query initialization for Slot

Read the complete text