CTRL-O : Language-Controllable Object-Centric Visual Representation Learning

Aniket Didolkar, Andrii Zadaianchuk, Rabiul Awal, Maximilian Seitzer, Efstratios Gavves, Aishwarya Agrawal · 2025

Read CTRL-O : Language-Controllable Object-Centric Visual Representation Learning by Aniket Didolkar, Andrii Zadaianchuk, Rabiul Awal, Maximilian Seitzer, Efstratios Gavves, Aishwarya Agrawal (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-21747.txt.

Introduction

Aniket Didolkar and co-authors present CTRL-O in a 2025 paper on object-based learning from images. Such models split a scene into fixed slots, with one slot for each object. The catch is that users cannot tell the model which objects to track, since the model uses its own fixed view of what counts as an object.

CTRL-O lets a user guide the slots with plain language. The model binds words to the right objects in complex real scenes, with no mask labels needed for training. The team then uses the slots for two tasks, making images from text and answering questions about images, with strong results.

Catalog record

CREATORAniket Didolkar, Andrii Zadaianchuk, Rabiul Awal, Maximilian Seitzer, Efstratios Gavves, Aishwarya Agrawal
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE53 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-21747

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

CTRL-O : Language-Controllable Object-Centric Visual Representation Learning Aniket Didolkar, Andrii Zadaianchuk, Rabiul Awal, Maximilian Seitzer, Efstratios Gavves, Aishwarya Agrawal Abstract Object-centric representation learning aims to decompose visual scenes into fixed-size vectors called “slots” or “object files”, where each slot captures a distinct object. Current state-of-the-art object-centric models have shown remarkable success in object discovery in diverse domains, including complex real-world scenes. However, these models suffer from a key limitation: they lack controllability. Specifically, current object-centric models learn representations based on their preconceived understanding of objects, without allowing user input to guide which objects are represented. Introducing controllability into object-centric models could unlock a range of useful capabilities, such as the ability to extract instance-specific representations from a scene. In this work, we propose a novel approach for user-directed control over slot representations by conditioning slots on language descriptions. The proposed C on TR o L lable O bject-centric representation learning approach, which we term CTRL-O , achieves targeted object-language binding in complex real-world scenes without requiring mask supervision. Next, we apply these controllable slot representations on two downstream vision language tasks: text-to-image generation and visual question answering. The proposed approach enables instance-specific text-to-image generation and also achieves strong performance on visual question answering. 1 Introduction Object-centric representation learning aims to decompose a visual scene into its constituent entities or objects and represent each entity as a distinct vector called a slot . Slot-based representations are inherently compositional and support many complex downstream tasks such as dynamics modeling ( Goyal et al. 2021 ; Wu et al. 2023a ; Manasyan et al. 2025 ) , control Zadaianchuk et al. 2020 ; Driess et al. 2023 ; Didolkar et al. 2024b ; Haramati et al. 2024 , and reasoning Assouel et al. 2022 ; Mamaghan et al. 2024 . Moreover, studies in cognitive neuroscience Spelke 2000 ; Pinker 1984 have shown that human perception uses mechanisms akin to slot-based representations. Figure 1: Left: Standard object-centric learning (OCL) assigns arbitrary slots with no control. Right: CTRL-O introduces language-based control, enabling specific object targeting and multimodal applications. Existing unsupervised object-centric models ( Eslami et al. 2016 ; Engelcke et al. 2020 ; Locatello et al. 2020 ; Seitzer et al. 2023 ; Manasyan et al. 2025 ; Didolkar et al. 2024b ) can successfully decompose complex real-world visual scenes. However, they face a fundamental limitation: they lack control over the object representations. While these models expose control over the number of scene parts, they do not allow users to extract specific object representations within a scene (e.g., specified by user queries in the form of language or position markers). For instance, a user can specify that a scene should be decomposed into K slots, but they cannot direct a given slot to bind to a particular object of interest such as “a cat” or “a black purse”. This lack of control over the semantic content of the representation can be limiting, as this restricts the model always to extract a fixed decomposition of a scene based on its own preconceived understanding of objects and parts. Such rigidity can be problematic for applications that require representations at varying granularity, such as extracting the representation of a car wheel instead of the entire car, or vice versa. Moreover, many downstream tasks may require or benefit from the knowledge of the semantic content in the slots. Due to the unsupervised nature of existing models, there is no way to identify the content of a slot without manually checking its corresponding mask. For example, if the user needs representations of the cat and a dog in a particular image to answer a question, they would have to manually inspect masks of all discovered objects to pick the corresponding slots. To address these limitations, we propose to inject controllability into object-centric representation learning. We achieve this by querying the model to represent specific objects in the image. Specifically, these queries condition the slot vectors to guide them to the objects described by the query. The queries can be in the form of natural language (such as object category names or referring expressions). The main challenge is to ensure that the slots conditioned on a specific query bind to the object referred to by that query. We term the challenge of binding slots to specific objects the visual grounding problem Kazemzadeh et al. 2014 ; Xu 2015 ; Greff et al. 2020 ; Perez et al. 2018 . We find that this problem is not trivial and introduce a novel controllable object-centric model — CTRL-O — to solve it. In our experiments, we demonstrate that the proposed approach can successfully bind slots to objects specified by user queries containing object categories or referring expressions in complex real-world scenes with limited supervision. In addition, we demonstrate the usefulness of the extracted controllable representations for two downstream tasks: visual question answering and instance-controllable image generation. Our contributions are as follows: • We introduce CTRL-O, a novel method to learn controllable object-centric representations via user-defined inputs. • We demonstrate that this approach supports extracting representations for complex reference expressions, enabling precise part specification within the representation. • We validate the effectiveness of CTRL-O on two real-world downstream tasks: Instance-Controllable Image Generation and Visual Question Answering. 2 Related Works Object-Centric Representation Learning Unsupervised object-centric representation learning (OCL) has gained a lot of interest in recent years ( Eslami et al. 2016 ; Greff et al. 2019 ; Engelcke et al. 2020 ; Locatello et al. 2020 ; Biza et al. 2023 ; Seitzer et al. 2023 ; Didolkar et al. 2024b ; Kipf et al. 2022 ; Zadaianchuk et al. 2023 ; Manasyan et al. 2025 ; Didolkar et al. 2024a ) . OCL aims to extract individual representations for various entities in unstructured sensory inputs such as images. Slot Attention ( Locatello et al. 2020 ) introduces an attention-based mechanism to decompose images into object-centric representations. Dinosaur ( Seitzer et al. 2023 ) builds upon this by utilizing self-supervised DINO features ( Caron et al. 2021 ; Oquab et al. 2023 ) to enhance unsupervised object discovery. While Dinosaur can effectively identify objects in real-world data ( Lin et al. 2014 ) , it lacks mechanisms for top-down control over the representations. In contrast, CTRL-O provides controllable OCL by incorporating language-based control queries, allowing for flexible guidance with minimal supervision during training. Some works ( Kipf et al. 2022 ; Kori et al. 2023 ) have explored conditioning mechanisms in object-centric models. SAVi ( Kipf et al. 2022 ) uses bounding boxes for the initial frame of a video for conditioning. CoSA ( Kori et al. 2023 ) conditions on learned vector representations. These methods are often limited to specific forms of conditioning and are primarily evaluated on synthetic datasets, while our method can handle many forms of conditioning on real-world data. Finally, several recent works connect object-centric representations with language ( Wang et al. 2020 ; Fan et al. 2023 ; Kim et al. 2023 ) . These works connect object representations with language post-hoc, assigning language labels to discovered slots. In contrast, CTRL-O integrates language and point conditioning directly into the learning process, allowing a user to control what representations should

Read the complete text