Human-like compositional learning of visually-grounded concepts using synthetic environments

Zijun Lin, M Ganesh Kumar, Cheston Tan · 2025

Read Human-like compositional learning of visually-grounded concepts using synthetic environments by Zijun Lin, M Ganesh Kumar, Cheston Tan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-06618.txt.

Introduction

Zijun Lin, Ganesh Kumar, and Cheston Tan watch an agent learn word-built visual ideas in a synthetic 3D world, as a child might. Word chains let people split a new phrase and find its match in sight, a flexibility no stock learner quite shows. Their 2025 world tells the agent a phrase, with thing, trait, and place words, and pays it to reach the right set. Word mixes run vast, and near-miss scenes pay nothing.

Plain trial learning grips the count words well yet trips on place words like above and below. A step-wise school path, the human way, cuts learning time by 15 percent on count words and cracks place words wide open. Trained on one word kind, the agent still splits fresh chains at test. Mixed worlds can thus teach flexible thought, they hold.

Catalog record

CREATORZijun Lin, M Ganesh Kumar, Cheston Tan
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE31 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-06618

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Human-like compositional learning of visually-grounded concepts using synthetic environments Zijun Lin, M Ganesh Kumar, Cheston Tan Abstract The compositional structure of language enables humans to decompose complex phrases and map them to novel visual concepts, showcasing flexible intelligence. While several algorithms exhibit compositionality, they fail to elucidate how humans learn to compose concept classes and ground visual cues through trial-and-error. To investigate this multi-modal learning challenge, we designed a 3D synthetic environment in which an agent learns, via reinforcement, to navigate to a target specified by a natural language instruction. These instructions comprise nouns, attributes, and critically, determiners, prepositions, or both. The vast array of word combinations heightens the compositional complexity of the visual grounding task, as navigating to a blue cube above red spheres is not rewarded when the instruction specifies navigating to “some blue cubes below the red sphere”. We first demonstrate that reinforcement learning agents can ground determiner concepts to visual targets but struggle with more complex prepositional concepts. Second, we show that curriculum learning—a strategy humans employ—enhances concept learning efficiency, reducing the required training episodes by 15% in determiner environments and enabling agents to easily learn prepositional concepts. Finally, we establish that agents trained on determiner or prepositional concepts can decompose held-out test instructions and rapidly adapt their navigation policies to unseen visual object combinations. Leveraging synthetic environments, our findings demonstrate that multi-modal reinforcement learning agents can achieve compositional understanding of complex concept classes and highlight the efficacy of human-like learning strategies in improving artificial systems’ learning efficiency. 1 Introduction Humans describe the world by combining linguistic parts-of-speech (concept classes) such as adjectives and nouns. As visual complexity increases, we use more words ( Sun & Firestone, 2022 ) and additional concept classes to enhance description accuracy ( Elsner et al., 2018 ; Qiao et al., 2020 ) . For example, determiners are essential for describing object quantity and ownership ( Lee et al., 2023 ) , while prepositions are critical for expressing spatial and temporal relationships between objects ( Agrawal et al., 2023 ) . In contrast, pretrained vision-language models struggle to ground visual objects to complex linguistic concepts and flexibly compose this knowledge to solve novel scenarios ( Kamath et al., 2023 ; Shen et al., 2023 ) . This gap restricts the alignment between human and machine communication for collaboration. Children typically master the use of determiners and prepositions by age three ( Abu-Akel et al., 2004 ; Brown, 1973 ) , applying these concepts to describe novel situations and generalizing beyond their experiences ( Tomasello, 1987 ; Washington & Naremore, 1978 ) . They learn incrementally, beginning with simple terms and advancing to complex concepts ( Valian, 1986 ; Gleason & Ratner, 2022 ; Richards, 1984 ) . The ability to compose linguistic elements for detailed descriptions, such as “some red balls above the green box,” is a skill even advanced vision-language models lack ( Okawa et al., 2024 ) . Tailoring learning curricula to individual pace and knowledge remains an ongoing challenge, both for humans and machine learning systems ( Bengio et al., 2009 ; Graves, 2008 ; Soviany et al., 2022 ; Wang et al., 2021 ; Narvekar et al., 2020 ) . In this work, we developed several 3D synthetic environments to study the visuo-linguistic grounding problem in a reinforcement learning setting, similar to how children could learn these concepts. A trial-and-error based learning environment facilitates artificial agents to learn from experience, a requisite to continually improve alignment during human-machine interaction. Specifically, we investigate the influence of a curriculum in improving compositional learning to ground determiner and preposition concept classes to visual objects. Notably, our synthetic environments allow precise control over the generation of training and held-out test sets while ensuring balanced data across classes, a challenge with real-world datasets. Leveraging on synthetic environments, we make the following contributions: • We demonstrate that reinforcement learning agents, when trained naively to maximize rewards, can ground descriptive attributes (determiners) to single objects but struggle to ground relational attributes between objects (prepositions). • We show that agents cannot generalize determiners and prepositions to new combinations without incremental curriculum training from simple to complex tasks. • We establish that agents can reason compositionally about novel I.I.D. and O.O.D. test instructions and navigate to unknown visual objects after learning determiner and preposition concepts. 2 Synthetic Environments for Grounding This section describes the synthetic environments designed for agents to learn to ground visual objects to determiner and preposition linguistic concepts by trial-and-error learning. 2.1 Environment Design Figure 1: Example environments, each with four target options. A reward of +10 is given when the agent navigates to the target matching the instruction. Punishments of -1, -3, and -10 are incurred for hitting a wall, reaching the wrong target, or failing to reach the correct target, respectively. A : Agent’s view of the determiner ( D ) environment. B : Agent’s view of the preposition ( P ) environment. C : Close-up view of the target object in P . D : Agent’s view of the combined determiner and preposition ( D+P ) environment. As shown in Fig. 1 , we built three synthetic 3D environments to enable the agents to learn four concepts — shape, color, determiner and preposition. While our primary emphasis is on determiner and preposition, the agents also need to acquire knowledge of color and shape throughout the learning process. Specifically, objects come in random colors (red, green, blue, yellow and black) and shapes (capsule, cube, cylinder, prism and sphere), each chosen from a set of 5. The target object is randomly placed in one of four predetermined locations within a rectangular room, as illustrated in Fig. 1 . Additionally, the positions of the individual objects within each location are randomized within a 3×3×3 block. This creates numerous visual representations that are constrained by the description of the instruction. The room layout remains constant and includes fixed visual landmarks like a door, window, shelf, and a reference man. The first-person perspective is simulated through a Unity-based camera, capturing the environmental dynamics in RGB images. These images serve as the visual input for the agent. Additionally, the environment generates textual instructions describing the target object, which are provided to the agent as language input. Details of each environment follows below. 2.2 Determiner Environment In the determiner environment ( D ), agents learn eight determiners: “A”, “Few”, “Some”, “Many”, “This”, “That”, “These”, and “Those”. Instructions follow the form “Determiner + Object(s)”. For example, “ few yellow cubes” or “ those green cylinders” (Fig. 1 A). The first four determiners describe object quantity: “A” denotes a single object, 2–3 objects as “Few”, 4–6 as “Some”, and 7–9 as “Many”. The last four determiners depend on proximity to a reference man (Fig. 1 A): “This” and “These” refer to closer objects (single or multiple 2 \sim 9), while “That” and “Those” refer to farther objects. Non-targets are designed to test grounding: (1) same determiner, random color/shape; (2) same color/shape, random determiner; (3) random determiner, color, and shape. For example, given the target instruction “ Many red cylinder”, non-target visual objects could be “ Few red cylinder”, “Many blue cube ”, and “ A green prism ”. 2.3

Read the complete text