Inter-environmental world modeling for continuous and compositional dynamics

Kohei Hayashi & Masanori Koyama, Julian Jorge Andrade Guerreiro · 2025

Read Inter-environmental world modeling for continuous and compositional dynamics by Kohei Hayashi & Masanori Koyama, Julian Jorge Andrade Guerreiro (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-09911.txt.

Introduction

Kohei Hayashi, Masanori Koyama, and Julian Jorge Andrade Guerreiro present a world model that can work across settings in work from 2025. World models let a system predict what will happen next so it can plan actions.

Their method, called WLA, learns smooth control signals that work in many settings at once. It uses ideas from Lie group theory and a part-based image encoder. Trained on video frames alone, with few or no action labels, the model adapted fast to new settings with new action sets. Tests used both made-up tasks and real-world data.

Catalog record

CREATORKohei Hayashi & Masanori Koyama, Julian Jorge Andrade Guerreiro
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE56 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-09911

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Inter-environmental world modeling for continuous and compositional dynamics Kohei Hayashi & Masanori Koyama, Julian Jorge Andrade Guerreiro Abstract Various world model frameworks are being developed today based on autoregressive frameworks that rely on discrete representations of actions and observations, and these frameworks are succeeding in constructing interactive generative models for the target environment of interest. Meanwhile, humans demonstrate remarkable generalization abilities to combine experiences in multiple environments to mentally simulate and learn to control agents in diverse environments. Inspired by this human capability, we introduce World modeling through Lie Action (WLA), an unsupervised framework that learns continuous latent action representations to simulate across environments. WLA learns a control interface with high controllability and predictive ability by simultaneously modeling the dynamics of multiple environments using Lie group theory and object-centric autoencoder. On benchmark synthetic and real-world datasets, we demonstrate that WLA can be trained using only video frames and, with minimal or no action labels, can quickly adapt to new environments with novel action sets. 1 Introduction Originally proposed as a framework to support automatic planning and decision-making, world models ( Ha & Schmidhuber, 2018 ) predict future states conditioned on actions, allowing agents to anticipate the consequences of their interactions. There have been extensive efforts in the development of world models and related learning methods. Many of them rely on discrete representations of actions and/or images ( Hafner et al., 2023 ; Hu et al., 2023 ) in order to leverage the techniques of autoregressive inference that are extensively developed in language modeling. Recent works such as Genie ( Bruce et al., 2024 ) and LAPO ( Schmidt & Jiang, 2023 ) have advanced the field by learning interactive world models with few/no explicit action labels. These models have shown promising results in capturing complex dynamics and discrete actions in specific environments. Humans, however, possess an extraordinary ability to generalize learned skills across diverse environments by leveraging continuous and compositional representations of actions. For example, after mastering basic movements in a few 2D action-adventure games, a person can quickly adapt to a game of a different type(e.g. Pac-Man) by leveraging the knowledge of common 2D concepts such as moving in continuous directions ( Schmidt, 1975 ; Poggio & Bizzi, 2004 ) . Continuous action representations allow for smooth transitions and fine-grained control, while compositionality enables complex actions to be built from simpler primitives, facilitating generalization and transfer learning ( Flash & Hochner, 2005 ; Botvinick & Plaut, 2004 ) . Inspired by this human capability, we hypothesize that, in order to learn an interactive world model that generalizes across environments, it is essential to construct an environment-agnostic simulator that embraces continuous and compositional action representations. In this paper, we introduce World modeling through Lie Action (WLA) , an unsupervised framework that aims to learn such a simulator. Our approach models the nonlinear and continuous transition operators in the observation space with a Lie group that acts linearly on partitioned latent vector spaces, where each partition corresponds to different objects and fundamental action axes. This allows us to capture the continuous and compositional nature of actions, enabling seamless generalization across different environments. Our method extends the capabilities of existing world models by introducing the continuity and compositionality in the form of Lie group structure. We demonstrate the effectiveness of WLA on 2D game and 3D robotics environments in terms of generalization and adaptability. Figure 1: The relation between observation space dynamics and latent space dynamics with Lie group action. We elaborate the design of \Phi,\Psi and the implementation of \mathcal{F}_{\Phi,\Psi} in Section 4 . 2 Problem Setting Before introducing our framework, we formalize the problem that WLA is meant to solve. WLA is a mathematical framework for learning an interface to interact with a family of diverse environments sharing common basic rules of composition and continuity, such as a family of 2D games. To further elaborate, we introduce our definition of environment . 2.1 Environment Definition 1 . An environment \mathcal{E} is a pair ( \mathcal{X},\mathcal{T} ) where \mathcal{X} is a space of observations, and \mathcal{T} is a set of nonlinear transition operators on \mathcal{X} . Let us use \mathcal{X}_{\mathcal{E}} and \mathcal{T}_{\mathcal{E}} to denote the space of observations and the set of transitions of \mathcal{E} , respectively. Thus, if x:I\to\mathcal{X}_{\mathcal{E}} is a sample trajectory of \mathcal{E} with time domain I\subset\mathbb{R}_{+} , then for all t,t+\delta\in I , we have g_{t,\delta}(x(t))=x(t+\delta) for some g_{t,\delta}\in\mathcal{T}_{\mathcal{E}} . We assume that the dataset consists of a set of trajectories sampled from all environments \{\mathcal{E}_{j}\}_{j} . For example, in the family of simple 2D action games, the transitions in one environment may be related to jumping and running. In another environment, actions may contain rotations and and powerups. Strictly speaking, because g_{t,\delta} differs for each x , we should use the notation g_{t,\delta,x} to denote the transition of x(\cdot) , and in cases where all trajectories in \mathcal{E} can be enumerated, we should use the notation g_{t,\delta,i} to denote the transition of x^{i}(\cdot) . However, for brevity, we may use g_{t,\delta} in place of g_{t,\delta,i} with the understanding that it is a variable that differs across trajectories. Also, because not all the observation sequences in the real world are realized by a finite set of discrete actions like buttons, we do not assume in this definition that the transitions in the environment can be grounded on such actions (e.g., button presses). 2.2 Controller Interface Problem We aim to construct an interactive interface for the world model that approximates a given (set of) environment(s), which in this study is/are assumed deterministic. We propose constructing such an interface through an inter-environmental simulator like WLA. In what follows, we rephrase this goal with the definition of environment introduced above. If \mathcal{A} is a topological space of all possible instantaneous inputs to the interface, we informally define the Controller Interface Problem (CIP) for the environment \mathcal{E} as below: Problem . Using D_{\mathcal{E}}=\{x:I\to\mathcal{X}_{\mathcal{E}}\} sampled from \mathcal{E} , construct a controller map \displaystyle\textrm{Ctrl}_{\mathcal{E}}:\mathcal{X}\times\mathcal{A}\to\mathcal{T}_{\mathcal{E}} (1) for which there exists a map (a:I\to\mathcal{A}) such that \textrm{Ctrl}_{\mathcal{E}}(x[0,t),a[0,t))(x(t))=x(t+dt) for all trajectories x\in D_{\mathcal{E}} , where t+dt is the infinitesimal future of t , and an abuse of notation was used so that \textrm{Ctrl}_{\mathcal{E}} takes as input the time series (x,a) up to t\in I , denoted as x[0,t) and a[0,t) . 1 1 1 Please see Section A for more formal definition of the controller map. In more informal words, CIP is a problem of building a controller interface that approximates the rule of dynamics in a target environment. There are two settings for CIP, which we call (1) unstructured CIP and (2) structured CIP. In unstructured CIP, the problem includes that of specifying the space of \mathcal{A} itself. In structured CIP, the problem is to find the map with \mathcal{A} that is prespecified, for example, by hardware restrictions. When \mathcal{A} is chosen to be finite, unstructured CIP may be analogous to building a set of controller buttons for a game \mathcal{E} , and structured CIP may be analogous to finding the correspondence map between a given controller input and transitions in the environment. Once the CIP is solved for \mathcal{E} , the user may iteratively apply the action sequence of choice to x\in\mathcal{X} to generate

Read the complete text