Learning Local Causal World Models with State Space Models and Attention
Francesco Petri, Luigi Asprino, Aldo Gangemi · 2025
Read Learning Local Causal World Models with State Space Models and Attention by Francesco Petri, Luigi Asprino, Aldo Gangemi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-02074.txt.
Introduction
Francesco Petri, Luigi Asprino, and Aldo Gangemi test a state space model, a recent rival to the transformer. They ask whether it can learn not just how a toy world moves, but the causal rules behind that motion. A deep grasp of cause is what an agent needs for hard tasks in the real world.
Side by side with a like transformer, the state space model matched or beat it on the toy world. It learned the dynamics and a causal model at the same time. The authors see the result as a door to more work that leans on the strengths of such models and adds sharper causal sense to them.
Catalog record
| CREATOR | Francesco Petri, Luigi Asprino, Aldo Gangemi |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 25 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-02074
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Learning Local Causal World Models with State Space Models and Attention Francesco Petri, Luigi Asprino, Aldo Gangemi Abstract World modelling, i.e. building a representation of the rules that govern the world so as to predict its evolution, is an essential ability for any agent interacting with the physical world. Despite their impressive performance, many solutions fail to learn a causal representation of the environment they are trying to model, which would be necessary to gain a deep enough understanding of the world to perform complex tasks. With this work, we aim to broaden the research in the intersection of causality theory and neural world modelling by assessing the potential for causal discovery of the State Space Model (SSM) architecture, which has been shown to have several advantages over the widespread Transformer. We show empirically that, compared to an equivalent Transformer, a SSM can model the dynamics of a simple environment and learn a causal model at the same time with equivalent or better performance, thus paving the way for further experiments that lean into the strength of SSMs and further enhance them with causal awareness. a]Institute for Cognitive Sciences and Technologies, National Research Council, Rome, Italy b]Sapienza University of Rome, Italy c]University of Bologna, Italy keywords World modelling, causal discovery, state-space models, object-oriented representations † † authorinfo: *Corresponding author: [email protected] 1 INTRODUCTION World modelling is the ability of an artificial agent to build an internal representation of the world in which it operates. This representation is employed by the agent to forecast the evolution of the world. The problem of building a world model spans many branches of artificial intelligence, such as planning and reinforcement learning [ 1 ] , physics modelling and reasoning [ 2 ] , and robotics [ 3 ] . An accurate representation of the world allows building simulations that, in turn, enable practitioners to gather additional data and test the performance of an artificial agent without interacting with their environment. This is convenient because interacting with an agent’s environment can be time-consuming, risky due to possible failures of physical components, and sometimes even impossible due to the potential unavailability of the environment (e.g. experimenting with the exploration of Mars by a rover). Recent efforts have produced highly accurate and efficient world models in a variety of environments [ 1 , 4 ] . In particular, the State-Space Model (SSM) is a new type of neural architecture that has the ability to remember past observations over extremely long time horizons, allowing it to model large environments and long-range interactions [ 5 ] . In addition, it has recently been shown that learning a causal model of the environment, i.e. a model that describes the causal relationships between objects in the environment, is necessary to achieve a good level of robustness to changes in the world [ 6 ] . In this paper, we investigate whether SSM can learn causal models and the benefits of doing so in the world modelling task. Following the SSM architectural style, we propose a novel model called Sparse Slot State Space Model (S2-SSM) which features a sparsity regularization technique originally designed for the Transformer architecture [ 7 ] to learn a local causal model of the interactions between the objects at each moment in time. We experiment with the world modelling and causal discovery tasks in a synthetic environment. We test the performance of S2-SSM against a Transformer-based architecture, i.e. SPARTAN [ 7 ] . We report preliminary results showing that S2-SSM is able to learn an accurate causal model. This contributes to enhancing the world modelling capability of the model and surpassing the performance of the Transformer architecture, or in the worst case scenario, matching it. 2 Related work 2.1 Causal Discovery Integrating a general theory of causation into machine learning methods is an ongoing research effort. Learning a causally informed representation of the data would enable a deeper understanding of the interactions between the entities described in the data, whether the domain is a physical world populated with objects [ 2 ] or something more abstract like language [ 8 ] , which would increase the interpretability of the resulting model. Several formal structures have been developed to study causal relationships between variables. Examples are Bayesian networks, which are widely used method for causal learning through statistical analysis [ 9 ] , and Structural Causal Models, which are expressed as a graph where each node represents a causal variable and the edges indicate cause-effect relations [ 10 ] . In particular, graph-based models are frequently combined with the attention operation in machine learning. The attention operation determines the relative importance (called weight) of each component in a sequence relative to the other components in that sequence. The attention weights function as a metric for the degree of correlation between two variables, thereby indicating the probability of a causal relationship between them within the graph model [ 11 , 12 ] . Transformers is a neural architecture, which has been applied in multiple domains, that relies on a multi-head attention mechanism [ 13 ] . In particular, Transformers have been utilised in the context of learning graph causal models [ 7 ] . In this study, we further explore the use of attention weights as indicators of causal relationships between variables in a different architectural framework, namely the state space models, which is particularly relevant to the task of world modelling. 2.2 World Modelling The world modelling problem has received tremendous attention in the last few years. In reinforcement learning, being able to simulate the dynamics of the environment is especially useful, because it enables the agent to act and learn in its own simulated world without paying the cost of interacting with the “real” environment. The Dreamer algorithm [ 14 ] has been relatively influential on the topic, including applications in robotic domains [ 3 ] , where real-world interaction has the unfortunate potential of breaking expensive equipment, in addition to time costs. Additional approaches in reinforcement learning include solutions based on causal discovery and reasoning [ 11 ] , which aim at learning a causal model of the environment to better understand the interactions between the agent and the world, and even provide an explanation for the actions taken by the agent. Many Transformer-based approaches have been studied, due to their generally good performance in different tasks and the sample efficiency they provide in this specific problem [ 1 , 4 , 7 ] . Recently, the newer State-Space Model architecture (SSM) has also been applied to this problem [ 5 ] . This sequence model based on principles from control theory has already been shown to have some advantages over Transformers, in particular a high inference efficiency and a great memory capacity, and they achieve similar levels of performance while not suffering from the quadratic memory cost that notoriously afflicts Transformers [ 15 , 5 ] . We particularly note the very recent SlotSSM [ 16 ] , which exploits object-oriented representations and attention to model the interaction between objects in a scene. Despite this interest, the adeptness of SSMs for causal learning has not been extensively assessed yet. This work aims to contribute in this direction.