Natural Building Blocks for Structured World Models: Theory, Evidence, and Scaling

Lancelot Da Costa 1,3 Sanjeev Namjoshi 1 Mohammed Abbas Ansari 2 Bernhard Schölkopf 3,4 1 VERSES AI Research Lab 2 University of Tübingen 3 ELLIS Institute, Tübingen 4 MPI for Intelligent Systems, Tübingen · 2025

Read Natural Building Blocks for Structured World Models: Theory, Evidence, and Scaling by Lancelot Da Costa 1,3 Sanjeev Namjoshi 1 Mohammed Abbas Ansari 2 Bernhard Schölkopf 3,4 1 VERSES AI Research Lab 2 University of Tübingen 3 ELLIS Institute, Tübingen 4 MPI for Intelligent Systems, Tübingen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-02091.txt.

Introduction

Lancelot Da Costa, Sanjeev Namjoshi, Mohammed Abbas Ansari, and Bernhard Schölkopf argue that research on world models, the internal simulators AI systems use to predict their surroundings, has become fragmented. Every group builds its own architecture from scratch. Their proposal defines a world model as a layered composition of two natural building blocks: discrete processes for logic and symbols, and continuous processes for physics and motion.

Hidden Markov models and switching linear dynamical systems serve as the two blocks, gaining planning ability when actions are added. Fixing most of the architecture leaves only four depth parameters to search, avoiding an explosion of structure choices. The authors review results in generation and planning from pixels that rival neural approaches while staying interpretable, and flag scalable joint learning of structure and parameters as the key open problem.

Catalog record

CREATORLancelot Da Costa 1,3 Sanjeev Namjoshi 1 Mohammed Abbas Ansari 2 Bernhard Schölkopf 3,4 1 VERSES AI Research Lab 2 University of Tübingen 3 ELLIS Institute, Tübingen 4 MPI for Intelligent Systems, Tübingen
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-02091

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Natural Building Blocks for Structured World Models: Theory, Evidence, and Scaling Lancelot Da Costa 1,3 Sanjeev Namjoshi 1 Mohammed Abbas Ansari 2 Bernhard Schölkopf 3,4 1 VERSES AI Research Lab 2 University of Tübingen 3 ELLIS Institute, Tübingen 4 MPI for Intelligent Systems, Tübingen Abstract The field of world modeling is fragmented, with researchers developing bespoke architectures that rarely build upon each other. We propose a framework that specifies the natural building blocks for structured world models based on the fundamental stochastic processes that any world model must capture: discrete processes (logic, symbols) and continuous processes (physics, dynamics); the world model is then defined by the hierarchical composition of these building blocks. We examine Hidden Markov Models (HMMs) and switching linear dynamical systems (sLDS) as natural building blocks for discrete and continuous modeling—which become partially-observable Markov decision processes (POMDPs) and controlled sLDS when augmented with actions. This modular approach supports both passive modeling (generation, forecasting) and active control (planning, decision-making) within the same architecture. We avoid the combinatorial explosion of traditional structure learning by largely fixing the causal architecture and searching over only four depth parameters. We review practical expressiveness through multimodal generative modeling (passive) and planning from pixels (active), with performance competitive to neural approaches while maintaining interpretability. The core outstanding challenge is scalable joint structure-parameter learning; current methods finesse this by cleverly growing structure and parameters incrementally, but are limited in their scalability. If solved, these natural building blocks could provide foundational infrastructure for world modeling, analogous to how standardized layers enabled progress in deep learning. Figure 1: Discrete and continuous building blocks demonstrating four types of structural depth. HMM (discrete) and recurrent sLDS (continuous) layers. Temporal depth unfolds dynamics over time, hierarchical depth creates abstraction levels, factorial depth separates independent variation sources, and generalized depth encodes latent memory through higher-order dynamics. 1 Introduction World modeling ( Ding et al. 2025 ) suffers from fragmentation—each research group builds custom architectures with little code reuse, mirroring deep learning before frameworks such as Caffe and Keras standardized layers and composition rules ( Chollet et al. 2015 ) . While neural world models ( Hafner et al. 2020 ; Hafner et al. 2019 ) achieve strong performance, they operate as black boxes; on the other hand, structured models offer interpretability but traditionally lack expressiveness (related work in Appendix A ). We propose that world models should be built from the fundamental stochastic processes they must capture. Most phenomena in the natural and complex systems sciences are modeled through Markov chains, stochastic differential equations (SDEs) (particularly diffusion processes), and their hierarchical compositions (multi-scale phenomena). We propose that the core building blocks for world modeling should be learnable approximations to these processes. Here, we focus on natural candidates: HMMs ( Rabiner & Juang 1986 ) and sLDS ( Fox et al. 2008 ) , which transform into POMDPs and controlled sLDS when augmented with actions. Our contributions are: (1) proposing world models be built from learnable approximations of the fundamental stochastic processes encountered in nature, (2) natural building blocks unifying passive modeling and active control, (3) four types of structural depth finessing combinatorial explosion of structure learning while retaining expressiveness, (4) reviewing empirical evidence: multimodal generation without neural networks, behavioral/neural modeling, planning from pixels competitive with state-of-the-art, and (5) identifying the core challenge for this technology to become foundational infrastructure for world modeling: scalable joint structure-parameter learning. 2 Natural Building Blocks for World Models 2.1 Building Blocks We propose two complementary building blocks and describe their hierarchical compositions to capture the essential stochastic processes in world modeling: 1. Hidden Markov Models (HMMs). HMMs capture discrete partially-observed dynamics, providing a fundamental model for time series (see Appendix B ). When augmented with control states via generalized depth (Section 2.2 ), they transform into POMDPs for agent modeling. These naturally handle logical reasoning, symbolic manipulation, and categorical decisions. While extremely expressive through quantization, they scale poorly to high dimensions. 2. Switching linear dynamical systems (sLDS). sLDS provide a continuous alternative to HMMs, approximating nonlinear dynamics through switching linear systems (see Appendix B ). Adding control states through generalized depth enables continuous control. A recent key innovation is the recurrent connection where continuous states influence discrete switching (i.e., rsLDS; Linderman et al. 2017 )—important for contact physics, such as bouncing. While less universal than HMMs (approximating SDEs), their continuous state-spaces scale naturally to higher-dimensional inputs. Hierarchical Composition. Higher-level states provide (stochastic) initial conditions for lower-level trajectories, enabling multi-scale modeling; from abstract planning to detailed execution. The framework supports discrete→discrete (logical hierarchies), discrete→continuous (plans to physics), and continuous→continuous (multi-scale physics) compositions. 2.2 Four Types of Structural Depth These building blocks support four complementary types of structural depth (Figure 1 ): 1. Temporal depth : Standard time unrolling (t=1,2,3,...) for sequential dynamics. 2. Hierarchical depth : Multiple abstraction levels, from high-level goals/concepts to detailed ones. 3. Factorial depth : Independent variation sources (e.g., object position, color, texture) that evolve separately. 4. Generalized depth : Memory by extending the latent space to incorporate higher-order latent variables; that is, velocity, acceleration, and higher-order motion, and their discrete counterparts. This extends classical HMMs and rsLDS to express non-Markovian (technically semi-Markovian) dynamics in latent space. These generalized latent states can optionally be defined as controllable enabling action: for example, controlling the first generalized order in an HMM makes it a POMDP, and in an rsLDS makes it a controlled rsLDS. 2.3 Inference, Learning & Structure learning The building blocks support efficient Bayesian inference through variational methods, providing uncertainty quantification over states and parameters ( Winn & Bishop 2005 ; Da Costa et al. 2020 ; Linderman et al. 2017 ) . However, de novo structure & Bayesian parameter learning in deep models remains a core challenge. Fast Structure Learning (FSL; Friston et al. 2024a ) tackles this challenge by incrementally growing structure and parameters jointly, however it is currently limited in scale. 3 Theoretical Features Expressiveness. HMMs with generalized depth express partially-observable, finite-memory (i.e., semi-Markovian) discrete processes and approximate their continuous counterparts through quantization. rsLDS approximate partially-observable nonlinear diffusion processes through recurrent switching linear dynamics. With generalized depth, rsLDS could approximate non-Markovian SDEs, e.g., ( Da Costa et al. 2025 ) , but this implementation remains future work (Section 5 ). Hierarchical composition extends this expressiveness to multi-scale phenomena (analysis in Appendix B ). Figure 2: Multimodal generative modeling without neural networks. Video (top) and jazz audio generation (bottom) using hierarchical generalized HMMs with FSL. Adapted from ( Friston et al. 2024a ) . Avoiding Combinatorial Explosion. Traditional structure learning ( Koller & Friedman 2009 ) faces super-exponential growth ( >10^{12} unlabeled DAGs for just 10 variables). We finesse this by largely fixing the model structure (hierarchical HMMs/rsLDS) and searching only over four depth parameters (Figure 1 ), reducing the search space dramatically while maintaining expressiveness. 4 Empirical Evidence Multimodal Generation (Passive). Figure 2 shows coherent video and jazz audio generation without neural networks using hierarchical generalized HMMs and FSL ( Friston et al. 2024a ) , demonstrating multimodal expressiveness with interpretable components. Planning from Pixels (Active). Heins et al. 2025 demonstrates planning in hierarchical sLDS with model structure learned online with FSL in Atari-style environments, achieving performance competitive with state-of-the-art neural

Read the complete text