Context and Diversity Matter: The Emergence of In-Context Learning in World Models

*†Fan Wang, *Zhiyuan Chen, *Yuxuan Zhong, Sunjian Zheng, Pengtao Shao, Bo Yu, †Shaoshan Liu, Jianan Wang, Ning Ding, Yang Cao, Yu Kang · 2025

Read Context and Diversity Matter: The Emergence of In-Context Learning in World Models by *†Fan Wang, *Zhiyuan Chen, *Yuxuan Zhong, Sunjian Zheng, Pengtao Shao, Bo Yu, †Shaoshan Liu, Jianan Wang, Ning Ding, Yang Cao, Yu Kang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-22353.txt.

Introduction

Can a world model learn a new setting just from context, with no retraining? Fan Wang and 10 co-authors study that skill in models that predict how a scene will change. They split the skill into two parts. One part spots which setting the model faces. The other part learns the rules of that setting from the prompt.

The team derives error bounds for both parts. Tests then confirm that both parts do occur in practice. Long context and varied training settings help the skill grow. The bounds show why those two factors matter most for models that adapt on the fly.

Catalog record

CREATOR*†Fan Wang, *Zhiyuan Chen, *Yuxuan Zhong, Sunjian Zheng, Pengtao Shao, Bo Yu, †Shaoshan Liu, Jianan Wang, Ning Ding, Yang Cao, Yu Kang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE51 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-22353

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Context and Diversity Matter: The Emergence of In-Context Learning in World Models *†Fan Wang, *Zhiyuan Chen, *Yuxuan Zhong, Sunjian Zheng, Pengtao Shao, Bo Yu, †Shaoshan Liu, Jianan Wang, Ning Ding, Yang Cao, Yu Kang Abstract The capability of predicting environmental dynamics underpins both biological neural systems and general embodied AI in adapting to their surroundings. Yet prevailing approaches rest on static world models that falter when confronted with novel or rare configurations. We investigate in-context environment learning (ICEL), shifting attention from zero-shot performance to the growth and asymptotic limits of the world model. Our contributions are three-fold: (1) we formalize in-context learning of a world model and identify two core mechanisms: environment recognition and environment learning; (2) we derive error upper-bounds for both mechanisms that expose how the mechanisms emerge; and (3) we empirically confirm that distinct ICL mechanisms exist in the world model, and we further investigate how data distribution and model architecture affect ICL in a manner consistent with theory ♠ \spadesuit . These findings demonstrate the potential of self-adapting world models and highlight the key factors behind the emergence of ICEL, most notably the necessity of long context and diverse environments. * * footnotetext: Equal Contribution † † footnotetext: Corresponding to: [email protected], [email protected] $\spadesuit$ $\spadesuit$ footnotetext: Source code: https://github.com/airs-cuhk/airsoul/tree/main/projects/MazeWorld 1 Introduction The ability to predict future environmental states is crucial for reasoning and decision-making in both animals and humans. Inspired by this principle, constructing predictive models, especially the world model [ 1 ] , to forecast environmental dynamics and outcomes forms the foundation for enabling agents to plan effective decisions and behaviors [ 2 , 3 , 4 , 5 , 6 ] . Consequently, world models are widely applied in fields such as navigation [ 7 , 8 , 9 , 10 , 11 ] , autonomous driving [ 12 , 13 , 14 , 15 , 16 ] , robotics [ 17 , 18 , 19 , 20 , 21 , 22 ] , and are considered a cornerstone of embodied artificial intelligence. Despite their proven effectiveness across various applications, previous prediction frameworks largely rely on static world models optimized for zero-shot, few-shot, or instantaneous performance. In contrast, humans and animals achieve real-time adaptation [ 23 ] through predictive coding—a process where prediction errors drive attention, generate feedback, and motivate learning and adjustment [ 24 , 25 ] . For instance, when confronted with rare environments, humans experience surprise yet rapidly recalibrate their predictions for that setting, whereas static models continue to fail unless explicitly retrained on the relevant data. The ability to dynamically modify predictive mechanisms based on observational evidence, rather than relying solely on fixed parametric memory and external mapping modules, can effectively enable the model to adapt to environments unseen during training. This capability can be effectively addressed by In-Context Learning (ICL), as evidenced by recent advances in Large Language Models [ 26 ] . However, the potential of ICL for resolving plasticity in world models, particularly In-Context Environment Learning (ICEL), remains largely underexplored in current literature. Addressing this gap could further improve the generalization scope for embodied AI. We isolate two canonical mechanisms that let world models adapt in context. In-Context Environment Recognition (ICER) taps the model’s parametric memory of previously seen environments and uses the context only to identify which one is active. ICEL ignores prior settings and instead extracts the full environment from the context itself. By deriving error bounds for both modes, we show how their emergence depends on data distribution, including environment diversity, complexity, and trajectory length. To test the theory, we introduce L inear-attention L ong-context world model, L2World , which effectively model long sequences with high computational efficiency. Across cart-pole control and vision-based indoor navigation, we find that distributional properties and long-context capacity govern whether the model invokes ICER or ICEL. Despite the light image encoders and decoders of L2World, it sets a new state-of-the-art for long-sequence observation prediction in navigation, outperforming approaches that rely on heavy diffusion-based image backbones. These results underscore the importance and the promise of deliberately incentivizing ICEL through carefully designed datasets and model architectures to strengthen the asymptotic generalization capability of world models, rather than focusing solely on instant or zero-shot frame-level performance. 2 Related Work 2.1 Dynamic Prediction and World Models Dynamic models, also referred to as world models [ 1 , 27 ] , encompass probabilistic, physical, or generative frameworks that formalize an AI system’s environmental understanding [ 28 , 29 , 30 , 31 ] . These models predict future states by leveraging historical observations and play a pivotal role in advancing reinforcement learning (RL) methodologies and related domains. Specifically, they constitute foundational components in model-based RL [ 32 , 33 ] , enable simulations that facilitate agent learning through virtual experiences (thereby reducing reliance on direct environmental interaction) [ 34 , 2 , 5 , 35 , 36 ] , and serve as auxiliary tasks to augment model supervision [ 37 , 38 , 39 ] . World models demonstrate robustness in integrating multi-modal raw sensor data, including visual, textual, inertial, and tactile inputs. To mitigate challenges posed by high-dimensional sensory inputs, representation learning paradigms such as Generative Adversarial Networks (GANs) [ 40 ] and Variational Autoencoders (VAEs) [ 41 ] are widely utilized to compress raw data into compact latent spaces. Subsequent temporal modeling in these reduced dimensions is achieved through latent world model architectures [ 2 , 42 , 4 , 5 , 43 , 39 , 44 , 45 ] , which capture sequential dependencies and enable coherent long-term predictions. In navigation, early systems relied on traditional, hand-crafted pipelines such as SLAM. Recent work replaces these modules with generative world models, including diffusion [ 7 ] , VAE [ 9 ] , and RL-enhanced variants [ 46 , 10 ] , which reconstruct dynamics or simulate semantics [ 47 , 48 ] . However, most existing methods disregard continual adaptation, especially across episodes, leaving a persistent gap between zero-shot performance and lifelong operation. 2.2 In-Context Learning and Meta-Learning The approach based on parametric memory, which predominantly relies on gradient-based optimization or In-Weight Learning (IWL), has faced criticism for its lack of plasticity and the associated challenges it poses in continual learning scenarios [ 49 ] . Conversely, ICL has emerged as a pivotal capability in large language models [ 26 ] , facilitating generalization to novel tasks without the necessity of parameter fine-tuning. ICL leverages contextual memory for task solutions, rather than depending on parametric memory. The concept of ICL is not novel. Meta-learning [ 50 , 51 ] , which focuses on acquiring learning capabilities rather than mastering specific skills, utilizes well-curated environments or data instead of relying on large-scale, uncurated pre-training data. Nevertheless, the lack of well-structured, task-rich, and cost-efficient datasets continues to present a significant challenge. ICL has been utilized to encode a diverse array of learning mechanisms, including language learning [ 52 ] , regression [ 53 ] , Bayesian inference [ 54 ] , reinforcement learning [ 55 , 56 , 57 ] , and world models [ 58 , 59 ] , highlighting its resemblance to biological plasticity [ 60 ] . Yet existing work concentrates on

Read the complete text