Overcoming Multi-step Complexity in Multimodal Theory-of-Mind Reasoning: A Scalable Bayesian Planner
Chunhui Zhang, Zhongyu Ouyang, Kwonjoon Lee, Nakul Agarwal, Sean Dae Houlihan, Soroush Vosoughi, Shao-Yuan Lo · 2025
Read Overcoming Multi-step Complexity in Multimodal Theory-of-Mind Reasoning: A Scalable Bayesian Planner by Chunhui Zhang, Zhongyu Ouyang, Kwonjoon Lee, Nakul Agarwal, Sean Dae Houlihan, Soroush Vosoughi, Shao-Yuan Lo (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-01301.txt.
Introduction
Chunhui Zhang and co-authors present a planner that helps AI systems reason about human minds in a 2025 paper. Theory of mind means inferring beliefs, desires, and intentions, the core of social understanding. Past computer methods leaned on fixed workflows or heavy fine-tuning. They scaled poorly when scenes mixed images and text, and they faltered as tasks grew harder.
The planner breaks the reasoning into stepwise Bayesian updates, the math of revising beliefs with new evidence. A small model handles the mind-specific odds, then hands its work to a much larger model that adds social and world knowledge. This weak-to-strong hand-off spans models from modest to very large sizes. On multimodal benchmarks, including scenes never seen in training, the method beat leading rivals by several points of accuracy.
Catalog record
| CREATOR | Chunhui Zhang, Zhongyu Ouyang, Kwonjoon Lee, Nakul Agarwal, Sean Dae Houlihan, Soroush Vosoughi, Shao-Yuan Lo |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 80 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-01301
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Overcoming Multi-step Complexity in Multimodal Theory-of-Mind Reasoning: A Scalable Bayesian Planner Chunhui Zhang, Zhongyu Ouyang, Kwonjoon Lee, Nakul Agarwal, Sean Dae Houlihan, Soroush Vosoughi, Shao-Yuan Lo Abstract Theory-of-Mind (ToM) enables humans to infer mental states—such as beliefs, desires, and intentions—forming the foundation of social cognition. Existing computational ToM methods rely on structured workflows with ToM-specific priors or deep model fine-tuning but struggle with scalability in multimodal environments. They remain trapped within the gravitational pull of multi-step planning complexity, failing to generalize as task demands increase. To overcome these limitations, we propose a scalable Bayesian ToM planner. It breaks down ToM complexity into stepwise Bayesian updates. Meanwhile, weak-to-strong control specializes smaller LMs to refine ToM-specific likelihood estimation, transferring their ToM reasoning behavior to larger LMs (7B to 405B) for social and world knowledge integration. This synergistic approach enables scalability, aligning large-model inference human mental states with Bayesian principles. Extensive experiments demonstrate a 4.6% improvement in accuracy over state-of-the-art methods on multimodal ToM benchmarks, including unseen scenarios, establishing a new standard for modeling human mental states in complex environments. † † affiliationnotice: See the contributions of student and senior authors. 1 Introduction Theory-of-Mind (ToM) is a cornerstone of human social cognition, enabling individuals to attribute and infer mental states in themselves and others. This capacity underpins the recognition that others may have perspectives distinct from one’s own ( Dennett 1988 ; Gopnik & Wellman 2012 ) . Similarly, ToM tasks in AI involve perceiving observable cues—such as an agent’s actions and surrounding context—to predict their goals and beliefs. Equipping AI systems with ToM capabilities could unlock their potential for human-like social understanding and interactions ( Lake et al. 2017 ; Wu et al. 2021 ; Ma et al. 2023 ) . Existing approaches to ToM reasoning follow two main strategies: (i) structured planning workflows with ToM-specific priors ( Baker et al. 2017 ; Jara-Ettinger 2019 ; Shu et al. 2021 ) , and (ii) integrating these priors into language models (LMs) via specialized training ( Rabinowitz et al. 2018 ; Shu et al. 2021 ; Sclar et al. 2022 ; Jin et al. 2024 ) . The challenges extends beyond method designs—ToM environments demand multimodal reasoning, integrating visual, textual, and contextual inputs into coherent mental state inferences. Understanding and predicting the goal of agent planning behaviors require models to reason across multiple steps while integrating multimodal cues. Figure 1 : Comparison of models on planning tasks in VirtualHome (a simulator used in multimodal ToM). As planning steps increase, smaller models (e.g., Llama3.1-8B, 70B) and inference-time scaling (e.g., o1-mini, CoT) fail to sustain accuracy. Only larger models (e.g., Llama3.1-405B) maintain performance, demonstrating that sustaining accuracy requires model scaling. However, these challenges introduce generalization barriers, making it unclear whether standard approaches can scale effectively. To examine these challenges, Fig. 1 compares the performance of various models in planning tasks with the physical simulator used by multimodal ToM benchmarks. While inference-time scaling methods—such as CoT reasoning, o1 systems, and fine-tuning experts—provide incremental improvements, they fail to maintain accuracy as task complexity increases. Only larger models, such as Llama3.1 405B, sustain performance over increasing planning steps, while smaller LMs exhibit rapid accuracy degradation. These scalability challenges stem from two fundamental factors: First , a reasoning boundary exists, limiting the number of effective reasoning steps regardless of the method used. Prior studies ( Chen et al. 2024 ; Zhang et al. 2024 ; Ye et al. 2025 ; Gao et al. 2025 ) show that CoT reasoning and o1/r1-like scaling methods plateau in effectiveness as task complexity grows, leading to diminishing returns. Second , multimodal ToM reasoning relies heavily on integrating broader social and world knowledge , which correlates strongly with model scale. Unlike closed-form logic tasks, ToM reasoning in complex environments requires grounding in extensive, diverse contexts. As shown in Zhang et al. 2024 ; Yu et al. 2024 ; Sun et al. 2024 ; Gao et al. 2024 ; Yuan et al. 2025 ; Diao et al. 2025b ; Zhang et al. 2025b , smaller LMs struggle to generalize effectively due to their limited pretraining capacity to encode and utilize world knowledge. These findings emphasize that simple fine-tuning or inference-time scaling alone is insufficient to improve ToM reasoning at scale. Instead, sustained performance requires both structured frameworks and models with expansive generalization capabilities, as demonstrated by Fig. 1 , deviating from the existing methods. Motivated by these scalability challenges, we introduce a more scalable Bayesian ToM solution that directly addresses inference-time reasoning complexity through Bayesian Inverse Planning (BIP) ( Baker et al. 2007 ; Baker et al. 2009 ; Baker et al. 2017 ; Shum et al. 2019 ; Jin et al. 2024 ) and overcomes world knowledge dependency by scaling LMs up to 405B for ToM-specific generalization. First , BIP mitigates the reasoning boundary by decomposing multimodal ToM reasoning into modular, stepwise Bayesian updates —such as state transitions, belief updates, and action likelihoods. This structured framework refines beliefs and hypotheses iteratively, ensuring tractability even in complex environments. Second , overcoming the reliance on broad social and world knowledge requires scaling to larger LMs , as illustrated in Fig. 1 . Prior approaches ( Jin et al. 2024 ) relied on smaller LMs for likelihood estimation, but their limited capacity hindered generalization in rich ToM settings. Our weak-to-strong control mechanism enables smaller, post-trained LMs to specialize in ToM-specific tasks and transfer these learned behaviors to larger LMs (up to 405B ) during inference. This innovation allows the larger LM to serve as the primary policy model , leveraging its extensive world knowledge while maintaining Bayesian consistency for stable and interpretable reasoning. The theoretical foundation of our approach is established by Theorem 1 , which formalizes its effectiveness through KL divergence analysis . Empirical results demonstrate that our scalable solution enhances the generalizability of Bayesian inference and achieves a 4.6% accuracy improvement over state-of-the-art methods on multimodal ToM tasks, including unseen scenarios, setting a new standard for modeling ToM in complex environments. 2 Preliminaries Behaviour modelling: a Markov decision process formulation The behaviour of an agent can be formulated as a forward generative model based on a Partially Observable Markov Decision Process (POMDP), defined by the tuple \langle S,A,\mathcal{T},G,R,\Omega,O,\gamma\rangle ( Kaelbling et al. 1998 ; Jin et al. 2024 ) . Here, s^{t}\in S and a^{t}\in A represent the state and action at time t , respectively. \mathcal{T}(s^{t}|s,a) denotes the state transition probabilities. The goal g\in G determines the reward r^{t}=R(s^{t},a^{t},g) . The agent’s observation o^{t}\in\Omega is obtained via the observation function o^{t}=O(s^{t}) . The discount factor is \gamma\in(0,1] . Crucially, the agent’s belief, b(s) , is a probability distribution over the state. This belief is dynamically updated during belief evolution P(b^{\tau}\mid b^{\tau-1},s^{\tau}) , where b(s) is factorized into probabilities over the possible locations of individual objects. In our practice, b^{0} is initialized as all possible object locations at the start. At each step \tau , if an object is observed inside a container, b^{\tau} is updated to include the container; otherwise, unobserved locations are removed from b^{\tau} . P(b^{\tau}|b^{\tau-1},s^{\tau}) can be approximated using likelihoods from an LM. Inverse inference: from observed behaviours to mental states Bayesian inverse planning (BIP) infers an agent’s goals and beliefs