Effects of Theory of Mind and Prosocial Beliefs on Steering Human-Aligned Behaviors of LLMs in Ultimatum Games

Neemesh Yadav, Yihuai Lan, Shan Dong, Mai Hieu Hien, Palakorn Achananuparp, Jing Jiang, Ee-Peng Lim · 2025

Read Effects of Theory of Mind and Prosocial Beliefs on Steering Human-Aligned Behaviors of LLMs in Ultimatum Games by Neemesh Yadav, Yihuai Lan, Shan Dong, Mai Hieu Hien, Palakorn Achananuparp, Jing Jiang, Ee-Peng Lim (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-24255.txt.

Introduction

Neemesh Yadav and co-authors test whether mind-reading prompts make language model bargainers act more like people, in a 2025 paper. Their arena is the ultimatum game, where one side offers a split and the other can refuse and leave both with nothing. Agents were seeded with greedy, fair, or selfless beliefs.

Across 2,700 simulated games, theory-of-mind reasoning, picturing the rival's thoughts, improved how well agents matched human norms and how steady their choices were. Plain reasoning models lagged behind mind-augmented ones. Each game role gained most from a different depth of mind reading. Agents mostly stuck to their seeded beliefs, though greedy and selfless ones still missed human outcome norms. The study maps how social reasoning steers machine negotiation.

Catalog record

CREATORNeemesh Yadav, Yihuai Lan, Shan Dong, Mai Hieu Hien, Palakorn Achananuparp, Jing Jiang, Ee-Peng Lim
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE103 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-24255

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Effects of Theory of Mind and Prosocial Beliefs on Steering Human-Aligned Behaviors of LLMs in Ultimatum Games Neemesh Yadav, Yihuai Lan, Shan Dong, Mai Hieu Hien, Palakorn Achananuparp, Jing Jiang, Ee-Peng Lim Abstract Large Language Models (LLMs) have shown potential in simulating human behaviors and performing theory-of-mind (ToM) reasoning, crucial for complex social interactions. We investigate ToM reasoning’s role in aligning agentic behaviors with human norms in negotiation tasks, using the ultimatum game as our referenced task. We initialized LLM agents with different prosocial beliefs (Greedy, Fair, Selfless) and reasoning methods (chain of thought and ToM reasoning of varying levels), examining their decision-making processes and outcomes across multiple LLMs, including reasoning models like o3-mini and DeepSeek-R1 Distilled Qwen 32B. We perform 2,700 simulations to show that ToM reasoning enhances behavioral alignment with human norms, decision-making consistency, and negotiation outcomes. Consistent with prior findings, reasoning LLMs exhibit limited capability compared to ToM-enhanced LLMs, with different game roles benefiting from different ToM orders. Agents almost always chose strategies consistent with their assigned beliefs, and doing so aligned better with human norms in nearly every role, yet Greedy and Selfless agents still fell short of their norms in outcomes. Human verification further revealed that Llama 3.3 70B produces reasoning most consistent with its actions and beliefs. Our findings advance understanding of ToM’s role in human-AI interaction and cooperative decision-making. The code used for our experiments can be found at https://github.com/Stealth-py/UltimatumToM . † † footnotetext: Revision note (October 2026). This version corrects the RQ4 belief–strategy analysis of the proceedings version (Section 5.4 , Figure 6 , Appendix C ). The original contrasts included belief–strategy combinations with no observations; corrected results show that belief-consistent strategies align better with human norms in nearly all cases. All other results are unchanged. 1 Introduction Interpersonal communication is fundamental to human interaction, and its effectiveness often hinges on our ability to understand and anticipate others’ perspectives—a cognitive capacity known as theory of mind (ToM) Premack and Woodruff (1978) . ToM enables individuals to infer mental states, such as beliefs, desires, and intentions Georgeff et al. (1999) , serving as a foundation of social cognition and interaction Astington and Jenkins (1995) . Recent advances in LLMs have prompted researchers to investigate whether these systems exhibit human-like ToM reasoning capabilities. State-of-the-art models like GPT-4 demonstrate promising performance on standard false-belief ToM benchmarks, such as the Sally-Anne and the Smarties tasks Zhou et al. (2023) ; Kosinski (2024) ; Strachan et al. (2024) and show potential for simulating human behavior across diverse scenarios Wang et al. (2024) ; Samuel et al. (2025) , suggesting their applicability to complex social dynamics. A significant research gap remains in our understanding of how strategic ToM reasoning influences behavioral alignment between LLMs and humans in social simulations. While LLM-based agents display human-aligned behavior in economic games like the ultimatum game Sreedhar and Chilton (2025) , and ToM-informed prompting improves negotiation outcomes from game-theoretic perspectives Qiu et al. (2024) , the extent to which ToM reasoning and prosocial beliefs drive behavioral alignment in negotiation contexts remains underexplored. Understanding this behavioral alignment has important implications for designing AI systems in high-stakes domains such as cooperative decision-making Piatti et al. (2024) . While Xie et al. (2024) investigated trust alignment in similar social simulations (Trust Games), their work does not address how alignment can be steered through the interplay of prosocial beliefs, strategic reasoning, and ToM. We investigate how ToM reasoning affects LLM behavioral alignment with humans in strategic negotiation tasks. By simulating the ultimatum game (UG) Güth et al. (1982) between a proposer and a responder as a canonical testbed, we evaluate different reasoning strategies, e.g., chain of thought (CoT) and varying ToM levels, across diverse belief combinations. This focused, single-game scope allows us to isolate our primary variable of interest—the conflict between an LLM’s intrinsic model bias and its assigned prosocial beliefs—and eliminate confounding noise such as deception or complex state-tracking. Our analysis identifies which reasoning approaches yield optimal alignment for each role and which belief configurations best replicate human decision-making. Our study takes a step forward in understanding LLM behaviors in simulations, how their behavior aligns to that of humans and if complex reasoning methods such as ToM help improve that alignment. Specifically, we focus on answering the following research questions: RQ1 : Which agent belief types achieve the highest acceptance rate (AC), and does high AC indicate behavior aligned with economic rationality? RQ2 : Which LLMs and reasoning methods lead to stronger behavioral alignment? RQ3 : Which proposer-responder belief combinations most closely match expected human norms? RQ4 : How does adherence to belief-consistent strategies impact the agents’ alignment with human norms? Based on 2,700 simulations, we find that ToM reasoning significantly and role-dependently enhances behavioral alignment with human norms compared to standard CoT reasoning. Additionally, we find significant behavioral misalignment for agents assigned Greedy or Selfless beliefs, even though they almost always select belief-consistent strategies. We hypothesize that this outcome-level gap reflects the intrinsic human-preference fine-tuning of LLMs, which may temper how fully agents enact extreme strategies. To summarize, our contributions are as follows: 1. We present a systematic framework for studying the agentic alignment with human norms, conditioned on prosocial belief types, of various LLMs in ultimatum games. 2. We analyze how different ToM reasonings compare with CoT and affect strategic bargaining between LLM agents. 3. We extensively study and discuss how LLM strategy choices, their reasoning methods and belief interactions affect behavioral alignment with human norms. 4. We validate the consistency of LLM strategies and reasoning with their decisions and expected belief-specific behaviors through human verification and statistical analysis. 2 Related Work Machine Theory of Mind Early ToM models relied on predefined rules and Belief-Desire-Intention (BDI) formalizations Pynadath and Marsella (2005) ; Ray et al. (2008) ; Rao et al. (1995) ; Georgeff et al. (1999) ; Norling (2004) ; Adam and Gaudou (2016) , lacking natural language capability. LLMs have renewed interest in machine ToM. Models like GPT-4 perform well on ToM evaluations Sap et al. (2022) ; van Duijn et al. (2023) ; Kosinski (2024) ; Strachan et al. (2024) yet struggle with negotiation Chan et al. (2024) , motivating subsequent work such as ToM-based opponent modeling Kuru et al. (2025) . Unlike Chan et al. (2024) , who treat ToM as an applied capability for better negotiation, we examine the effects of multi-level ToM reasoning on the model’s behavioral fidelity, compared to standard human norms. LLMs in Social Simulations LLMs have been explored across social simulations and strategic decision-making games Bianchi et al. (2024) ; Breum et al. (2024) ; Sicilia et al. (2024) ; Donkers and Ziegler (2025) ; Phelps and Russell (2023) ; Hua et al. (2024) ; Noh and Chang (2024) , where guided reasoning aids strategic decision-making Phelps and Russell (2023) ; Hua et al. (2024) and ToM improves simulated dialogue outcomes Qiu et al. (2024) ; Zhang et al. (2024) . Recent studies probe single-game behavior in the Ultimatum Game Sreedhar and Chilton (2025) ; Ferraz et al. (2026) , Prisoner’s Dilemma Fontana et al. (2025) ; Orland and Takemoto (2025) , Public Goods Games Piedrahita et al. (2025) , and altruistic decision-making more broadly

Read the complete text