ToMPO: Training LLM Strategic Decision Making from a Multi-Agent Perspective
Yiwen Zhang, Ziang Chen, Fanqi Kong, Yizhe Huang, Xue Feng · 2025
Read ToMPO: Training LLM Strategic Decision Making from a Multi-Agent Perspective by Yiwen Zhang, Ziang Chen, Fanqi Kong, Yizhe Huang, Xue Feng (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-21134.txt.
Introduction
Strategic choices depend on what rivals will do, and language models are increasingly asked to make them. Yiwen Zhang and four co-authors present ToMPO in a 2025 paper on training strategic decision making. Earlier work focused on social conversation games or ignored how decision types interlock.
The authors first define a strategic decision problem with two decision types and their temporal dependencies. Their algorithm, Theory of Mind Policy Optimization, trains the model to perceive other agents' strategies while optimizing its own. Standard reinforcement learning struggles to represent opponents during training. ToMPO builds that opponent modeling into the update. The goal is models that reason about a game's situation as a whole, rather than reacting move by move.
Catalog record
| CREATOR | Yiwen Zhang, Ziang Chen, Fanqi Kong, Yizhe Huang, Xue Feng |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 44 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-21134
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
ToMPO: Training LLM Strategic Decision Making from a Multi-Agent Perspective Yiwen Zhang, Ziang Chen, Fanqi Kong, Yizhe Huang, Xue Feng Abstract Large Language Models (LLMs) have been used to make decisions in complex scenarios, where they need models to think deeply, reason logically, and decide wisely. Many existing studies focus solely on multi-round conversations in social tasks or simulated environments, neglecting the various types of decisions and their interdependence. Current reinforcement learning methods struggle to consider the strategies of others during training. To address these issues, we first define a strategic decision-making problem that includes two types of decisions and their temporal dependencies. Furthermore, we propose T heory o f M ind P olicy O ptimization (ToMPO) algorithm to optimize the perception of other individual strategies and the game situation trends. Compared to the Group Relative Policy Optimization (GRPO) algorithm, ToMPO enhances the LLM’s strategic decision-making mainly by: 1) generating rollouts based on reasoning the strategies of other individuals, 2) estimating advantages at both the graph-level and sample-level, and 3) balancing global and partial rewards. The ToMPO algorithm outperforms the GRPO method by 35% in terms of model output compliance and cooperative outcomes. Additionally, when compared to models with parameter sizes 100 times larger, it shows an 18% improvement. This demonstrates the effectiveness of the ToMPO algorithm in enhancing the model’s strategic decision-making capabilities. 0 0 footnotetext: † Equal contribution. 🖂 Corresponding author. 1 Introduction Large Language Models (LLMs) utilize natural language understanding and generation capabilities to achieve leading performance in decision-making scenarios, assisting people in generating ( Gou et al. 2024 ) , simulating ( Mao et al. 2025 ) , and predicting ( Zhang et al. 2024a ) decisions across various categories. While LLMs excel in coding and math tasks, they struggle with strategic decision-making, which requires understanding others’ intentions, predicting behaviors, and adjusting their own strategies dynamically ( Zhang et al. 2024b ) . LLMs demonstrate varying strategic abilities in matrix games ( Lorè & Heydari 2024 ; Herr et al. 2024 ) and can be enhanced through a game-theoretic workflow ( Hua et al. 2024 ) . Recent research further explores LLM strategic decision-making through multi-level thinking ( Zhang et al. 2024c ; Gou et al. 2024 ) , Theory of Mind ( Duan et al. 2024 ; Cross et al. ) , task-solving ( Zhang et al. 2025a ; Wang et al. 2024 ) , as well as influences between individuals and groups ( Mi et al. 2025 ; Zhang et al. 2025b ) . (detailed related work in section 9 ) These studies provide methods for LLMs to adapt to human society, emerge human behaviors, and serve social issues. However, these studies restrict the strategic decision-making capabilities of LLM to two-agent chatroom environments or single-game scenarios. This approach fails to provide the necessary methods for LLM to enhance its performance in diverse, long-term multi-agent decision-making tasks. By focusing on these key issues, our paper analyzes the strategic decision-making capabilities of LLMs in complex social environments, where LLMs must sequentially make decisions that impact both individuals and groups. During this period, the prior decisions made by LLM will have a certain degree of influence on subsequent decisions. This implies that individual behaviors may lead to changes in the social structure of the group, and at the same time, changes in the group structure will affect subsequent individual decisions. For instance, in real life, before signing a cooperation agreement with multiple distributors, enterprises will conduct various evaluations. After the cooperation agreement is signed, they will implement the cooperation with varying levels of investment over a specified period. Each cooperation has a certain impact on whether the enterprise decides to continue the next collaboration. Furthermore, if there is a desire to terminate the cooperation during the process, it cannot be done immediately; that is, reversing the decision is not possible. This decision-making process helps highlight the real-world challenges faced by individuals and groups over time, posing a challenge to the model’s capabilities. In this context, we first define the problem as a sequential decision-making process that primarily involves graph-level and effort-level decisions. Then we build three kinds of complex social environments to test SOTA (State-of-the-Art) LLM performance. To optimize performance, we propose a reinforcement learning algorithm that integrates a multi-agent perspective into the LLM-based policy model training process. Based on the preliminary tests, we created an expert dataset containing the effort-level decisions made by models that achieve high rewards, across various topological positions and at different stages in the game. The policy model effectively learns decision-making at the effort level from the expert dataset through a supervised fine-tuning process. We enhanced the policy model for graph-level decision-making through reinforcement fine-tuning, which incorporates multi-agent considerations in reward modeling during the training process. Our contribution can be summarized as: • We define a problem for real-world strategic decision-making and design corresponding general simulation environments for decision data generation and examination. • We evaluate the performance of the State-of-the-Art (SOTA) models and provide a dataset including the expert model’s strategic decisions under different topological structures and at different game time processes. • We propose a reinforcement learning algorithm, Theory of Mind Policy Optimization (ToMPO), and apply it to the Qwen-2.5-7B-instruct model, achieving improvement in strategic decision-making capabilities. 2 Problem Formulation In contrast to the scenarios discussed in Theory of Mind ( Strachan et al. 2024 ; Liu et al. 2025c ) and single LLM long-term planning ( Huang et al. 2024 ; Ma et al. 2025 ) , we require the LLM to operate as an agent within a multi-agent environment consisting of at least three agents, making two types of decisions sequentially. During any decision-making process, an agent considers the strategies of other agents and its subsequent strategy, depending on its own state. These considerations will autonomously change based on the agent’s social status, game progress, and others’ performance. Graph-Effort Strategic Decision-Making We define the decision-making process as a set \langle\mathcal{N},\mathcal{S},\mathcal{A},T,\tau,f,r,\gamma\rangle , with the set of all agents or players \mathcal{N}=\{1,2,\dots,N\} , state space \mathcal{S} , total game round T , decision type sequence \tau , the state transition function f\in\{f_{G},f_{E}\} , utility function r , and discount factor \gamma . The agent state at round t includes the agent’s social relationship structure ( \mathcal{G} ) and effort ( \mathcal{E} ) at this round. L and M represent how many steps of actions related to structure forming and effort investment, respectively in one round. \tau represents the overall action type sequence. For example, when L=1,M=2 , let \tau=\{(G,E,E),(G,E,E),\ldots\} represent a sequence where the LLM agent must make a graph-level decision at step 0 in one game round. This decision involves choosing whether to join one group or establish relationships with others. At steps 1 and 2, the agent will determine how much to invest based on the social relations established in step 0. This pattern continues in subsequent rounds. a_{i,{t,j}} is the action decision agent i made at step j of round t (equation 2 ). Action space \mathcal{A}=\{A_{i}\}_{i\in\mathcal{N}}=\{a_{i,\tau}\}_{i\in\mathcal{N}} (equation 3 ). \forall i\in\mathcal{N},\,\,t\in[0,T-1],\,\,j\in[0,L+M-1] (1) S_{t}=(G_{t},E_{t}),\quad\tau(t,j)\in\{\text{G},\text{E}\},\quad a_{i,{t,j}}=\begin{cases}g_{i,{t,j}}&\text{if }\tau(t,j)=\text{G}\\ e_{i,{t,j}}&\text{if }\tau(t,j)=\text{E}\end{cases} (2) A_{t}=(A_{G}^{t,L},A_{E}^{t,M})=(\{A_{G}^{t,0},A_{G}^{t,1},...,A_{G}^{t,{L-1}}\},\{A_{E}^{t,{L}},A_{E}^{t,{L+1}},...,A_{E}^{t,{L+M-1}}\}) (3) Decision-Making Optimization with Credit Assignment According to Credit Assignment ( Nguyen et al. 2018 ; Pignatelli