Retaining Suboptimal Actions to Follow Shifting Optima in Multi-Agent Reinforcement Learning Yonghyeon Jo, Sunwoo Lee, Seungyul Han Abstract Value decomposition is a core approach for cooperative multi-agent reinforcement learning (MARL). However, existing methods still rely on a single optimal action and struggle to adapt when the underlying value function shifts during training, often converging to suboptimal policies. To address this limitation, we propose Successive Sub-value Q-learning (S2Q), which learns multiple sub-value functions to retain alternative high-value actions. Incorporating these sub-value functions into a Softmax-based behavior policy, S2Q encourages persistent exploration and enables Q tot Q^{\text{tot}} to adjust quickly to the changing optima. Experiments on challenging MARL benchmarks confirm that S2Q consistently outperforms various MARL algorithms, demonstrating improved adaptability and overall performance. Our code is available at https://github.com/hyeon1996/S2Q. 1 Introduction Multi-agent reinforcement learning (MARL) has emerged as a powerful and versatile framework for solving complex sequential decision-making problems involving multiple interacting agents ( Canese et al. 2021 ; Li et al. 2022 ; Wen et al. 2022 ) . Its applicability spans a wide range of domains, including robotic control, autonomous driving and traffic management, intelligent manufacturing, as well as strategic and competitive environments such as real-time strategy games ( Canese et al. 2021 ; Li et al. 2022 ; Orr & Dutta 2023 ; Shalev-Shwartz et al. 2016 ; Bahrpeyma & Reichelt 2022 ; Vinyals et al. 2019 ) . Within the centralized training with decentralized execution (CTDE) paradigm, where agents can leverage centralized information during training while acting independently at execution time, value-decomposition-based credit assignment methods ( Sunehag et al. 2017 ; Rashid et al. 2020a ; Wang et al. 2020a ) have proven highly effective across many MARL benchmarks. Nevertheless, conventional approaches face fundamental challenges due to the requirements of decentralized execution. In particular, QMIX ( Rashid et al. 2020a ) , one of the most representative CTDE methods, enforces the Individual-Global-Max (IGM) condition ( Son et al. 2019 ) through a monotonicity constraint, ensuring that the joint action-value function increases monotonically with respect to the utility of each agent. Although this design enables competitive performance across diverse tasks, it also restricts the expressiveness of the joint action-value function, limiting its ability to represent complex interactions among agents. Several lines of work have attempted to overcome this limitation. For instance, Rashid et al. 2020b introduces an unconstrained target network to improve the fidelity of optimal-action estimation, while Zhou et al. 2022 incorporates auxiliary information to mitigate representational bottlenecks under partial observability. More recent studies reformulate the value function into forms that are more amenable to decomposition ( Shen et al. 2022 ; Li et al. 2024 ) , further improving stability and performance. Despite these advances, existing methods still struggle when the optimality of the value function shifts during training. This challenge arises not only from the representational rigidity imposed by the monotonicity constraint but also from the heavy reliance on \epsilon -greedy policy in the typically large joint action spaces of MARL. To address these issues, we propose Successive Sub-value Q-learning (S2Q), a novel MARL framework that successively learns multiple sub-value functions, each designed to identify a distinct suboptimal action. Leveraging these sub-value functions, S2Q replaces naive \epsilon -greedy policy with a Softmax-based strategy that encourages persistent visitation around promising joint actions. This allows S2Q to accurately detect shifts in the optimal action and rapidly adapt its policy, thereby avoiding premature convergence to suboptimal solutions. We further provide theoretical and empirical analyses to validate the effectiveness of our approach, demonstrating that S2Q achieves substantial improvements over other recent MARL methods on challenging benchmarks, including the StarCraft II Multi-Agent Challenge ( Samvelyan et al. 2019 ) and Google Research Football ( Kurach et al. 2020 ) . 2 Preliminaries 2.1 Decentralized POMDPs In cooperative multi-agent RL, the problem is formulated as a Decentralized Partially Observable Markov Decision Process (Dec-POMDP), represented by the tuple G=\langle\mathcal{S},\mathcal{A},P,r,O,\mathcal{O},N,\gamma\rangle , where \mathcal{S} denotes the state space, \mathcal{A} the action space of each agent, P the transition probability function, r the reward function, O the observation function, \mathcal{O} the observation space, N the number of agents, and \gamma the discount factor ( Oliehoek et al. 2016 ) . At each timestep t , the environment resides in a global state s_{t}\in\mathcal{S} , while agent i receives a local observation o_{t}^{i}:=O(s_{t})\in\mathcal{O} . Based on its trajectory history \tau_{t}^{i}:=(o_{0}^{i},a_{0}^{i},\ldots,o_{t}^{i}) , agent i selects an action a_{t}^{i}\in\mathcal{A} according to a decentralized policy \pi^{i} . After receiving the joint action \mathbf{a}_{t}=(a_{t}^{1},\ldots,a_{t}^{N}) from all agents, the environment transitions to the next state s_{t+1}\sim P(\cdot\mid s_{t},\mathbf{a}_{t}) and the reward r_{t}:=r(s_{t},\mathbf{a}_{t}) . The overall objective of MARL is to maximize the expected discounted return, \mathbb{E}[\sum_{t=0}^{\infty}\gamma^{t}r_{t}]. 2.2 Value Decomposition under CTDE Under the Centralized Training with Decentralized Execution (CTDE) paradigm, agents act independently during execution while utilizing global state information during training. Among the most prominent methods in this setting is QMIX ( Rashid et al. 2020a ) , which employs a monotonic mixing function to enforce the Individual-Global-Max (IGM) condition, ensuring that maximizing individual utility Q^{i} cannot reduce the joint value Q^{\text{tot}} . Although such alignment is beneficial, the monotonicity constraint limits the expressiveness of Q^{\text{tot}} , often hindering effective minimization of the temporal-difference (TD) loss. To address this limitation, Weighted QMIX (WQMIX) ( Rashid et al. 2020b ) introduces an auxiliary unconstrained joint action-value function Q^{*} : \mathbb{E}_{(s_{t},\bm{\tau}_{t},\mathbf{a}_{t})\sim\mathcal{B}}\left[(Q^{*}(s_{t},\bm{\tau}_{t},\mathbf{a}_{t})-y_{t})^{2}\right],\quad y_{t}=r_{t}+\gamma Q_{\mathrm{targ}}^{*}(s_{t+1},\bm{\tau}_{t+1},\mathbf{a}^{\prime}_{t+1}), (1) where \bm{\tau}_{t}=(\tau_{t}^{1},\ldots,\tau_{t}^{N}) is the joint history, Q_{\text{targ}}^{*} the target, and a_{t}^{i\prime}=\arg\max Q^{i}(\tau_{t}^{i},\cdot) the individual target action. WQMIX updates Q^{\text{tot}} toward the target y_{t} by adaptively weighting the TD error with w(s_{t},\mathbf{a}_{t}) , where w(s_{t},\mathbf{a}_{t})=1 if Q^{\text{tot}}(s_{t},\bm{\tau}_{t},\mathbf{a}_{t})0 is applied to properly handle potentially negative values of Q_{\mathrm{targ}}^{*} . When Q_{\mathrm{targ}}^{*} is positive, we can simply use Q_{\mathrm{targ}}^{*} instead of \max(Q_{\mathrm{targ}}^{*},C) in equation 2 for implementation. From the proposed successive Q-learning, each sub-value function Q^{\mathrm{sub}}_{k} can select the k -th suboptimal joint action of a given Q^{*} exactly, as guaranteed by the following theorem, provided that the suppression constant \alpha is sufficiently large. Theorem 4.1 . Let Q^{*}(s_{t},\bm{\tau}_{t},\mathbf{a}_{t}) and \{Q^{\mathrm{sub}}_{k}\}_{k=0}^{K} be the joint action-value function and sub-value functions obtained by minimizing equation 1 and equation 2 , respectively, and let \{\mathbf{a}_{0,t}^{*},\dots,\mathbf{a}_{K,t}^{*}\} denote the K{+}1 successive suboptimal joint actions of Q^{*} for given s_{t} and \bm{\tau}_{t} at timestep t s.t. Q^{*}(s_{t},\bm{\tau}_{t},\mathbf{a}_{0,t}^{*})\geq\dots\geq Q^{*}(s_{t},\bm{\tau}_{t},\mathbf{a}_{K,t}^{*})\geq Q^{*}(s_{t},\bm{\tau}_{t},\mathbf{a}_{t}),\quad\forall\mathbf{a}_{t}\notin\{\mathbf{a}_{0,t}^{*},\dots,\mathbf{a}_{K,t}^{*}\}. If the reward function r is bounded and suppression factor \alpha is sufficiently large, then, \mathbf{a}_{k,t}^{*}=\arg\max_{\mathbf{a}_{t}}Q^{\mathrm{sub}}_{k}(s_{t},\bm{\tau}_{t},\mathbf{a}_{t})\quad\forall k\in\{0,...,K\}. (3) Proof) Proof of Theorem 4.1 is provided in Appendix B . According to Theorem 4.1, it is guaranteed that, with bounded rewards and a sufficiently large suppression factor \alpha , the successive learning procedure maintains Q_{0}^{\text{sub}},\dots,Q_{K}^{\text{sub}} , where Q_{0}^{\text{sub}} represents the global optimum and the remaining sub-value functions capture successive suboptimal actions. By explicitly tracking the high-value joint actions, S2Q allows Q^{*} to effectively update its values and ensures that Q_{0}^{\text{sub}} can promptly adapt when the optimal action changes. Fig. 2 illustrates this process, where each Q_{k}^{\text{sub}} suppresses previously identified actions before selecting the next best candidate. Importantly, although this procedure is presented in the context of WQMIX, it is general and can be applied to any CTDE method by replacing Q^{*} with Q^{\text{tot}} . Consequently, S2Q can follow changes in the value landscape more closely and adapt to new optima faster than conventional approaches. 4.3 Coordinated Execution via Communication during Training Figure 3: Overall framework of S2Q Algorithm 1 S2Q Framework 1: Initialize Q^{*} , [Q^{sub}_{k}]^{K}_{k=0} , (E,D) 2: for each training iteration do 3: for each environment step t do 4: if training then 5: Obtain z_{t}=E(\bm{\tau}_{t}) 6: Obtain \hat{\mathbf{P}}_{t},\hat{s}_{t}=D(z_{t}) 7: Sample k\sim\hat{\mathbf{P}}_{t} 8: Sample \mathbf{a}_{t}\sim\epsilon\text{-greedy}(Q^{sub}_{k}) 9: else 10: Sample \mathbf{a}_{t}\sim\text{greedy}(Q^{sub},0) 11: end if 12: end for 13: Compute target y 14: Update Q^{*} , Q^{sub}_{k} via Equation 1 , 2 15: Update E , D 16: end for As described above, S2Q continuously tracks suboptimal actions, but for these actions to contribute effectively, Q^{*} must be trained toward global convergence. However, prior MARL methods typically rely on agent-wise \epsilon -greedy exploration, under which the probability of joint exploration decreases exponentially with the number of agents, causing most joint actions to remain near the current optimum. To overcome this limitation, we explicitly execute tracked suboptimal actions with priority determined by a Softmax distribution \mathbf{P}_{t} over their Q^{*} values, thereby ensuring more frequent visits, enabling Q^{*} to find better optimal and suboptimal actions, defined as \mathbf{P}_{t}=(P_{0,t},\dots,P_{K,t}):=\text{Softmax}\Big(Q^{*}(s_{t},\bm{\tau}_{t},\mathbf{a}_{0,t}^{*})/T,\dots,Q^{*}(s_{t},\bm{\tau}_{t},\mathbf{a}_{K,t}^{*})/{T}\Big), (4) where T is a temperature parameter. The behavior policy first samples k\sim\mathbf{P}_{t} , and then executes actions from Q_{k}^{\text{sub}} according to \epsilon -greedy rule, i.e., a_{t}^{i}\sim\epsilon\text{-greedy}(Q_{k}^{i}) . This design first selects k based on Q^{*} and then explores around the corresponding suboptimal action, and we empirically show in Appendix D that S2Q visits a wider range of spaces than conventional \epsilon -greedy exploration. Exact computation of \mathbf{P}_{t} , however, requires access to global information that decentralized agents cannot directly observe. More importantly, since all agents must select the same sub-value function index k in order to execute Q_{k}^{\text{sub}} consistently, coordination among agents is necessary. To resolve these challenges, we introduce communication during training, allowing agents to estimate \mathbf{P}_{t} jointly and synchronize their choice of Q_{k}^{\text{sub}} . For this purpose, we adopt an encoder–decoder architecture for general representation learning, following prior communication methods ( Guan et al. 2022 ; Wang et al. 2019 ) . Specifically, encoder E maps local histories \bm{\tau}_{t} into a latent representation z_{t}=E(\bm{\tau}_{t}) , and the decoder D reconstructs both the global state and an approximate distribution (\hat{s}_{t},\hat{\mathbf{P}}_{t})=D(z_{t}) . Agents then synchronize on the same k sampled from \hat{\mathbf{P}}_{t} instead of \mathbf{P}_{t} and execute the corresponding Q_{k}^{\text{sub}} , ensuring consistent and accurate use of sub-value functions. At test time, by contrast, communication is not required in the default setup of S2Q. Since Q_{0}^{\text{sub}}=Q^{\text{tot}} alone suffices to produce the greedy optimal action \mathbf{a}_{0,t}^{*} , evaluation remains fully decentralized and communication-free. This provides a practical advantage over conventional communication methods that rely on message passing even during evaluation. Nevertheless, in environments where communication is indispensable for task success, we also consider a variant, denoted S2Q-Comm, in which the latent z_{t} is provided to each Q_{k}^{i} during both training and execution. Through this practical design, S2Q integrates successive sub-value functions with a Softmax-based behavior policy, enabling Q^{\text{tot}} to adapt quickly when the optimal action changes. The overall S2Q framework is illustrated in Fig. 3 , where each local history \tau^{i}_{t} is processed along the S2Q Learning module, and the Encoder-Decoder module to generate the softmax probabilities \mathbf{P}_{t} , and its prediction \hat{\mathbf{P}}_{t} . During training, agents use \hat{\mathbf{P}}_{t} to synchronize their selection of the sub-value function Q_{k}^{\text{sub}} , while evaluation relies solely on the local outputs. The training procedure is summarized in Algorithm 1 , and further details on loss functions and implementation are provided in Appendix C . 5 Experiments (a) SMAC-Hard+: Corridor (b) GRF: academy_3_vs_2 (c) SMAC-Comm: 1o_2r_vs_4r Figure 4: Experiment environments In this section, we evaluate S2Q on two widely used MARL benchmarks: the StarCraft Multi-Agent Challenge (SMAC) ( Samvelyan et al. 2019 ) , which involves micromanagement tasks in StarCraft II, and Google Research Football (GRF) ( Kurach et al. 2020 ) , which features cooperative soccer-like scenarios with an opponent team including a goalkeeper. As shown in Fig. 4 (a), we consider the SMAC-Hard+ suite, consisting of one hard map ( 5m_vs_6m ) and five super hard maps ( MMM2 , 27m_vs_30m , corridor , 6h_vs_8z , 3s5z_vs_3s6z ), all of which require a high degree of coordination among agents. Fig. 4 (b) illustrates the GRF setups, including academy_3_vs_2 and academy_4_vs_3 , where the number of agents and team formations are varied. In addition, Fig. 4 (c) depicts the SMAC-Comm suite, which explicitly requires agent communication for successful task completion, including 1o_10b_vs_1r , 1o_2r_vs_4r , 5z_vs_1ul , and bane_vs_hM . Further details of the environmental setup are provided in Appendix E . For all reported result tables and plots, we present the mean and standard deviation across five random seeds. 5.1 Performance Analysis For performance evaluation, we consider communication-free tasks from SMAC-Hard+ and GRF , along with communication-demanding scenarios from SMAC-Comm , with appropriate baselines for each setting to ensure fair comparison. In the SMAC-Hard+ and GRF benchmarks, we compare S2Q with QMIX and methods addressing the limitations of monotonic value decomposition, including WQMIX 1 1 1 Throughout our experiments, we adopt OW-QMIX variant of WQMIX ( Rashid et al. 2020b ) . , which leverages Q^{*} for more accurate value estimation; DOP ( Wang et al. 2020d ) and FOP ( Zhang et al. 2021 ) , which extend the actor–critic paradigm to promote better global coordination; PAC ( Zhou et al. 2022 ) and RiskQ ( Shen et al. 2023 ) , which integrate counterfactual prediction or risk-aware objectives; and MARR ( Yang et al. 2024 ) , which improves sample efficiency through reset mechanisms. We also include MASIA ( Guan et al. 2022 ) , which reconstructs global state information from local observations, to evaluate the effect of communication. For the SMAC-Comm scenarios, where information exchange among agents is critical under partial observability, we adopt S2Q-Comm, a variant that provides the latent z_{t} to each agent’s utility Q^{i}(\tau_{t}^{i},z_{t},a_{t}^{i}) , and compare it with QMIX and QMIX-Comm, a variant of QMIX augmented with learned latent information produced by an encoder–decoder structure that reconstructs the state, FullCom, which augments QMIX with full observation sharing, and recent communication-focused MARL methods such as MASIA, NDQ ( Wang et al. 2019 ) , and CAMA ( Shao et al. 2023 ) , which enhance communication via information-theoretic regularization or complementary attention, as well as MAIC ( Yuan et al. 2022 ) and T2MAC ( Sun et al. 2024 ) , which employ selective and targeted communication for improved coordination efficiency. All algorithms are implemented using the authors’ official code, and S2Q is trained with the best-performing hyperparameter ( K=2 , T=0.1 ). Since the Q -values quickly turn positive in environments we consider, we adopt Q_{\mathrm{targ}}^{*} instead of \max(Q_{\mathrm{targ}}^{*},C) in equation 2 for implementation, as mentioned in Section 4.2 . Additional experimental details, including the hyperparameter setup of S2Q are provided in Appendix F . Figure 5: Performance comparison: Average test win rates in the SMAC-Hard+ and GRF tasks SMAC-Hard+ and GRF : Fig. 5 compares performance on SMAC-Hard+ and GRF. S2Q consistently outperforms existing baselines, achieving faster convergence and higher asymptotic returns. This advantage is most evident in exploration-intensive scenarios such as 6h_vs_8z and 3s5z_vs_3s6z , where conventional methods adapt slowly to the optimal joint action shifts, as shown in the payoff matrix experiment in Section 4.1 . By continually tracking suboptimal actions with successive sub-value functions, S2Q allows Q^{\text{tot}} to rapidly adjust toward the new optimum, enabling more efficient exploration and quicker convergence, even in the challenging GRF tasks. Figure 6: Performance comparison: Average test win rates in the SMAC-Comm tasks SMAC-Comm : Fig. 6 presents results for communication-critical tasks. S2Q-Comm performs comparably to baselines in the simpler 1o_2r_vs_4r scenario, but demonstrates substantial gains in more challenging settings such as 5z_vs_1ul and bane_vs_hM , where tight coordination and real-time adaptation are required. Synchronized sub-value function selection through the learned latent z_{t} enables agents to consistently exploit informative suboptimal actions, leading to more efficient cooperation. In addition, comparison with QMIX-Comm shows that these improvements arise from S2Q’s successive sub-value learning, rather than relying solely on information sharing. These findings suggest that S2Q-Comm can serve as a general approach for leveraging communication efficiently, providing a promising direction for scaling to larger teams and more complex partially observable environments. Together, these experiments show that by continually tracking and exploiting suboptimal actions, S2Q achieves faster convergence and more efficient learning than state-of-the-art MARL methods. We further evaluate S2Q on additional stochastic environments, including SMACv2 ( Ellis et al. 2023 ) , which randomizes team compositions, and SMAC-Hard ( Deng et al. 2024 ) , with mixed scripted opponents. For clarity, we refer to the latter benchmark as SMAC-Hard (Mixture Opponent) to distinguish it from the SMAC-Hard+ environment. Despite the increased stochasticity, S2Q consistently outperforms all baselines, suggesting that our method can effectively address stochastic dynamics to a meaningful extent. Furthermore, we extend S2Q to other CTDE methods such as VDN ( Sunehag et al. 2017 ) and QPLEX ( Wang et al. 2020a ) , observing similarly consistent performance gains. Detailed results for these additional experiments are provided in Appendix G . 5.2 Trajectory analysis Figure 7: Training behavior of S2Q. Left: training step = 2M. Right: training step = 5M To investigate the training dynamics of S2Q ( K=2 ), we analyze the distribution of agents’ actions in the 6h_vs_8z scenario, shown in Fig. 7 (a). Fig. 7 (b) reports the evolution of hit rate and win rate over training. Since SMAC tasks require agents to defeat the opposing team, the dominant actions are move and hit . Early in training, agents tend to prioritize survival, favoring move and exhibiting low hit rates. However, as training progresses, the hit rate gradually increases, leading to higher win rates, indicating that the optimal policy shifts from evasive behavior towards more aggressive strategies where hit dominates. Fig. 7 (c) illustrates how S2Q adapts to this shift in optimality by leveraging its successive sub-value functions. Initially, move is considered the optimal action ( \mathbf{a}_{0,t}^{*} ), while hit actions ( \mathbf{a}_{1,t}^{*} , \mathbf{a}_{2,t}^{*} ) are tracked as suboptimal. As training proceeds and Q^{*} recognizes that hit yields higher returns, S2Q gradually increases the execution frequency of hit via the Softmax-based behavior policy. This enables Q_{0}^{\text{sub}} to promptly adjust to the newly optimal action, resulting in an increase in hit rate and ultimately higher win rates. These results provide a clear illustration of how S2Q tracks suboptimal actions and uses them to adapt efficiently when the optimal action changes. Additional trajectory analysis for the SMAC-Comm, SMAC-Hard (Mixture Opponent) , and GRF environments are also provided in Appendix H.4 . While detailed analyses are provided in Appendix H.4 , we observe a consistent trend across environments: agents frequently begin by selecting locally greedy actions conditioned on their initial states, but gradually transition toward the true optimal actions as training progresses. 5.3 Ablation study In this section, we present ablation studies evaluating the contributions of S2Q’s core components and hyperparameters. Component analysis is averaged across all scenarios in SMAC-Hard+, while hyperparameter analysis is conducted on 6h_vs_8z , where S2Q’s benefits are most evident. Additional results, including additional hyperparameter analysis in MMM2 and academy_4_vs_3 , which exhibit trends consistent with those observed in 6h_vs_8z , ablations on the suppression constant \alpha , the weighting factor w_{c} for WQMIX-based TD learning, and computational complexity, are reported in Appendix H , to validate the effectiveness and generality of S2Q across diverse settings. Table 1: Component evaluation of S2Q on SMAC-Hard+ tasks. Method Avg. Win Rate(%) S2Q 73.43 \mathbf{\pm} 5.29 S2Q_oracle 77.47 \mathbf{\pm} 4.32 S2Q_independent 46.22 \mathbf{\pm} 8.20 S2Q_no_wTD 70.59 \mathbf{\pm} 4.78 S2Q_no_soft 55.17 \mathbf{\pm} 6.71 S2Q_random 48.05 \mathbf{\pm} 9.37 QMIX 43.94 \mathbf{\pm} 10.06 (a) (b) Figure 8: Hyperparameter analysis. Effect of (a) the number of sub-networks K and (b) Softmax temperature T . Component evaluation : To evaluate the contribution of each component in S2Q, we design five ablations: S2Q_oracle uses the true softmax distribution \mathbf{P}_{t} instead of the estimated \hat{\mathbf{P}}_{t} ; S2Q_independent samples k independently, removing coordinated execution; S2Q_no_wTD removes the weighted TD learning by replacing Q^{*} with Q_{0}^{\text{sub}} ; S2Q_no_soft eliminates Softmax-based execution, always acting according to Q_{0}^{\text{sub}} ; and S2Q_random uniformly samples k rather than using Softmax probabilities. Table 8 reports the average win rate across six SMAC-Hard+ scenarios. The results show that ’S2Q_oracle’ serves as a reference upper bound, since it can access to the true softmax distribution \mathbf{P} and therefore achieves higher performance than S2Q. In contrast, ’S2Q_independent’ exhibits a substantial performance drop relative to S2Q because it cannot accurately estimate the selection probabilities and consequently fails to maintain proper synchronization across agents. Together, these ablations clearly demonstrate the critical role of accurately estimating the softmax probabilities in enabling effective coordinated sub-value selection. In addition, ‘S2Q_no_wTD’ maintains strong performance, indicating that successive sub-value learning and its execution are more critical to performance than the auxiliary weighting scheme. In contrast, ‘S2Q_no_soft’ suffers a noticeable drop in win rate, showing that simply retaining suboptimal actions without prioritizing their execution is insufficient. ‘S2Q_random’ further degrades performance, as ignoring the relative importance of different sub-value functions dilutes learning. These findings highlight that the proposed successive sub-value learning and the Softmax-guided execution are indispensable for enabling S2Q to achieve rapid convergence toward high-quality joint policies. Number of sub-value functions K : Controls the number of sub-value functions to learn. Fig. 8 (a) shows performance for K\in[0,1,2,3] . The results demonstrate that even with relatively small values of K , S2Q can effectively address diverse suboptimal actions in complex environments with large joint action spaces, such as SMAC, yielding rapid convergence and high final win rates. In particular, K=2 achieves the best performance, reaching nearly 70% win rate. On the other hand, setting K=0 fails to capture meaningful suboptimal actions, thereby limiting the effectiveness of exploration. Conversely, a larger value, such as K=3 , may introduce excessive variance and destabilize learning. These findings indicate that a moderate number of candidate sub-networks achieves a favorable trade-off between diversity and stability, maximizing the effectiveness of S2Q. Softmax temperature T : The temperature T regulates the sharpness of the Softmax distribution \mathbf{P}_{t} . Fig. 8 (b) shows the performance for T\in\{0.01,0.1,0.2,1.0\} . The results indicate that T=0.1 yields the best balance between convergence speed and final win rate. A very small temperature ( T=0.01 ) produces overly deterministic behavior early in training, limiting exploration and slowing progress toward the optimum. In contrast, a high temperature ( T=1.0 ) promotes excessive exploration, delaying convergence and occasionally leading to suboptimal plateaus. Interestingly, T=0.2 also performs competitively, suggesting that S2Q is relatively robust to moderate changes in temperature, as long as sampling remains focused enough to prioritize promising sub-values. 6 Limitations While S2Q shows strong performance across diverse MARL environments, using multiple sub-value functions increases computation and memory requirements. However, our analysis in Appendix H.5 shows that this overhead is moderate relative to the performance gains, making the trade-off favorable in most practical settings where faster convergence and higher final performance are critical. In addition, S2Q relies on a Softmax-based selection strategy with a temperature parameter controlling the exploration-exploitation balance. Nevertheless, we find that S2Q is not highly sensitive to this parameter, and a reasonable value can be chosen with minimal tuning effort, allowing S2Q to remain practical and easily deployable across a wide range of environments. 7 Conclusion In this work, we addressed the fundamental challenge posed by monotonic value decomposition in MARL and introduced S2Q, a novel framework based on successive Q-learning. By sequentially learning multiple sub-value functions, S2Q continuously tracks high-value alternative actions and rapidly adapts when the optimal joint action shifts, leading to more effective exploration and faster convergence. This design enables efficient coordination without requiring explicit communication or access to global state information. Extensive experiments on challenging MARL benchmarks, including SMAC and GRF, demonstrate that S2Q consistently avoids suboptimal solutions induced by monotonicity constraints and achieves substantial improvements over state-of-the-art methods. These results position S2Q as a practical and general framework for advancing cooperative MARL. Ethics Statement This work focuses on algorithmic improvements in cooperative multi-agent reinforcement learning (MARL) under the CTDE paradigm. Our study uses publicly available simulation environments, including SMAC ( Samvelyan et al. 2019 ) , Google Research Football ( Kurach et al. 2020 ) , and SMACv2 ( Ellis et al. 2023 ) , which do not involve human subjects, personal data, or sensitive information. Our work is motivated by cooperative and coordination tasks, and does not explicitly encourage or evaluate harmful uses. We adhere to the ICLR Code of Ethics and affirm that all research was conducted in alignment with principles of fairness, transparency, and academic integrity. Reproducibility Statement To ensure reproducibility, we provide a detailed description of S2Q in Section 4.3 , with implementation details in Appendix C . Benchmarks including SMAC ( Samvelyan et al. 2019 ) , GRF ( Kurach et al. 2020 ) , and SMACv2 ( Ellis et al. 2023 ) are publicly available, with descriptions and code links in Appendix E . An anonymized code repository containing the full implementation is submitted as supplementary material where configurations are detailed in Appendix F . Finally, considered baselines and their repositories are listed in Appendix F.3 . Together, these resources collectively enable independent reproduction of our results reported in this paper. Appendix A The Use of Large Language Models (LLMs) In preparing this paper, we made limited use of large language models (LLMs) exclusively for polishing grammar and improving the clarity of our writing. No part of the research process, including problem formulation, method design, experimentation, analysis, or interpretation, was conducted using LLMs. Their role was strictly restricted to editorial assistance, ensuring the presentation of our work is clear and readable. Appendix B Missing Proof Theorem B.1 (Successive Q-learning) . Let Q^{*}(s_{t},\bm{\tau}_{t},\mathbf{a}_{t}) and \{Q^{\mathrm{sub}}_{k}\}_{k=0}^{K} be the joint action-value function and sub-value functions obtained by minimizing equation 1 and equation 2 , respectively, and let \{\mathbf{a}_{0,t}^{*},\dots,\mathbf{a}_{K,t}^{*}\} denote the K{+}1 successive suboptimal joint actions of Q^{*} for given s_{t} and \bm{\tau}_{t} at timestep t s.t. Q^{*}(s_{t},\bm{\tau}_{t},\mathbf{a}_{0,t}^{*})\geq\dots\geq Q^{*}(s_{t},\bm{\tau}_{t},\mathbf{a}_{K,t}^{*})\geq Q^{*}(s_{t},\bm{\tau}_{t},\mathbf{a}_{t}),\quad\forall\mathbf{a}_{t}\notin\{\mathbf{a}_{0,t}^{*},\dots,\mathbf{a}_{K,t}^{*}\}. If the reward function r is bounded and suppression factor \alpha is sufficiently large, then, \mathbf{a}_{k,t}^{*}=\arg\max_{\mathbf{a}_{t}}Q^{\mathrm{sub}}_{k}(s_{t},\bm{\tau}_{t},\mathbf{a}_{t})\quad\forall k\in\{0,...,K\}. (B.1) Proof. We consider the minimization of the joint value loss equation 1 together with the proposed sub-value losses equation 2 . At convergence, the Bellman optimality equation implies that Q^{*} satisfies y_{t}=r_{t}+\gamma Q^{*}_{\mathrm{targ}}=Q^{*} . Using this relation, each sub-value function Q^{\mathrm{sub}}_{k} satisfies Q^{\mathrm{sub}}_{k}(s_{t},\bm{\tau}_{t},\mathbf{a}_{t})=\begin{cases}Q^{*}(s_{t},\bm{\tau}_{t},\mathbf{a}_{t})-\alpha\max(Q^{*}(s_{t},\bm{\tau}_{t},\mathbf{a}_{t}),C)&\text{if }\mathbf{a}_{t}\in\mathcal{A}_{k-1,t},\\ Q^{*}(s_{t},\bm{\tau}_{t},\mathbf{a}_{t})&\text{otherwise,}\end{cases} for every (s_{t},\bm{\tau}_{t},\mathbf{a}_{t}) pairs. For given (s_{t},\bm{\tau}_{t}) pair, if the suppression factor \alpha satisfies \alpha\geq\max_{\tilde{\mathbf{a}}_{t}\in\mathcal{A}_{k-1,t}}\frac{Q^{*}(s_{t},\bm{\tau}_{t},\tilde{\mathbf{a}}_{t})-Q^{*}(s_{t},\bm{\tau}_{t},\mathbf{a}_{k,t}^{*})}{\max(Q^{*}(s_{t},\bm{\tau}_{t},\tilde{\mathbf{a}}_{t}),C)}\geq 0, (B.2) then Q^{\mathrm{sub}}_{k}(s_{t},\bm{\tau}_{t},\tilde{\mathbf{a}}_{t})=Q^{*}(s_{t},\bm{\tau}_{t},\tilde{\mathbf{a}}_{t})-\alpha\,\max(Q^{*}(s_{t},\bm{\tau}_{t},\tilde{\mathbf{a}}_{t}),C)\leq Q^{*}(s_{t},\bm{\tau}_{t},\mathbf{a}_{k,t}^{*})=Q^{\mathrm{sub}}_{k}(s_{t},\bm{\tau}_{t},\mathbf{a}_{k,t}^{*}),\quad\forall\tilde{\mathbf{a}}_{t}\in\mathcal{A}_{k-1,t} . From the definition of a_{k,t}^{*} , Q^{\mathrm{sub}}_{k}(s_{t},\bm{\tau}_{t},\tilde{\mathbf{a}}_{t})=Q^{*}_{k}(s_{t},\bm{\tau}_{t},\tilde{\mathbf{a}}_{t})\leq Q^{*}(s_{t},\bm{\tau}_{t},\mathbf{a}_{k,t}^{*})=Q^{\mathrm{sub}}_{k}(s_{t},\bm{\tau}_{t},\mathbf{a}_{k,t}^{*}),\forall\tilde{\mathbf{a}}_{t}\in\mathcal{A}\backslash\mathcal{A}_{k-1,t} , so we can conclude \mathbf{a}_{k,t}^{*}=\arg\max_{\mathbf{a}_{t}}Q^{\mathrm{sub}}_{k}(s_{t},\bm{\tau}_{t},\mathbf{a}_{t}). (B.3) Thus, if we take the supremum over all state–observation histories and the maximum over all k\in{0,\dots,K} , the suppression factor must satisfy \alpha\geq\sup_{s_{t},\bm{\tau}_{t}}\max_{k\in\{0,\cdots,K\}}\max_{\tilde{\mathbf{a}}_{t}\in\mathcal{A}_{k-1,t}}\frac{Q^{*}(s_{t},\bm{\tau}_{t},\tilde{\mathbf{a}}_{t})-Q^{*}(s_{t},\bm{\tau}_{t},\mathbf{a}_{k,t}^{*})}{\max(Q^{*}(s_{t},\bm{\tau}_{t},\tilde{\mathbf{a}}_{t}),C)}\geq 0 , and this supremum is finite because the bounded-reward assumption implies that Q^{*} is also bounded, which ensures that equation B.3 holds for all (s_{t},\bm{\tau}_{t}) pairs and for all k\in\{0,\dots,K\} . This completes the proof of the theorem. ∎ Appendix C Implementation Details In this section, we detail the implementation of the proposed S2Q method. First, we present the encoder–decoder architecture described in Section 4.3 together with its associated loss function in Appendix C.1 . Then, we introduce the parameterization for the practical implementation of S2Q and the corresponding total loss function in Appendix C.2 . Furthermore, details on re-defined weighting function w_{k} , which enables S2Q to more effectively shift Q^{\text{tot}} towards the optimal policy, are provided in Appendix C.3 . C.1 Training the Encoder-Decoder Structure As introduced in Section 4.3 , exact computation of \mathbf{P}_{t} requires access to the global state s . Therefore, we proposed the approximation of \mathbf{P}_{t} based on an encoder-decoder architecture. For practical implementation, we parameterize the encoder-decoder structure with neural network parameter \psi and represent it as E_{\psi} and D_{\psi} . The encoder E_{\psi} utilizes an rnn layer, and maps the concatenation of all agents’ local histories \tau^{i}_{t} into a latent representation z_{t}=E_{\psi}(\bm{\tau_{t}}) . Then, the decoder D_{\psi} reconstructs both the global state and an approximate distribution (\hat{s}_{t},\hat{\mathbf{P}}_{t})=D_{\psi}(z_{t}) . Then, the encoder-decoder architecture is trained to approximate \mathbf{P}^{K} based on a cross-entropy (CE) loss \text{CE}(\mathbf{P}^{K},\hat{\mathbf{P}}^{K}) , while the reconstruction mean square error (MSE) loss \text{MSE}(\mathbf{s},\hat{\mathbf{s}}) encourages z to capture essential state information, thereby improving the prediction accuracy of \hat{\mathbf{P}}^{K} . The overall loss of the encoder-decoder architecture \mathcal{L}_{\text{latent}}(\psi) is defined as: \mathcal{L}_{\text{latent}}(\psi)=\text{CE}(\mathbf{P}^{K},\hat{\mathbf{P}}^{K})+\text{MSE}(\mathbf{s},\hat{\mathbf{s}}). (C.1) C.2 Practical Implementation of S2Q For practical implementation of S2Q, we parameterize both the monotonic sub-value functions Q_{0}^{\text{sub}},\dots,Q_{K}^{\text{sub}} and the unrestricted value function Q^{*} using neural network parameter \theta . As described in Section 2.2 , WQMIX ( Rashid et al. 2020b ) trains the unrestricted value function Q^{*} via temporal-difference (TD) loss in equation 1 , which we rewrite for the parameteriszed Q -function as: \mathcal{L}_{Q^{*}}(\theta)=\mathbb{E}_{(s_{t},\bm{\tau}_{t},\mathbf{a}_{t})\sim\mathcal{B}}\left[(Q^{*}_{\theta}(s_{t},\bm{\tau}_{t},\mathbf{a}_{t})-y_{t})^{2}\right],\quad y_{t}=r_{t}+\gamma Q_{\bar{\theta}}^{*}(s_{t+1},\bm{\tau}_{t+1},\mathbf{a}^{\prime}_{t+1}), (C.2) where \bm{\tau}_{t}=(\tau_{t}^{1},\ldots,\tau_{t}^{N}) is the joint history, Q_{\bar{\theta}}^{*} the target network, and a_{t}^{i\prime}=\arg\max Q^{i}(\tau_{t}^{i},\cdot) the individual target action. Then, WQMIX further updates Q^{\text{tot}} toward the target y_{t} by adaptively weighting the TD error using w(s_{t},\mathbf{a}_{t}) , defined as w(s_{t},\mathbf{a}_{t})=1 if Q^{\text{tot}}(s_{t},\bm{\tau}_{t},\mathbf{a}_{t})0 generated via a multi-head attention mechanism, enhancing the flexibility of the model. Throughout our experiments, results for VDN, QMIX, and QPLEX are based on the implementations provided in PyMARL2 ( Hu et al. 2021 ) , an open-source MARL framework that includes implementations of diverse algorithms along with various improvements, such as TD (\lambda) , larger batch sizes, and the Adam optimizer. The code is available at: https://github.com/hijkzzz/pymarl2 . F.2 Hyperparameter Setup In our experiments, while we adopt the default hyperparameter configuration from PyMARL2 ( Hu et al. 2021 ) for settings shared across CTDE methods, we focus on identifying effective values for the hyperparameters introduced in S2Q. The best parameters for each scenario are summarized in Table F.2 . According to Table F.2 , across a variety of tasks, K=2 , T=0.1 , and \alpha=1.0 consistently achieve the best performance. While the weighting factor w exhibits some task-specific variation, values of 0.75 or 0.9 generally perform well. These results demonstrate the robustness and flexibility of S2Q, achieving high win rates and rapid convergence across diverse scenarios. Table F.1: Common Q -learning Hyperparameters Hyperparameter Value \epsilon Decay Value 1.0 \rightarrow 0.05 \epsilon Anneal Time 100000 Target Update Interval 200 Discount Factor \gamma 0.99 Buffer Size 5000 Batch Size 128 Learning Rate 0.001 Optimizer Adam Optimizer Alpha 0.99 Optimizer Eps 1e-5 Gradient Clip Norm 10.0 Num GRU Layers 1 RNN Hidden State Dim 64 Double Q True Table F.2: Scenario-specific hyperparameter setup of S2Q Scenario K T \alpha w SMAC-Hard+ 5m_vs_6m 2 0.1 1.0 0.9 MMM2 2 0.1 1.0 0.9 27m_vs_30m 2 0.1 1.0 0.9 corridor 2 0.1 1.0 0.75 6h_vs_8z 2 0.1 1.0 0.9 3s5z_vs_3s6z 2 0.1 1.0 0.75 GRF academy_3_vs_2 2 0.1 1.0 0.75 academy_4_vs_3 2 0.1 1.0 0.75 SMAC-Comm 1o_2r_vs_4r 2 0.1 1.0 0.9 1o_10b_vs_1r 2 0.1 1.0 0.9 5z_vs_1ul 2 0.1 1.0 0.9 bane_vs_hM 2 0.1 1.0 0.9 SMACv2 terran_5_vs_5 2 0.1 1.0 0.75 zerg_5_vs_5 2 0.1 1.0 0.75 protoss_5_vs_5 2 0.1 1.0 0.75 SMAC-Hard (Mixture Opponent) 5m_vs_6m_Hard 2 0.1 1.0 0.9 2s3z_Hard 2 0.1 1.0 0.9 3s_vs_5z_Hard 2 0.1 1.0 0.9 F.3 Description of Other MARL Methods for Comparison QMIX ( Rashid et al. 2020a ) : introduces a mixing network that combines per-agent value functions into a joint action-value under a monotonicity constraint, ensuring consistency between centralized training and decentralized execution. We base our implementation of QMIX on the following repository: https://github.com/hijkzzz/pymarl2 WQMIX ( Rashid et al. 2020b ) : extends QMIX by introducing weighted projections in the mixing process, alleviating the representational limitations of monotonicity and enabling more accurate learning of optimal joint action-values. We base our implementation of WQMIX on the following repository: https://github.com/hijkzzz/pymarl2 RiskQ ( Shen et al. 2023 ) : introduces a quantile-based value factorization that models joint return distributions as weighted mixtures of per-agent utilities, satisfying the risk-sensitive IGM principle and enabling coordination under uncertainty. The official code can be found at: https://github.com/xmu-rl-3dv/RiskQ PAC ( Zhou et al. 2022 ) : leverages counterfactual predictions of optimal joint actions to provide assistive information for value factorization, using a novel counterfactual loss and variational encoding to improve coordination under partial observability. The official code can be found at: https://github.com/hanhanAnderson/PAC-MARL FOP ( Zhang et al. 2021 ) : factorizes the optimal joint policy in maximum-entropy MARL into individual actor-critic policies, with theoretical guarantees of convergence to the global optimum. The official code can be found at: https://github.com/PKU-RL/FOP-DMAC-MACPF?tab=readme-ov-file DOP ( Wang et al. 2020d ) : integrates value function decomposition into multi-agent actor-critic methods, enabling efficient off-policy learning while addressing credit assignment and centralized-decentralized mismatch, with guarantees of convergence. The official code can be found at: https://github.com/TonghanWang/DOP MARR ( Yang et al. 2024 ) : improves sample efficiency in MARL by introducing a reset strategy and data augmentation, enabling high-replay training in parallel environments with fewer environment interactions. The official code can be found at: https://github.com/CNDOTA/ICML24-MARR MASIA ( Guan et al. 2022 ) : enables efficient multi-agent communication by aggregating received messages into compact, task-relevant representations using a permutation-invariant encoder and self-supervised objectives, improving coordination and decision-making. The official code can be found at: https://github.com/chenf-ai/Multi-Agent-Communication-Considering-Representation-Learning NDQ ( Wang et al. 2019 ) : combines value function factorization with communication minimization, enabling agents to act independently most of the time while selectively exchanging messages using information-theoretic regularizers to improve coordination. The official code can be found at: https://github.com/TonghanWang/NDQ MAIC ( Yuan et al. 2022 ) : enables agents to generate targeted incentive messages that directly influence teammates’ value functions, promoting efficient explicit coordination while remaining compatible with different value function factorization methods. The official code can be found at: https://github.com/mansicer/MAIC CAMA ( Shao et al. 2023 ) : uses complementary attention to enhance high-contribution entities and compress low-contribution ones, addressing distracted attention and limited observability, thereby improving coordination in cooperative MARL. The official code can be found at: https://github.com/thu-rllab/CAMA T2MAC ( Sun et al. 2024 ) : enables agents to communicate selectively with trusted partners and integrate information at the evidence level, improving coordination and communication efficiency in cooperative MARL. The official code can be found at: https://github.com/ZangZehua/T2MAC Appendix G Additional Experiment Results In this section, we present additional experiment results to further demonstrate the effectiveness of our proposed S2Q method. In Appendice G.1 and G.2 , we provide additional performance evaluation on SMACv2 ( Ellis et al. 2023 ) and SMAC-Hard (Mixture Opponent) ( Deng et al. 2024 ) , while we extend S2Q to other representative CTDE baselines, VDN and QPLEX, in Appendix G.3 . G.1 Performance Comparison in SMACv2 Figure G.1: Performance comparison: Average test win rates in the SMACv2 tasks From Fig. G.1 , which shows the performance in SMACv2 scenarios, we can observe that S2Q demonstrates superior performance over the MARL baselines across all scenarios, achieving both superior performance and faster convergence. Notably, its advantage is most pronounced in the zerg_5_vs_5 and protoss_5_vs_5 scenarios, where the high degree of stochasticity induced by varying ally and enemy team compositions makes the environment particularly challenging. In this setting, S2Q demonstrates its ability to effectively track the values of suboptimal actions, thereby enabling more efficient exploration and guiding the Q^{\text{tot}} towards the optimal policy. G.2 Performance Comparison in SMAC-Hard (Mixture Opponent) Figure G.2: Performance comparison: Average test win rates in the SMAC-Hard (Mixture Opponent) tasks Fig. G.2 illustrates the performance on SMAC-Hard (Mixture Opponent) scenarios, including 5m_vs_6m_Hard , 2s3z_Hard , and 3s_vs_5z_Hard . As mentioned in Appendix E.5 , we append “ _Hard ” to each scenario name to distinguish from the the scenarios from the SMAC-Hard+ benchmark. According to the results, S2Q consistently outperforms all MARL baselines across every considered task, demonstrating its superiority. Notably, in the 3s_vs_5z_Hard scenario, where agents must coordinate precise hit-and-run maneuvers against opponents that exhibit diverse tactical behaviors, S2Q demonstrates outstanding adaptability. This suggests that S2Q captures subtle opponent-conditioned value structures that conventional MARL baselines fail to exploit. G.3 Generality of S2Q Across CTDE Methods Although our primary discussion of S2Q has been in the context of WQMIX, the proposed procedure is general and readily extends to other CTDE methods, such as VDN ( Sunehag et al. 2017 ) and QPLEX ( Wang et al. 2020a ) . By replacing the joint action-value function Q^{*} with Q^{\text{tot}} , S2Q can be integrated into these value-decomposition baselines without modification. Across all scenarios, we observed significant performance improvements over the corresponding baselines. Consistent with the results obtained in WQMIX, the performance gains were most evident in the challenging 6h_vs_8z environment, which requires extensive exploration. These results demonstrate the practicality and robustness of S2Q, highlighting its ability to follow changes in the value landscape more closely and adapt to new optima faster than conventional approaches. Table G.1: Performance comparison on SMACv2 environments Scenario VDN VDN+S2Q QPLEX QPLEX+S2Q 5m_vs_6m 62.47 \pm 11.12 68.25 \pm 4.78 58.18 \pm 3.52 61.86 \pm 2.65 MMM2 0.00 \pm 0.00 14.12 \pm 2.43 50.78 \pm 10.63 58.03 \pm 8.27 27m_vs_30m 13.75 \pm 5.33 61.35 \pm 2.33 47.34 \pm 8.26 63.18 \pm 5.54 Corridor 56.25 \pm 26.14 70.49 \pm 13.37 48.62 \pm 24.02 64.68 \pm 18.51 6h_vs_8z 8.94 \pm 4.31 48.03 \pm 9.37 3.44 \pm 1.13 41.32 \pm 6.23 3s5z_vs_3s6z 47.66 \pm 19.66 49.82 \pm 10.31 43.19 \pm 17.02 48.73 \pm 10.46 Appendix H Additional Ablation Study In this section, we provide a deeper analysis on S2Q. In Appendix H.1 , we provide further analysis on the hyperparameters K and T in additional tasks, while in Appendix H.2 , provide additional ablation study on the suppression constant \alpha and weighting factor w_{c} . Furthermore, in Appendix H.3 , we analyze the role of auxiliary signals in scenarios where communication is not critical, and in Appendix H.4 , provide additional examples of shift in value function along with trajectory analysis in SMAC-Comm environment. Finally, Appendix H.5 presents computational complexity analysis on S2Q, to show that the computational overhead from leveraging multiple sub-value functions remains moderate, compared to the significant performance gain of S2Q. H.1 Hyperparameter Analysis in Additional Tasks Figure H.1: Hyperparameter analysis. From left to right: (1) effect of K in MMM2 , (2) effect of temperature T in MMM2 , (3) effect K in academy_4_vs_3 , and (4) effect of T in academy_4_vs_3 To further emphasize the generality of S2Q beyond the settings examined in Section 5.3 , we additionally evaluate the effects of the number of sub-value networks K and the Softmax temperature T in two more challenging environments: MMM2 scenario from SMAC-Hard+ and academy_4_vs_3 scenario from GRF. Number of sub-value functions K : As in earlier results, K=2 achieves the best overall performance in both MMM2 and academy_4_vs_3 , offering the strongest balance between convergence speed and final win rate. In contrast, K=0 fails to provide meaningful exploratory diversity, while larger values such as K=3 introduce increased variance and mild instability. These observations further support that a moderate number of candidate sub-networks yields a favorable trade-off between diversity and stability. Softmax temperature T : A moderate temperature (e.g., T=0.1 ) provides the most effective balance between focused sampling and sufficient exploration. Very low temperature values ( T=0.01 ) result in overly deterministic behavior early in training, suppressing useful exploration, while higher temperatures (e.g., T=1.0 ) promote excessive exploration and slow convergence. Notably, T=0.2 performs competitively, indicating that S2Q remains robust to reasonable variations in temperature as long as sampling remains sufficiently concentrated on promising sub-values. Across both environments, we observe trends that closely mirror those reported earlier for 6h_vs_8z , demonstrating that the behavior of S2Q is consistent and robust across domains. Overall, the additional experiments on SMAC-Hard+ and GRF confirm that S2Q maintains stable and consistent performance characteristics across diverse and challenging multi-agent environments, reinforcing its generality and robustness. H.2 Ablation Study on Additional Hyperparameters (a) (b) Figure H.2: Hyperparameter analysis. Effect of (a) suppression constant \alpha and (b) weight factor w_{c} . Suppression constant \alpha : Fig. H.2 (a) evaluates the effect of the suppression constant \alpha\in[0.1,0.5,1.0,1.5,2.0] , which determines how strongly the values of previously identified actions are reduced and thereby controls how far sampled suboptimal actions lie from the current optimum. The results indicate that \alpha=1.0 yields the best performance: this setting encourages exploration of meaningful, moderately distant sub-actions that are informative for updating the value landscape. A smaller value, \alpha=0.5 , also performs well since it biases sampling toward nearby sub-actions that remain relevant to the optimum. In contrast, a very small value such as \alpha=0.1 fails to sufficiently suppress the previously selected optimal actions, preventing the discovery of meaningful suboptimal actions and resulting in a significant degradation in performance. On the other hand, larger values ( \alpha=1.5,2.0 ) substantially hurt performance: excessive suppression forces sampling of actions that are very distant from the optimum, producing updates that are less informative (and often misleading) for tracking the true optimal policy, which destabilizes learning and slows convergence. These findings suggest that a moderate suppression level is necessary to balance exploration breadth with the relevance of sampled sub-actions. Weighting factor w_{c} : Fig. H.2 (b) reports the effect of varying the weighting factor w_{c} , which determines how TD-error is scaled by reinforcing corrective signals that reduce underestimation. The results show that w_{c}=0.9 yields the highest win rates, striking a balance between effective error correction and stable value learning. A slightly smaller value, w_{c}=0.75 , also performs well, indicating robustness to moderate relaxations. However, more aggressive reductions such as w_{c}=0.5 and w_{c}=0.25 significantly weaken the propagation of TD-error, leading to information loss and degraded performance. These findings highlight that maintaining a relatively high w ensures that informative TD signals are consistently transmitted, while still guiding the value functions towards the optimal values. H.3 Effect of Auxiliary Supervision Figure H.3: Performance comparison: Average test win rates in the SMAC-Hard+ and GRF task for MARL algorithms and their communication variant To demonstrate that the performance gains of S2Q arise from the proposed successive sub-value learning, rather than the auxiliary latent information, we conduct additional experiments on SMAC-Hard+ scenarios, including MMM2 and 6h_vs_8z ) and academy_4_vs_3 scenario from GRF. In these environments, we equipped all non-communication baselines with the same encoder-decoder architecture used in S2Q-Comm, and denote these variants by appending “ -Comm ”. This setup ensures that every algorithm receives an identical latent signal derived from reconstructing the global state, thereby isolating the effect of the auxiliary representations from the effect of S2Q’s successive sub-value learning. The results, presented in Fig. H.3 show that injecting these latent-based auxiliary signals has minimal influence on performance across the considered tasks where communication is not critical for task success. In some cases, performance even degrades due to the latent introducing additional noise. Similar trends appear in the GRF academy_4_vs_3 environment, where communication is not the primary bottleneck. Notably, S2Q-Comm consistently outperforms all baselines, including their “-Comm” variants. These findings confirm that the advantage of S2Q-Comm does not originate from the auxiliary state-reconstruction signal or the latent variable itself. Instead, the gains stem from S2Q’s core mechanisms. H.4 Shift in Value Function and Additional Trajectory Analysis Figure H.4: Value function shift in 5m_vs_6m_Hard ) In addition to the motivation discussed in Section 4.1 and the trajectory analysis presented in Section 5.2 , we provide further examples of shifts in the value function. Specifically, we examine value-function changes in SMAC-Hard (Mixture Opponent) 5m_vs_6m_Hard and GRF academy_4_vs_3 , with results depicted in Fig. H.4 and H.5 . In 5m_vs_6m_Hard , agents initially focus on greedy attacks to secure easy eliminations, but the value function subsequently learns that repositioning to form favorable formations yields higher long-term returns, causing the optimal action to shift from attack to move . Figure H.5: Value function shift in academy_4_vs_3 Similarly, in academy_4_vs_3 , agents first attempt low-success long shots, but eventually discover that approaching the goal or passing increases scoring probability, shifting the optimal action from shoot to pass . These examples further highlight S2Q’s capability to track and adapt to evolving optimal behaviors across diverse environments. Figure H.6: Training behavior of S2Q. Left: training step = 0.3M. Right: training step = 2M In addition, we further investigate the training dynamics of S2Q ( K=2 ), we analyze the 5z_vs_1ul scenario from SMAC-Comm, where the agents must learn to coordinate based on the latent information z . 6h_vs_8z . Unlike scenarios that reward aggressive engagement, the optimal strategy in 5z_vs_1ul is to move and group with the distant ally, while direct attack actions often lead to heavy losses due to the enemy’s superior strength. As shown in Fig. H.6 (a), the action distribution reveals that agents initially favor attack , reflecting a local optimum that prioritizes immediate engagement. Consequently, the attack ratio increases in the early stages of training. However, as training progresses, S2Q enables agents to recognize the risks of premature aggression, resulting in a gradual decline in attack frequency and a corresponding rise in the use of move . Fig. H.6 (b) illustrates how this shift is accompanied by improvements in survival time and overall win rate. Fig. H.6 (c) highlights how S2Q adapts to this changing notion of optimality. Early in training, attack is tracked as the dominant action ( \mathbf{a}_{0,t}^{*} ), while move is considered suboptimal. As the value landscape evolves, Q^{*} reassigns higher returns to move , prompting S2Q to increase its execution frequency through the Softmax-based behavior policy. This dynamic reallocation allows Q_{0}^{\text{sub}} to align with the true optimal strategy, thereby reducing reliance on local optima. These findings emphasize that even in environments requiring implicit coordination and communication, such as 5z_vs_1ul , S2Q effectively leverages its successive sub-value component to explore, track, and ultimately converge to the optimal policy. H.5 Computational Complexity S2Q introduces additional computational overhead due to its use of multiple sub-value functions and the encoder-decoder architecture for coordinating k -selection. To evaluate this overhead and demonstrate that it remains minimal relative to performance benefits, we compare S2Q and QMIX in controlled experiments on an NVIDIA GeForce RTX 3090 GPU and an Intel Xeon Gold 6348 (2.60 GHz, 28 cores) processor running Ubuntu 20.04 with PyTorch. In MMM2 and 27m_vs_30m scenarios where QMIX achieves competitive performance, we measure the time required to reach a 50% win rate. Table H.1 summarizes the results. According to Table H.1 , S2Q incurs only a moderate increase in computation, approximately 8.4% for MMM2 and 3.5% for 27m_vs_30m . More importantly, due to faster learning enabled by successive sub-value function tracking, S2Q reaches the target win rate of 50% significantly faster than QMIX, demonstrating that the modest overhead is outweighed by the improved convergence speed and overall performance. Table H.1: Comparison of training time between S2Q (for different K ) and QMIX. Scenario Metric S2Q (K=1) S2Q (K=2) S2Q (K=3) QMIX MMM2 Time / 1M 122.0 min 129.3 min 136.5 min 119.3 min T at 50% win 1.48M 1.28M 1.67M 2.16M Time at 50% win 180.6 min 165.5 min 228.0 min 257.7 min 27m_vs_30m Time / 1M 241.0 min 245.1 min 258.0 min 236.8 min T at 50% win 1.39M 1.15M 1.57M 1.86M Time at 50% win 335.0 min 281.9 min 405.1 min 440.4 min In addition to time-wise comparison, we also evaluate GPU memory consumption to quantify the overhead introduced by maintaining multiple sub-value networks. Table H.2 reports peak GPU usage for S2Q with different values of K and for QMIX. According to the results, memory usage increases moderately with K due to the additional sub-networks, but the overall overhead remains reasonable: in MMM2 , S2Q with K=2 requires only 248 MB more memory than QMIX, and even K=3 remains within a 19% increase. A similar pattern holds in the more computationally demanding 27m_vs_30m scenario, where S2Q (K=2) uses 336 MB more memory than QMIX. The incremental cost from K=2 to K=3 is also modest, considering the added representational capacity. Importantly, this memory overhead scales predictably and linearly with K . Combined with the substantial reduction in training time reported in Table H.1 , these results demonstrate that S2Q achieves significant improvements in learning efficiency while incurring only limited computational and memory overhead. This confirms that the successive sub-value learning framework offers a favorable trade-off between memory usage and performance gains, even in large-scale SMAC scenarios. Table H.2: Comparison of memory usage between S2Q (for different K ) and QMIX. Scenario Metric S2Q (K=1) S2Q (K=2) S2Q (K=3) QMIX MMM2 Perk GPU usage (MB) 1967.0 MB 2087.0 MB 2188.0 MB 1839.0 MB 27m_vs_30m Perk GPU usage (MB) 3665.0 MB 3761.0 MB 4076.0 MB 3425.0 MB