Revisiting Action Factorization for Complex Action Spaces Journal: TAAS CCS: Computing methodologies Reinforcement learning

Timothy Flavin, Sandip Sen · 2026

Read Revisiting Action Factorization for Complex Action Spaces Journal: TAAS CCS: Computing methodologies Reinforcement learning by Timothy Flavin, Sandip Sen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-26574.txt.

Introduction

Timothy Flavin and Sandip Sen survey how reinforcement learning agents should split complex actions into parts. Many control problems mix discrete choices with continuous ones. Driving mixes steering with signalling; games mix aiming with firing. This split is called action factorization. Frameworks support many factorization methods, but their default test tasks use simple uniform action spaces. Landmark hybrid benchmarks are heavy or outdated.

The authors ran a broad cross-sectional study. Six factorization methods met three learning algorithms across three action space types in four light environments. That yielded two hundred twenty valid configurations to compare. Two new fast environments were built to isolate specific challenges. The study maps which factorizations suit which settings. It gives practitioners evidence for a choice that is usually made by habit.

Catalog record

CREATORTimothy Flavin, Sandip Sen
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE89 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.26574

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Revisiting Action Factorization for Complex Action Spaces Journal: TAAS CCS: Computing methodologies Reinforcement learning Timothy Flavin, Sandip Sen Abstract . Many real-world control problems involve hybrid discrete-continuous action spaces. For example, steering and signaling in autonomous driving, and aiming and firing in robotics or video-games. Despite real-world hybrid factorization and reinforcement learning framework support for complex action spaces (e.g., Gymnasium, PettingZoo, TorchRL, SeedRL, Mujoco, etc), the default environments within those frameworks often implement uniform action space configurations (LunarLander, Walker2D, Cheetah, SMAC, SUMO, Ant, Atari). Landmark hybrid-action benchmarks (RoboCup 2D HFO, SC2LE, Platform, CARLA, etc) are mostly heavyweight or archival implementations originating from papers which test one or a small number of competing factorization methods on one kind of control. This article provides a cross-sectional study of factorization methods [independent networks, shared encoder, VDN, QPLEX, Joint, Auto-Regressive] on each of three families of algorithms [PPO, SAC, DQN] across three action spaces [discretized, hybrid, continuous] over four lightweight environments [Platform, hybrid-LunarLander, Hybrid-Shoot, CoopPush]. Accounting for some invalid pairings such as joint-continuous, we are left with 220 configurations to analyze each method. We provide two new C++ parallel gymnasium and petting-zoo compliant environments [CoopPush, Hybrid-Shoot] to isolate particular challenges such as state-dependent inter-action dependence. Finally, we introduce VDN-PPO and PPO-MIX which use a branching critic to assign credit to multi-headed PPO. These variants out-perform all other tested PPO factorizations. Our results suggest that branching dueling architectures balance compute and performance most effectively, with Auto-Regressive actions reaching the highest performance overall and native continuous SAC outperforming discrete and hybrid algorithms, albeit both at increased computational cost. 1. Introduction While neural networks have been widely accepted in Deep Reinforcement Learning (D-RL) for their capacity to generalize over dependent hybrid feature spaces, there remains an open debate on factorizing complex action spaces. In the single-agent domains, sub-actions \mathcal{A}_{1},\ldots,\mathcal{A}_{H} may control individual motor torques or buttons that combine to form the agent’s full joint-action space \mathcal{A}=\prod_{h=1}^{H}\mathcal{A}_{h} (exponential size in the number of sub-actions). In multi-agent RL (MARL) actions \mathcal{A}_{1},\ldots,\mathcal{A}_{K} refers to individual agents’ action spaces which may themselves be decomposed into sub-actions. Not all actions are interdependent, so the full joint action space need not always be explored. For example, the value of movement that avoids danger may not depend upon where an agent is looking or firing, but the decision to fire is highly dependent upon where the agent is aiming. There are a range of approaches in the literature covering the spectrum from full independence to joint action learning (JAL). In this paper we cover the following general strategies: (1) Independent: Each sub-action is sampled from a fully independent policy network \pi_{h}(a_{h}\mid s) , with all networks updated using the same global training signal. (2) Shared Encoder: A shared state representation \phi(s) branches into multiple heads: (a) No Mixing: Each action head is exposed to the global reward signal where the impact of the other heads is considered independent noise. The gradients combine in the encoder so the result is not truly independent. (b) Value Decomposition (VDN-style): A single state-value V(s) and per-head advantages A_{h}(s,a_{h}) are learned. The joint action-value is additively factored: Q(s,\mathbf{a})=V(s)+\sum_{h=1}^{H}A_{h}(s,a_{h}) . (c) Monotonic (QPLEX-style): A single state-value V(s) is learned, and per-head advantages are combined via a monotonic mixing network: Q(s,\mathbf{a})=V(s)+f_{\text{mix}}\big(A_{1}(s,a_{1}),\dots,A_{H}(s,a_{H})\big) , enforcing \frac{\partial f_{\text{mix}}}{\partial A_{h}}\geq 0 . (d) Concatenated SAC (SAC-Concat): The actor jointly samples continuous dimensions via a squashed Gaussian and discrete dimensions via Gumbel-Softmax. The critic treats these as a single input vector \mathbf{a}=[a_{c},a_{d}] to approximate a unified action-value Q(s,\mathbf{a}) . (e) Branching Dueling SAC (SAC-BDQ): The critic utilizes a branching architecture conditioned on the continuous actions. The network encodes the state and continuous actions to compute a base value V(s,a_{c}) and branches to output discrete advantages A_{h}(s,a_{d,h}) per categorical head, yielding Q(s,a_{c},\mathbf{a}_{d})=V(s,a_{c})+\sum_{h=1}^{H}A_{h}(s,a_{d,h}) . (3) Auto-Regressive (AR): The joint policy is factored sequentially using the chain rule. Each action dimension h conditions on the state s and previously sampled actions a_{<h}=(a_{1},\dots,a_{h-1}) . Sub-actions are drawn as a_{h}\sim\pi_{h}(\cdot\mid s,a_{<h}) , yielding the joint policy \pi(\mathbf{a}\mid s)=\prod_{h=1}^{H}\pi_{h}(a_{h}\mid s,a_{<h}) . (4) Joint: The action space is the Cartesian product of all sub-actions \mathcal{A}=\prod_{h=1}^{H}\mathcal{A}_{h} . Continuous actions can be discretized into bins for a single Multinomial policy distribution. While these factorization strategies exist for each algorithm individually, their performance is often evaluated in isolation within a single algorithmic family (e.g., value-based methods) on a set of benchmarks that may or may not be available for comparison across families. It remains an open question how the effectiveness of a given strategy, such as value decomposition, translates between value-based (DQN) ( 22 ) , policy gradient (PG,PPO) ( 26 ; 31 ) , and actor-critic (DDPG,SAC) ( 10 ; 17 ) algorithms. Furthermore, it is unclear how the choice between discrete, continuous, or hybrid action spaces interacts with these factorization methods. For example, continuous actions can approach arbitrary precision, but mixing continuous and discrete gradients may destructively interfere in a shared encoder. Auto-regressive actions can tractably factorize a large joint space, but the “best” action requires a sequence of evaluations that adds learning variance, inference latency, and potential implementation complexity by normalizing flows ( 25 ) . This paper provides a cross-sectional comparison of factorization methods with regards to runtime, performance, and implementation effort, to guide future researchers on model selection. In order to measure performance, a tunable environment is needed. Benchmarks like Gymnasium ( 37 ) and Atari100k ( 45 ) for single agent discrete RL, PettingZoo ( 35 ) and SMACv2 ( 6 ) for cooperative MARL, and Deepmind ( 33 ) and Berkeley’s ( 5 ) continuous control suites are imperative for developing and contextualizing new algorithms and their ability to scale to reasonably complex tasks. We believe that a principled benchmark environment is missing for factorization with tunable inter-action dependence, action-rate, and action datatype. The remainder of this paper is structured as follows. Section 4 introduces the four benchmark environments including our two novel contributions. Section 5 describes each model family and introduces VDN-PPO and PPO-MIX. Section 6 details the experimental setup and runtime normalization methodology. Section 7 presents results and analysis. Our key findings are: (i) shared encoder architectures provide the best compute to performance trade-off for most settings. (ii) VDN-PPO and PPO-MIX substantially outperform shared-encoder PPO on discrete action spaces by redistributing credit to action heads with higher state agency. (iii) monotonic mixing (QPLEX) does not improve over shared encoder in simple single-agent settings. (iv) Action type (discrete, hybrid, continuous) has less impact on performance than factorization strategy. 2. Background / Preliminaries 2.1. Reinforcement Learning Standard single-agent reinforcement learning problems are often modeled as a Markov Decision Process (MDP) consisting of the tuple (\mathcal{S},\mathcal{A},P,R,\gamma) . \mathcal{S}\subset\mathbb{R}^{N} is the set of all environment states, \mathcal{A}=\prod_{h=1}^{H}\mathcal{A}_{h} is the joint action space factored over H action heads, P:\mathcal{S}\times\mathcal{A}\rightarrow\Delta(\mathcal{S}) the transition distribution, and R:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R} is the reward function with \gamma as the discount factor and r_{t}\in R the reward at time t. We denote the state and joint action at time t as s_{t},\mathbf{a}_{t} sampled from factored policy \pi(\mathbf{a}_{t}|s_{t})=\prod_{h=1}^{H}\pi_{h}(a_{h,t}\mid s_{t}) . Our goal is to find a policy \pi that maximizes the expected sum of discounted rewards, \mathbb{E}[G_{t}] where G_{t}=\sum_{\tau=t}^{\infty}\gamma^{\tau-t}R(s_{\tau},\mathbf{a}_{\tau}) and \mathbf{a}_{t}\sim\pi(\cdot|s_{t}),s_{t+1}\sim P(\cdot|s_{t},\mathbf{a}_{t}) . We do not know P or R in

Read the complete text