Logit Dynamics in Softmax Policy Gradient Methods

Yingru Li · 2025

Read Logit Dynamics in Softmax Policy Gradient Methods by Yingru Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-12912.txt.

Introduction

Yingru Li presents a 2025 analysis of the step sizes inside softmax policy gradient methods, a core tool for teaching agents by reward. Each update shifts the log-scores of the acts, and those shifts set how fast learning runs. The author derives an exact formula for the shift's length. It hangs on two gauges: the chance of the act just taken, and a clash gauge that falls as the policy spreads out.

Together they expose a built-in self-regulation. A sure policy takes small steps, while a torn policy strides. Learning vigor thus tunes itself to confidence with no outside hand. That insight helps explain why these methods stay stable in practice. Trainers gain a lens for reading runs gone slow or wild.

Catalog record

CREATORYingru Li
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE16 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-12912

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Logit Dynamics in Softmax Policy Gradient Methods Yingru Li Abstract We analyzes the logit dynamics of softmax policy gradient methods. We derive the exact formula for the L2 norm of the logit update vector: ‖ Δ ​ 𝐳 ‖ 2 ∝ 1 − 2 ​ P c + C ⁡ ( P ) \|\Delta\mathbf{z}\|_{2}\propto\sqrt{1-2P_{c}+C(P)} This equation demonstrates that update magnitudes are determined by the chosen action’s probability ( P c P_{c} ) and the policy’s collision probability ( C ⁡ ( P ) C(P) ), a measure of concentration inversely related to entropy. Our analysis reveals an inherent self-regulation mechanism where learning vigor is automatically modulated by policy confidence, providing a foundational insight into the stability and convergence of these methods. 1 Introduction Policy gradient methods [ 2 , 1 ] update parameters based on the product of three factors: learning rate, advantage estimate, and score function. In softmax policies, the score function has a specific structure that creates rich, self-regulating dynamics. This analysis dissects these dynamics by focusing on the logits—the raw outputs of a model before the softmax normalization. By focusing on the dynamics of the logits themselves, we can gain fundamental insights into the learning process that are independent of the specific policy parameterization \theta (e.g., whether the policy is a linear model or a deep neural network). This analysis reveals the intended change in the policy’s output distribution, offering a clear view of the adaptive mechanisms inherent in the softmax function. 2 Mathematical Foundation This section outlines the core mathematical concepts, clarifying how the general policy gradient update translates into specific, structured updates for the policy’s logits. 2.1 Policy Gradient Framework The policy gradient theorem [ 2 ] provides the gradient of the expected return J(\theta) with respect to policy parameters \theta : \nabla_{\theta}J(\theta)=\mathbb{E}_{s,a}\left[\nabla_{\theta}\log\pi_{\theta}(a|s;\theta)A(s,a)\right] (1) Here, A(s,a) is the advantage function, and \pi_{\theta}(a|s;\theta) is the policy. For softmax policies, the probability of selecting action a_{i} given state s is determined by logits z_{k}(s,\theta) : \pi_{\theta}(a_{i}|s;\theta)=\frac{e^{z_{i}(s,\theta)}}{\sum_{k}e^{z_{k}(s,\theta)}}\triangleq P_{i} (2) The term P_{i} serves as a shorthand for this probability. 2.2 From Parameter Gradients to Logit Updates The update to the parameters \theta is driven by the gradient \nabla_{\theta}\log\pi_{\theta} . Using the chain rule, we can decompose this gradient’s effect on the logits \mathbf{z} : \nabla_{\theta}\log\pi_{\theta}=\left(\frac{\partial\mathbf{z}}{\partial\theta}\right)\left(\nabla_{\mathbf{z}}\log\pi_{\theta}\right) (3) The term \frac{\partial\mathbf{z}}{\partial\theta} is the Jacobian of the logits with respect to the parameters, which depends on the specific architecture of the policy network. The second term, \nabla_{\mathbf{z}}\log\pi_{\theta} , captures the desired change in the logits to increase the probability of an action. This paper analyzes the dynamics of this second term. We define the logit update , \Delta z_{j} , as the effective change to the logit z_{j} that the algorithm seeks to induce for a given experience (state s , chosen action a_{c} , advantage A ), scaled by a learning rate \eta . This provides a parameterization-agnostic view of the learning update. 2.3 The Logit Score Function and Update Equations The gradient of the log-policy with respect to the logits, which we can call the logit score function, takes a remarkably simple form: Fundamental Score Function for Softmax Logits \frac{\partial\log\pi_{\theta}(a_{i}|s;\theta)}{\partial z_{j}(s,\theta)}=\delta_{ij}-P_{j} (4) where \delta_{ij} is the Kronecker delta (1 if i=j , 0 otherwise), and P_{j}=\pi_{\theta}(a_{j}|s;\theta) . The logit update, \Delta z_{j}(s,\theta) , follows from applying the principle of stochastic gradient ascent to the objective. For a single experience (s,a_{c},A) , the update is: \Delta z_{j}(s,\theta)=\eta\cdot A\cdot\left(\frac{\partial\log\pi_{\theta}(a_{c}|s;\theta)}{\partial z_{j}(s,\theta)}\right) (5) Substituting the score function from Equation 4 (with i=c ) into Equation 5 yields the detailed logit update equations: Logit Update Equations For action a_{c} chosen with advantage A\equiv A(s,a_{c}) : \displaystyle\Delta z_{c}(s,\theta) \displaystyle=\eta(1-P_{c})A (chosen action’s logit) (6) \displaystyle\Delta z_{o}(s,\theta) \displaystyle=-\eta P_{o}A \displaystyle\text{(other actions' logits, }o\neq c\text{)} (7) An important consequence of these update rules is the conservation of the sum of logit changes: Conservation Property The sum of logit updates is zero: \sum_{j}\Delta z_{j}(s,\theta)=\eta A\left((1-P_{c})+\sum_{o\neq c}(-P_{o})\right)=\eta A\left(1-\sum_{j}P_{j}\right)=\eta A(1-1)=0 (8) This zero-sum property is significant. It shows that the updates adjust the relative strengths of the logits without altering their sum. This ensures that changes to the probability distribution arise from a structured redistribution of probability mass, rather than from a uniform shift across all logits (which the softmax function would render irrelevant to the final probabilities anyway). 3 Analysis of Logit Update Dynamics 3.1 Update Dynamics for Chosen Action The factor (1-P_{c}) creates a natural modulation of update strength, as described in Table 1 : Table 1: Logit Dynamics for Chosen Action A Sign P_{c} Update Scaler Update Magnitude Dynamics A>0 0.1 0.9 \uparrow\uparrow\uparrow Large Rapid increase 0.5 0.5 \uparrow\uparrow Moderate Steady increase 0.9 0.1 \uparrow Small Slow increase A<0 0.1 0.9 \downarrow\downarrow\downarrow Large Rapid decrease 0.5 0.5 \downarrow\downarrow Moderate Steady decrease 0.9 0.1 \downarrow Small Slow decrease 3.2 Update Dynamics for Other Actions Other actions receive updates proportional to their current probabilities, as described in Table 2 : Table 2: Logit Dynamics for Other Actions A Sign P_{o} Update Scaler Update Magnitude Effect A>0 0.1 -0.1 \downarrow Small Minor reduction 0.5 -0.5 \downarrow\downarrow Moderate Clear reduction 0.8 -0.8 \downarrow\downarrow\downarrow Large Major reduction A<0 0.1 -0.1 \uparrow Small Minor increase 0.5 -0.5 \uparrow\uparrow Moderate Clear increase 0.8 -0.8 \uparrow\uparrow\uparrow Large Major increase 4 Update Magnitude and Collision Probability 4.1 The Update Magnitude Formula The total magnitude of the logit update vector for an experience is: \|\Delta z\|_{2}=\sqrt{\sum_{j}(\Delta z_{j})^{2}}=\sqrt{(\eta(1-P_{c})A)^{2}+\sum_{o\neq c}(-\eta P_{o}A)^{2}}=\eta|A|\sqrt{(1-P_{c})^{2}+\sum_{o\neq c}P_{o}^{2}} (9) By expanding (1-P_{c})^{2}=1-2P_{c}+P_{c}^{2} and noting that P_{c}^{2}+\sum_{o\neq c}P_{o}^{2}=\sum_{a}P_{a}^{2} , this can be rewritten in a more revealing form: Logit Update Magnitude Formula \|\Delta z\|_{2}=\eta|A|\sqrt{1-2P_{c}+C(P)} (10) where C(P)=\sum_{a}P_{a}^{2} is the collision probability. 4.2 Understanding Collision Probability The collision probability C(P) is a fundamental measure of distribution concentration: Key Insight Collision Probability Properties: • Definition : C(P)=\sum_{a}P_{a}^{2} - the probability that two independent samples from P are identical • Range : 1/n\leq C(P)\leq 1 for n actions • Minimum : C(P)=1/n when P is uniform (maximum entropy/spread) • Maximum : C(P)=1 when P is deterministic (minimum entropy/concentration) • Information theoretic : C(P)=e^{-H_{2}(P)} where H_{2} is the Rényi entropy of order 2 4.3 How Collision Probability Shapes Learning The logit update magnitude formula reveals how C(P) controls learning dynamics: \|\Delta z\|_{2}=\eta|A|\underbrace{\sqrt{1-2P_{c}+C(P)}}_{\text{sensitivity factor}} (11) This sensitivity factor depends on two components: 1. Action-specific term : (1-2P_{c}) - larger when choosing an unlikely action ( P_{c}\to 0 ). 2. Distribution-wide term : C(P) - larger for more concentrated (low-entropy) distributions. Key Dynamics: • When P_{c}\approx 0 (exploring): \|\Delta z\|_{2}\approx\eta|A|\sqrt{1+C(P)} . The update is large, especially if the policy is already concentrated elsewhere (high C(P) ). • When P_{c}\approx 1 (exploiting): \|\Delta z\|_{2}\approx\eta|A|\sqrt{1-2+1}=0 , since P_{c}\approx 1 implies C(P)\approx 1 . Confident choices lead to minimal adjustments. • Maximum possible: \|\Delta z\|_{2}\approx\eta|A|\sqrt{2} when P_{c}\to 0 and another action is deterministic, so C(P)\to 1 . 5 Conclusion The mathematical structure of softmax policy gradients, when analyzed through its effect on logits, reveals an elegant self-regulating system for learning. Our detailed analysis of these logit dynamics highlights several key aspects: 1. Probability-Dependent Individual Logit Updates: The change to the chosen action’s logit ( \Delta z_{c} ) is scaled by (1-P_{c}) , ensuring that rarely-chosen actions (low P_{c} ) receive proportionally larger updates, facilitating rapid learning from surprising experiences. Conversely, highly probable actions ( P_{c}\approx 1

Read the complete text