Algorithm-Relative Trajectory Valuation in Policy Gradient Control
Shihao Li, Jiachen Li, Jiamin Xu, Christopher Martin, Wei Li, Dongmei Chen · 2025
Read Algorithm-Relative Trajectory Valuation in Policy Gradient Control by Shihao Li, Jiachen Li, Jiamin Xu, Christopher Martin, Wei Li, Dongmei Chen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-07878.txt.
Introduction
Shihao Li, Jiachen Li, and Jiamin Xu ask what makes a practice trajectory valuable, and find the answer depends on the algorithm watching it. Using Shapley-style valuation inside an uncertain linear control problem, trajectories with strong excitation look worse under plain REINFORCE training, a negative correlation near zero point four in magnitude.
A variance mechanism explains the flip. Strong excitation lowers gradient variance, but near saddle points higher variance helps escape, raising a trajectory worth. Stabilizers such as whitening or Fisher preconditioning kill that channel and the correlation turns positive. Data valuation for control must therefore name its algorithm, not pretend worth is universal.
Catalog record
| CREATOR | Shihao Li, Jiachen Li, Jiamin Xu, Christopher Martin, Wei Li, Dongmei Chen |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 38 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-07878
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Algorithm-Relative Trajectory Valuation in Policy Gradient Control Shihao Li, Jiachen Li, Jiamin Xu, Christopher Martin, Wei Li, Dongmei Chen Abstract We study how trajectory value depends on the learning algorithm in policy-gradient control. Using Trajectory Shapley in an uncertain LQR, we find a robust negative correlation between a trajectory’s information content—Persistence of Excitation (PE)—and its marginal value under vanilla REINFORCE (e.g., r ≈ − 0.38 r\!\approx\!-0.38 ). We prove a variance-mediated mechanism : (i) for fixed energy, higher PE yields lower gradient variance; (ii) near saddle regions, higher variance increases the probability of escaping poor basins and thus raises marginal contribution. When the update is stabilized (state whitening or Fisher preconditioning), this variance channel is neutralized and information content dominates, flipping the correlation positive (e.g., r ≈ + 0.29 r\!\approx\!+0.29 ). Hence, trajectory value is algorithm-relative : it emerges from the interaction between data statistics and update dynamics. Experiments on LQR validate the two-step mechanism and the flip, and show that decision-aligned scores (Leave-One-Out) complement Shapley for pruning near the full set, while Shapley remains effective for identifying high-impact (and toxic) subsets. † † volume: vol † † year: 2026 † † workshop: Learning for Dynamics & Control Conference (L4DC) 2026 keywords Data valuation, Shapley values, Policy gradient, Reinforcement learning, LQR control, Trajectory attribution, Persistence of excitation 1 Introduction Data-driven control via reinforcement learning (RL) has enabled impressive results in robotics and autonomy 7 ; 1 , yet the opacity of learned controllers complicates verification and trust in safety-critical applications 12 ; 21 . This motivates data attribution : understanding which training experiences most influence a learned policy. While classical system identification focuses on parameter estimation accuracy, data-driven control requires understanding which data most improves the final controller performance —a question of marginal contribution rather than information content alone. A natural intuition, rooted in system identification 10 , is that trajectory value scales with information content , quantified by Persistence of Excitation (PE). Modern ML attribution tools—Influence Functions 8 ; 15 and Shapley values 17 ; 3 —offer principled frameworks for quantifying these marginal performance contributions. However, policy-gradient (PG) learning introduces algorithm-data interactions absent in i.i.d. settings: temporal correlation, optimizer-dependent updates, and gradient variance 20 ; 19 . Key finding and explanation. Using Trajectory Shapley to value data for a PG controller on uncertain LQR 14 ; 22 ; 18 , we discover a robust negative correlation between PE and trajectory value under vanilla REINFORCE (Spearman r\approx-0.38 ). This contradicts the natural information-content intuition: high-PE trajectories, despite carrying more system knowledge, are valued less . We prove this is an algorithmic artifact arising from a variance-mediated mechanism. For fixed trajectory energy, higher PE yields lower gradient variance (Theorem 1 ). Near saddle points in parameter space, higher variance increases escape probability, raising marginal contribution (Theorem 2 ). Thus vanilla REINFORCE favors high-variance updates from low-PE trajectories for exploration, while underutilizing stable, informative gradients from high-PE data. Proof via algorithmic intervention. Stabilizing the optimizer (state whitening or natural gradient) neutralizes the variance channel, allowing information content to dominate. The PE-value correlation flips positive ( r\approx+0.29 ; Theorem 3 ), confirming our thesis: trajectory value is algorithm-relative —it emerges from data-algorithm interaction, not data alone. Contributions. We make four main contributions. First, we demonstrate a robust negative correlation between PE and trajectory value under vanilla policy gradient, contradicting the natural intuition from system identification that more informative data should be more valuable. Second, we develop a theoretical framework establishing the variance-mediated mechanism: high PE leads to low variance, which in turn reduces marginal value (Theorems 1 – 2 ). Third, we provide causal evidence via algorithmic stabilization (state whitening and natural gradient) showing that the negative correlation is an algorithmic artifact (Theorem 3 ). Fourth, we derive practical insights for data curation: decision-aligned Leave-One-Out valuation outperforms classical Shapley for pruning decisions, while Shapley excels at identifying high-impact and toxic trajectory subsets. Related work. Data valuation in supervised learning 3 ; 11 ; 9 and RL influence methods 5 ; 2 have not studied how PG algorithm design affects trajectory value. Work on trajectory importance 16 ; 13 focuses on sampling efficiency rather than attribution. We bridge control theory (PE, LQR) and ML (Shapley, gradient dynamics) to reveal algorithm-relativity in data valuation. Organization. Section 2 formalizes the LQR setting and Shapley valuation framework. Section 3 develops our theoretical framework in three steps: proving high PE yields low variance, showing high variance increases marginal value near saddles, and demonstrating that stabilization neutralizes this mechanism. Section 4 validates these predictions experimentally and derives practical curation insights. Section 5 discusses implications and limitations. 2 Problem Formulation and Background We study trajectory valuation for policy-gradient learning on the uncertain LQR problem. This section establishes the setting, introduces Trajectory Shapley as our valuation tool, and defines key quantities (PE, gradient variance). 2.1 Setting: Uncertain LQR with Policy Gradient System and objective. Consider discrete-time linear system x_{k+1}=Ax_{k}+Bu_{k}+w_{k},\quad w_{k}\sim\mathcal{N}(0,\Sigma_{w}), (1) with unknown (A,B) and stage cost \ell_{k}:=x_{k}^{\top}Qx_{k}+u_{k}^{\top}Ru_{k} ( Q\succeq 0 , R\succ 0 ). We minimize J(K):=\mathbb{E}[\sum_{k=0}^{H-1}\ell_{k}] . Stochastic policy and REINFORCE. The agent uses linear-Gaussian policy u_{k}=-Kx_{k}+\varepsilon_{k} with \varepsilon_{k}\sim\mathcal{N}(0,\sigma_{a}^{2}I_{m}) . For trajectory \tau=\{(x_{k},u_{k},r_{k})\}_{k=0}^{H-1} with r_{k}:=-\ell_{k} , the REINFORCE estimator is \hat{\nabla}_{K}J(\tau)=-\frac{1}{\sigma_{a}^{2}}\sum_{k=0}^{H-1}G_{k}\,\varepsilon_{k}x_{k}^{\top}, (2) where G_{k}:=\sum_{j=k}^{H-1}r_{j} . This unbiased but high-variance estimator makes learning sensitive to trajectory statistics. 2.2 Trajectory Shapley Valuation Characteristic function. Given dataset \mathcal{D}=\{\tau_{1},\ldots,\tau_{N}\} , training on S\subseteq\mathcal{D} for T steps yields final gain K_{T}(S) . Our characteristic function is v(S):=-J(K_{T}(S)). (3) Shapley value. Let \Pi_{N} denote permutations of \mathcal{D} and \mathrm{Pred}_{\pi}(i) be trajectories preceding \tau_{i} in permutation \pi . The Shapley value \phi_{i}=\mathbb{E}_{\pi\sim\Pi_{N}}\Big[v(\mathrm{Pred}_{\pi}(i)\cup\{\tau_{i}\})-v(\mathrm{Pred}_{\pi}(i))\Big] (4) measures expected marginal contribution, satisfying efficiency and symmetry. We approximate ( 4 ) via Monte Carlo sampling. Leave-One-Out (LOO) valuation. For comparison, the Leave-One-Out score is \text{LOO}_{i}:=v(\mathcal{D})-v(\mathcal{D}\setminus\{\tau_{i}\}), (5) measuring marginal contribution from the full dataset. Unlike Shapley, LOO is grand-coalition specific and computationally cheaper ( O(N) vs O(N\cdot 2^{N}) ). 2.3 Trajectory Characteristics Persistence of Excitation (PE). Define augmented vector z_{k}:=[x_{k}^{\top},u_{k}^{\top}]^{\top} and information matrix \mathcal{I}_{\tau}:=\sum_{k=0}^{H-1}z_{k}z_{k}^{\top} . PE is \mathrm{PE}(\tau):=\lambda_{\min}(\mathcal{I}_{\tau}) , measuring directional richness. High PE indicates balanced excitation across modes. Energy and state covariance. Trajectory energy is E(\tau):=\mathrm{tr}(\mathcal{I}_{\tau})=\sum_{k=0}^{H-1}(\|x_{k}\|^{2}+\|u_{k}\|^{2}) and state covariance is S_{x}(\tau):=\sum_{k=0}^{H-1}x_{k}x_{k}^{\top} . All comparisons condition on fixed energy E(\tau)=E_{0} to isolate directional richness from magnitude. Gradient variance. From ( 2 ), the gradient covariance has structure \mathrm{Cov}(\hat{\nabla}_{K}J(\tau))\propto\sum_{k=0}^{H-1}\mathbb{E}[G_{k}^{2}]\,(x_{k}x_{k}^{\top})\otimes I_{m}, (6) relating variance to state covariance S_{x}(\tau) . The information-value intuition. Classical system identification suggests value should scale with PE. Under this intuition, high-PE trajectories should receive higher Shapley values. We test whether \mathrm{PE}(\tau) correlates positively with \phi_{i} for vanilla policy gradient. 3 Theoretical Analysis We formalize the variance-mediated mechanism linking PE to trajectory value. Our analysis proceeds in three steps: (1) high PE implies low gradient variance, (2) near saddle points, higher variance increases escape probability and marginal contribution, (3) stabilizing the optimizer neutralizes the variance channel and flips the correlation. Key insight: variance aids exploration near saddles. Policy gradient landscapes for LQR often exhibit saddle points where gradient descent can stagnate. In these regions, gradient noise becomes beneficial: higher variance increases the probability of escaping poor local basins, analogous to simulated annealing or stochastic gradient descent with large learning rates. This exploration benefit can outweigh the value of information content, creating