PivotRL: High Accuracy Agentic Post-Training at Low Compute Cost
Junkeun Yi ∗,1 Damon Mosk-Aoyama ∗,1 Baihe Huang ∗,2 Ritu Gala 1 Charles Wang 1 Sugam Dipak Devare 1 Khushi Bhardwaj 1 Abhibha Gupta 1 Oleksii Kuchaiev 1 Jiantao Jiao ∗∗,1,2 Jian Zhang ∗∗,1 Venkat Srinivasan ∗∗,†,1 1 NVIDIA 2 UC Berkeley ∗ Equal Contribution ∗∗ Equal Advising † Corresponding Author: [email protected] · 2026
Read PivotRL: High Accuracy Agentic Post-Training at Low Compute Cost by Junkeun Yi ∗,1 Damon Mosk-Aoyama ∗,1 Baihe Huang ∗,2 Ritu Gala 1 Charles Wang 1 Sugam Dipak Devare 1 Khushi Bhardwaj 1 Abhibha Gupta 1 Oleksii Kuchaiev 1 Jiantao Jiao ∗∗,1,2 Jian Zhang ∗∗,1 Venkat Srinivasan ∗∗,†,1 1 NVIDIA 2 UC Berkeley ∗ Equal Contribution ∗∗ Equal Advising † Corresponding Author: [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-21383.txt.
Introduction
Yi and co-authors introduce PivotRL, a framework for agentic post-training that pairs low compute with high accuracy. Supervised fine tuning is cheap but degrades out of domain. End-to-end reinforcement learning keeps generalization yet costs many rollout turns. PivotRL works on existing fine tuning trajectories. It runs short local rollouts and filters for pivot turns, where sampled actions vary widely in outcome. Rewards credit functionally equivalent actions, not exact string matches. On coding tasks, it matches end-to-end RL with four times fewer rollout turns.
Catalog record
| CREATOR | Junkeun Yi ∗,1 Damon Mosk-Aoyama ∗,1 Baihe Huang ∗,2 Ritu Gala 1 Charles Wang 1 Sugam Dipak Devare 1 Khushi Bhardwaj 1 Abhibha Gupta 1 Oleksii Kuchaiev 1 Jiantao Jiao ∗∗,1,2 Jian Zhang ∗∗,1 Venkat Srinivasan ∗∗,†,1 1 NVIDIA 2 UC Berkeley ∗ Equal Contribution ∗∗ Equal Advising † Corresponding Author: [email protected] |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 48970 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.21383
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
PivotRL: High Accuracy Agentic Post-Training at Low Compute Cost Junkeun Yi ∗,1 Damon Mosk-Aoyama ∗,1 Baihe Huang ∗,2 Ritu Gala 1 Charles Wang 1 Sugam Dipak Devare 1 Khushi Bhardwaj 1 Abhibha Gupta 1 Oleksii Kuchaiev 1 Jiantao Jiao ∗∗,1,2 Jian Zhang ∗∗,1 Venkat Srinivasan ∗∗,†,1 1 NVIDIA 2 UC Berkeley ∗ Equal Contribution ∗∗ Equal Advising † Corresponding Author: [email protected] Abstract Abstract. Post-training for long-horizon agentic tasks has a tension between compute efficiency and generalization. While supervised fine-tuning (SFT) is compute efficient, it often suffers from out-of-domain (OOD) degradation. Conversely, end-to-end reinforcement learning (E2E RL) preserves OOD capabilities, but incurs high compute costs due to many turns of on-policy rollout. We introduce PivotRL , a novel framework that operates on existing SFT trajectories to combine the compute efficiency of SFT with the OOD accuracy of E2E RL. PivotRL relies on two key mechanisms: first, it executes local, on-policy rollouts and filters for pivots : informative intermediate turns where sampled actions exhibit high variance in outcomes; second, it utilizes rewards for functional-equivalent actions rather than demanding strict string matching with the SFT data demonstration. We theoretically show that these mechanisms incentivize strong learning signals with high natural gradient norm, while maximally preserving policy probability ordering on actions unrelated to training tasks. In comparison to standard SFT on identical data, we demonstrate that PivotRL achieves + 4.17 % +4.17\% higher in-domain accuracy on average across four agentic domains, and + 10.04 % +10.04\% higher OOD accuracy in non-agentic tasks. Notably, on agentic coding tasks, PivotRL achieves competitive accuracy with E2E RL with 4 × 4\times fewer rollout turns. PivotRL is adopted by NVIDIA’s Nemotron-3-Super-120B-A12B, acting as the workhorse in production-scale agentic post-training. 1 Introduction Long-horizon agentic tasks require many turns of Large Language Model (LLM) interaction with an environment. This includes tasks such as conversational tool use ( Schick et al. 2023 ; Qin et al. 2023 ) , agentic coding ( Jimenez et al. 2024 ) , terminal interaction ( Xie et al. 2024 ) , and web search ( Yao et al. 2022 ; Wei et al. 2025 ) . These tasks have emerged as the frontier in AI systems capable of executing complex, real-world workflows. However, post-training for these long-horizon agentic capabilities introduces a fundamental tension between training efficiency and generalization. While supervised fine-tuning (SFT) offers a compute-efficient mechanism for acquiring new capabilities, it frequently struggles to generalize beyond the training distribution and catastrophically degrades out-of-domain (OOD) performance ( Chu et al. 2025 ; Luo et al. 2025b ) . In contrast, end-to-end reinforcement learning (E2E RL) typically yields higher in-domain accuracy while robustly retaining OOD capabilities ( Ouyang et al. 2022 ; Chen et al. 2025 ) . Nevertheless, E2E RL incurs high compute overheads, as each parameter update necessitates repeated, many turns of on-policy rollout with environmental interactions. This dichotomy naturally raises the following question: Can we combine the data efficiency of SFT with the generalization capabilities of E2E RL, achieving both in-domain accuracy and OOD retention without incurring full-trajectory rollouts? To address this challenge, a natural attempt would be to repurpose SFT trajectory demonstrations for RL. Specifically, one could sample on-policy rollouts conditioned on a randomly selected intermediate turn from an SFT trajectory, assigning a positive reward only if the generated action exactly matches the SFT data demonstration. While this mitigates the under-utilization of SFT data which only trains on a single demonstrated completion, our preliminary experiments reveal that it fails to improve OOD accuracy relative to standard SFT on the same data. We empirically trace this failure to two bottlenecks. First, randomly selected intermediate turns frequently provide negligible learning signals; under Group Relative Policy Optimization (GRPO) ( Shao et al. 2024 ) , sampled actions at such turns often uniformly succeed or fail, yielding a normalized advantage close to zero and thus producing no meaningful gradient update. Second, exact string matching with SFT data is excessively restrictive in generative action spaces, where numerous functionally equivalent actions validly diverge from the singular demonstrated completion. For instance, a tool call or search query may be perfectly appropriate despite not perfectly matching the demonstration. Motivated by these insights, we introduce PivotRL , a novel framework for long-horizon agentic RL to overcome both bottlenecks simultaneously. PivotRL executes local, on-policy rollouts and filters for pivots : informative assistant turns where sampled actions exhibit mixed outcomes (i.e., both successes and failures). Consequently, training requires only brief, partial rollouts from these pivot states rather than exhaustive full trajectories. To effectively score these rollouts, we leverage domain-appropriate verifiers to assign rewards for functionally equivalent actions rather than strictly penalizing deviations from SFT data demonstrations ( Shao et al. 2024 ; Rohatgi et al. 2025 ) . We substantiate our methodology with a lightweight theoretical analysis on our core design choices. First, we prove that the Fisher norm of the natural gradient of the statewise reward objective scales with the reward standard deviation. Consequently, the GRPO update along the KL path scales directly with this variance, validating our strategy of filtering for mixed-outcome pivots to maximize the local in-domain learning signals. Second, we show that functional reward-based RL shifts probability mass toward actions that are functionally equivalent to the expert demonstration, while preserving the conditional distributions on all other actions. This maintains the reference policy’s relative ordering of task-unrelated actions, thereby mitigating out-of-domain (OOD) degradation. Empirically, we first show that under identical training data—the same prompts and expert trajectories—PivotRL yields larger in-domain improvements than SFT while exhibiting substantially less OOD regression. When training in the conversational tool use, agentic coding, terminal tool use and search domain, PivotRL achieves an average in-domain accuracy improvement of +14.11 1 1 1 Throughout this paper, performance scores and differences are reported in percentage points. over the base model, which significantly improves the +9.94 points attained by SFT. Crucially, PivotRL does not introduce OOD regressions (+ 0.21 change), whereas SFT suffers a severe regression of -9.48 in non-agentic domains including math, science QA and competitive coding. Second, we provide a direct comparison with E2E RL on SWE-Bench, a rigorous standard for evaluating long-horizon agentic capabilities with well-established E2E RL baselines ( Jimenez et al. 2024 ; Wang et al. 2025b ) . On this benchmark, we demonstrate that PivotRL attains competitive accuracy with E2E RL, but only with a 4\times fewer rollout turns. PivotRL is deployed at production scale for leading open LLMs. Along with SFT and E2E RL, it serves as the primary workhorse for agentic post-training for NVIDIA’s Nemotron-3-Super LLM ( Team 2026 ) . 2 Preliminaries and Motivating Observations Turn-level agentic training. Let \tau be an interaction trajectory collected in agentic tasks. Decomposing \tau at assistant decision boundaries gives \tau=(s_{0},a_{0}^{*},\ldots,s_{T},a_{T}^{*}) , where s_{t} is the full interaction history from the beginning of trajectory \tau_{i} up to, but not including, the t -th assistant action, and a_{t}^{*} is the demonstrated assistant completion at that state. In turn-level agentic training, an action is the full assistant completion at a model-call boundary, instead of an individual token. The standard negative log-likelihood loss for SFT is given by \displaystyle\mathcal{L}_{\mathrm{sft}}(\theta)=-\mathbb{E}_{\tau\sim\mathcal{D}_{\mathrm{sft}},(s_{t},a_{t}^{*})\sim\tau}[\log\pi_{\theta}(a^{*}_{t}\mid s_{t})]. Let \pi_{0} denote the reference policy, E2E RL samples rollouts from \pi_{\mathrm{old}} and optimizes the GRPO ( Shao et