DISA: Offline Importance Sampling for Distribution-Matching LLM-RL

Shaobo Wang Yujie Chen Yafeng Sun Wenjie Qiu Zhihui Xie, Sihang Li Yucheng Li Huiqiang Jiang Xingzhang Ren Xuming Hu, Dayiheng Liu Linfeng Zhang, Shanghai Jiao Tong University Qwen Team, Alibaba Group, The Hong Kong University of Science and Technology (Guangzhou), The University of Science and Technology of China, Nanjing University The University of Hong Kong · 2026

Read DISA: Offline Importance Sampling for Distribution-Matching LLM-RL by Shaobo Wang Yujie Chen Yafeng Sun Wenjie Qiu Zhihui Xie, Sihang Li Yucheng Li Huiqiang Jiang Xingzhang Ren Xuming Hu, Dayiheng Liu Linfeng Zhang, Shanghai Jiao Tong University Qwen Team, Alibaba Group, The Hong Kong University of Science and Technology (Guangzhou), The University of Science and Technology of China, Nanjing University The University of Hong Kong (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-17295.txt.

Introduction

Shaobo Wang and colleagues introduce DISA, a method that moves partition-function calibration for distribution-matching reinforcement learning out of the training loop. Proposal trajectories are drawn offline, the partition function is estimated by importance sampling, and that estimate is frozen before policy training starts. On two backbones across nine math and code benchmarks it matches or beats the coupled baseline and keeps far more strategic diversity than reward-maximizing rivals.

Catalog record

CREATORShaobo Wang Yujie Chen Yafeng Sun Wenjie Qiu Zhihui Xie, Sihang Li Yucheng Li Huiqiang Jiang Xingzhang Ren Xuming Hu, Dayiheng Liu Linfeng Zhang, Shanghai Jiao Tong University Qwen Team, Alibaba Group, The Hong Kong University of Science and Technology (Guangzhou), The University of Science and Technology of China, Nanjing University The University of Hong Kong
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE62987
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.17295

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

DISA: Offline Importance Sampling for Distribution-Matching LLM-RL Shaobo Wang Yujie Chen Yafeng Sun Wenjie Qiu Zhihui Xie, Sihang Li Yucheng Li Huiqiang Jiang Xingzhang Ren Xuming Hu, Dayiheng Liu Linfeng Zhang, Shanghai Jiao Tong University Qwen Team, Alibaba Group, The Hong Kong University of Science and Technology (Guangzhou), The University of Science and Technology of China, Nanjing University The University of Hong Kong Abstract Modern reasoning agents are increasingly evaluated on their ability to generate multiple valid solution paths, plans, or tool-use traces for a given input. Standard reward-maximizing RL tends to collapse onto the most easily reinforced high-reward mode, whereas distribution-matching RL aims to allocate probability mass across the entire reward-shaped solution set. Achieving this objective requires computing a prompt-dependent partition function over the trajectory space. Because existing distribution-matching methods learn this partition function online alongside the policy, calibration errors in the partition function directly distort policy updates and remain impossible to diagnose independently. We introduce DISA , short for Decoupled Importance-Sampled Anchoring , which moves this calibration problem outside the RL loop. DISA draws proposal trajectories offline, estimates the partition function via importance sampling, and freezes the resulting partition-function estimate before policy optimization begins. This decoupling preserves the distribution-matching objective while strictly separating partition-function estimation from policy learning in data, gradients, loss, and diagnostics. Theoretically, our analysis shows that an exact frozen partition function preserves the reward-tilted optimum, while residual offline estimation error enters through a bounded perturbation envelope. Empirically, on two open-weight backbones across six math and three code benchmarks, DISA matches or exceeds the online-coupled distribution-matching baseline FlowRL, outperforms reward-maximization baselines GRPO and GSPO on math averages, and exceeds LoRA-SFT distillation by up to 13.8 13.8 Mean@8 points on the same offline trajectories. An LLM-as-judge evaluation further shows that DISA retains substantially more strategy-level diversity than reward-maximization baselines, and sensitivity studies on the proposal strength and inverse temperature follow the bias-variance pattern predicted by the analysis. 1 Introduction Reasoning LLMs and LLM-based agents increasingly produce long trajectories: chain-of-thought solutions span many steps ( 40 ) , agentic systems add planning and tool use ( 9 ) , and inference often aggregates multiple samples per prompt ( 39 ; 18 ; 42 ; 28 ) . In these settings, a post-trained model should not only place high probability on one answer; it should produce several distinct correct trajectories for the same prompt. This calls for post-training objectives that recover the reward landscape while preserving diversity across repeated samples ( 49 ; 19 ; 7 ) . Figure 1 : Two families of post-training and the gradient-backflow asymmetry that distinguishes them. (a) On a multi-modal reward r(q,\cdot) , reward maximization (PPO/GRPO/GSPO) collapses to a single mode, while distribution matching targets \pi_{\mathrm{ref}}e^{\beta r}/Z(q) and preserves multi-modal structure. (b) Prior methods co-train the partition function Z_{\phi} with \pi_{\theta} on a shared loss, so partition-function error rides into \nabla_{\theta} . DISA replaces Z_{\phi} with a frozen partition function g_{\psi^{\star}} (dashed wall = stop-gradient), so \nabla_{\theta} flows from the policy alone. Existing RL objectives split into two families. Reward maximization methods such as PPO, GRPO, DAPO, and GSPO update the policy from scalar rewards under KL-style regularization ( 32 ; 34 ; 44 ; 48 ) ; they are effective for single-answer accuracy but can collapse diverse valid reasoning paths onto one mode ( 49 ; 19 ; 7 ) . Distribution matching instead fits a target where trajectory probability is proportional to exponentiated reward ( 3 ; 22 ; 43 ; 2 ; 49 ) , preserving multi-modal solution structure. Its bottleneck is the per-prompt partition function: prior methods co-train this partition function with the policy on a shared online loss, so partition-function error flows directly into the policy gradient and has no separable quality signal. The key observation is that this partition function is policy-independent. It is determined by the prompt distribution, reference policy, verifier reward, and inverse temperature, so it can be estimated before policy training begins. We estimate the partition function offline by importance sampling (IS) ( 25 ; 31 ; 37 ; 11 ) , using a fixed proposal that covers reward-bearing trajectories ( 26 ) . This turns the per-prompt partition function into a reusable offline estimate, separated from the policy in data, gradients, loss, and diagnostics. We propose Decoupled Importance-Sampled Anchoring (DISA ), an offline-then-online realization of distribution-matching RL. Stage 1 draws proposal trajectories and forms log-space IS labels for the partition function. Stage 2 fits a lightweight prompt regressor to these labels and freezes it as a partition-function estimate. Stage 3 uses the frozen partition function inside the trajectory-balance objective, with no gradient through the regressor. Appendix C gives the formal guarantees: an exact frozen partition function preserves the reward-tilted optimum, prompt-only partition-function bias preserves it as an on-policy stationary point, and residual partition-function error enters through a bounded loss-perturbation envelope. Our contributions are as follows: • We identify policy–partition-function coupling as the core obstacle in distribution-matching RL for LLM post-training: it injects partition-function error into the policy gradient and leaves the partition function without an independent quality signal. • We propose DISA , a three-stage pipeline that estimates the partition function offline, amortizes it into a frozen prompt-conditioned partition function, and uses that partition function in online trajectory-balance RL. We prove exact-partition-function preservation, biased-partition-function stationarity, and a bounded partition-function-error perturbation in Appendix C . • We validate DISA against GRPO, GSPO, and FlowRL under matched backbones, corpora, and RL infrastructure on six math benchmarks, including AIME 2024/2025, AMC 2023, MATH-500, Minerva, and OlympiadBench, and three code benchmarks, namely LiveCodeBench, CodeForces, and HumanEval+. On Qwen3-4B-Base, DISA matches or exceeds FlowRL on math, with 65.5 vs. 64.8 Mean@8, exceeds GRPO by 16.1 points, and achieves the strongest code pass@16, with 31.4 vs. FlowRL 30.9 , GSPO 29.9 , and GRPO 27.7 . An AIME LLM-as-judge evaluation shows that DISA retains the largest fraction of the backbone’s strategy diversity, with 3.72 on a 1 -to- 5 scale compared with FlowRL 3.40 and GRPO/GSPO \approx 3.0 . 2 Method: Decoupled Importance-Sampled Anchoring (DISA ) Figure 2 : The pipeline of DISA , which includes three stages. Stage 1, offline IS estimation: for each prompt q , draw N trajectories from a proposal p_{\mathrm{T}} and form the per-prompt label \log\widehat{Z}_{\mathrm{IS}}(q) via the logsumexp aggregator. Stage 2, amortization: fit a regressor g_{\psi} to these labels by least squares. Stage 3, anchored RL: freeze g_{\psi^{\star}} and substitute it for \log Z_{\phi} in the trajectory-balance loss, so the partition function enters the policy update as a per-prompt constant with no gradient through \psi . 2.1 Background Notation and setup. Throughout the paper, q\in\mathcal{Q} denotes a prompt and o\in\mathcal{O} a complete output trajectory. A policy \pi(o\mid q) assigns probabilities to trajectories. We are given a reference policy \pi_{\mathrm{ref}} , taken in practice to be the supervised-fine-tuned model, and a verifiable binary reward r:\mathcal{Q}\times\mathcal{O}\to\{0,1\} (math-verify for math, execution checks for code) that scores a trajectory once it is complete. The goal of RL post-training is to update the trainable policy \pi_{\theta}=\pi_{\theta} from \pi_{\mathrm{ref}} using the reward signal. Reward maximization: PPO and GRPO. Let a sampled trajectory be o_{i}=(y_{i,1},\ldots,y_{i,T_{i}}) . PPO ( 32 ) does not use a trajectory-level probability ratio; it clips token-level

Read the complete text