AIS: Adaptive Importance Sampling for Quantized RL

Jiajun Zhou, Wei Shao, Lingchao Zheng, Yuwei Fan, Ngai Wong · 2026

Read AIS: Adaptive Importance Sampling for Quantized RL by Jiajun Zhou, Wei Shao, Lingchao Zheng, Yuwei Fan, Ngai Wong (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-13907.txt.

Introduction

Zhou and four colleagues tackle a mismatch in quantized reinforcement learning for language models, where low-precision rollouts feed a higher-precision trainer. The mismatch biases the policy gradient and can collapse training on reasoning tasks, though early on it also supplies a useful exploration bonus. Their Adaptive Importance Sampling method corrects the bias, tuning the strength of its intervention batch by batch as training proceeds.

Catalog record

CREATORJiajun Zhou, Wei Shao, Lingchao Zheng, Yuwei Fan, Ngai Wong
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE58256
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.13907

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

AIS: Adaptive Importance Sampling for Quantized RL Jiajun Zhou, Wei Shao, Lingchao Zheng, Yuwei Fan, Ngai Wong Abstract Reinforcement learning (RL) for large language models (LLMs) is dominated by the cost of rollout generation, which has motivated the use of low-precision rollouts (e.g., FP8) paired with a BF16 trainer to improve throughput and reduce memory pressure. This introduces a rollout–training mismatch that biases the policy gradient and can cause training to collapse outright on reasoning benchmarks. We show that the mismatch is non-stationary and acts as a double-edged sword: early in training it provides a stochastic exploration bonus, exposing the gradient to trajectories the trainer would otherwise under-sample, but the same perturbation transitions into a destabilizing source of bias as the policy concentrates. To solve this, we propose Adaptive Importance Sampling (AIS) , a correction framework that adjusts the strength of its intervention on a per-batch basis. AIS combines three real-time diagnostics, weight reliability , divergence severity , and variance amplification , into a single mixing coefficient that interpolates between the uncorrected and fully importance-weighted gradients, suppressing the destabilizing component of the mismatch while preserving its exploratory benefit. We integrate AIS into GRPO and evaluate it on the diffusion-based LLaDA-8B-Instruct and the autoregressive Qwen3-8B and Qwen3.5-9B across mathematical reasoning and planning benchmarks. AIS matches the BF16 baseline on most tasks while retaining the 1.5 1.5 to 2.76 × 2.76\times rollout speedup of FP8. 1 Introduction Reinforcement learning (RL) has become a dominant post-training paradigm for large language models (LLMs), driving recent advances in reasoning ( Yue et al. 2025 ) , code generation ( Luo et al. 2025a ; Liu and Zhang 2025 ) , and mathematics ( Luo et al. 2025b ; Yu et al. 2025 ) through optimization against verifiable reward signals ( Guo et al. 2025 ; Team et al. 2025 ) . A single training step interleaves three stages: trajectory rollout, log-probability evaluation, and a policy gradient update. Among these, the rollout stage dominates both wall-clock time and memory, accounting for up to 70\% of end-to-end latency ( Zheng et al. 2024 ; He et al. 2025 ; Zheng et al. 2025 ) , with the imbalance further amplified on reasoning tasks that demand long chain-of-thought traces. To accelerate this bottleneck, the recent large-scale industry has increasingly adopted low-precision quantization at the rollout stage. DeepSeek V3.2 integrates FP8 attention into its post-training pipeline ( DeepSeek-AI 2025 ) , and DeepSeek V4 pushes rollout precision further down to FP4 ( DeepSeek-AI 2026 ) . Open-source frameworks such as veRL ( Sheng et al. 2024 ) provide production-level FP8 rollout pipelines for GRPO training on Qwen3-class models ( NVIDIA 2026 ; Qiu et al. 2026 ) . Across these reports, FP8 rollout consistently delivers 1.2 to 1.5\times speedup and roughly halves rollout memory, both directly through faster low-precision kernels and indirectly through the larger batches a smaller memory footprint allows. Standard RL algorithms assume trajectories are sampled from the current policy being optimized. When a quantized model (FP8/INT8) generates the rollout while the trainer stays at higher precision (BF16), the resulting precision gap introduces a rollout-training mismatch ( Qiu et al. 2026 ) that biases every gradient estimate. Because the policy gradient sums over token-level log-probability terms, small per-token discrepancies compound across long generation horizons, and on some reasoning benchmarks, the resulting bias is severe enough to collapse training accuracy outright: FP8 rollout drops AIME25 accuracy on Qwen3-8B by 6.63\% . Prior research has sought to bypass the mismatch problem by enforcing uniform precision across both rollout and training ( SGLang RL Team 2025 ) . While this avoids distributional shift, it sacrifices the numerical integrity of the trainer. We instead embrace the common production paradigm of BF16 training with FP8 rollout, treating mismatch as an unavoidable yet manageable reality. Standard Importance Sampling (IS) ( Hanna et al. 2019 ; Yao et al. 2025b ) remains the primary defense for bias reduction, yet it often falls short in variance control, resulting in erratic training trajectories. Figure 1: Reward trajectory of Qwen3-8B on GSM8K under FP8 rollout. Crucially, we challenge the conventional wisdom that frames this mismatch solely as an adversary. As demonstrated in Figure 1 , quantization mismatch operates as a double-edged sword: early in training, it serves as a stochastically driven exploration bonus, exposing the gradient to a broader behavioral set and preventing early stagnation. Yet, as training converges, the same noise devolves into a source of instability. This dual nature suggests that the challenge is not to eradicate the mismatch, but to govern it. The appropriate response is therefore not to eliminate the mismatch but to manage it, suppressing its destabilizing component while preserving its useful one, a balance that no fixed-threshold rule can sustain throughout training. We propose Adaptive Importance Sampling (AIS), which targets exactly this balance. At each gradient step, AIS computes three real-time diagnostics from quantities the trainer already evaluates— weight reliability , divergence severity , and variance amplification —and combines them into a single mixing coefficient that interpolates between the uncorrected gradient and a fully importance-weighted one. This coefficient tightens correction when the mismatch becomes destabilizing and relaxes it when the mismatch is benign, suppressing harmful variance while preserving the exploratory signal that quantized rollouts can provide. The mechanism reduces to standard training as the mismatch reduces, and adds negligible overhead because all three diagnostics are scalar reductions over statistics already computed during the policy update. We integrate AIS into GRPO ( Shao et al. 2024 ) and evaluate on LLaDA-8B-Instruct ( Nie et al. 2025 ) , Qwen3-8B, and Qwen3.5-9B ( Yang et al. 2025 ) across mathematical reasoning, planning, and general benchmarks. To our knowledge, this is the first study of low-precision rollout for RL on diffusion LLMs. AIS recovers most of the accuracy gap to BF16 across reasoning benchmarks for both architectures, substantially outperforming truncation sampling, while matching or improving over the BF16 reference on most tasks and retaining the 1.5 to 2.76\times rollout speedup and roughly 50\% memory reduction of FP8. 2 Preliminaries 2.1 LLM Quantization Quantization reduces the numerical precision of model parameters and activations to enable faster inference and lower memory consumption. Given a full-precision tensor \mathbf{X}\in\mathbb{R}^{n} , quantization maps each element to a discrete set of representable values. For symmetric linear quantization to b bits, the quantized representation is: Q(\mathbf{X})=\text{clamp}\!\left(\left\lfloor\frac{\mathbf{X}}{s}\right\rceil,\;-2^{b-1},\;2^{b-1}-1\right)\cdot s,\quad s=\frac{\max(|\mathbf{X}|)}{2^{b-1}-1}, (1) where s is the scaling factor and \lfloor\cdot\rceil denotes rounding to the nearest integer. The quantization error \boldsymbol{\epsilon}=Q(\mathbf{X})-\mathbf{X} introduces perturbations that propagate through the network’s forward pass. In this work, we focus on FP8 (E4M3) quantization ( Micikevicius et al. 2022 ) , which represents values using 4 exponent bits and 3 mantissa bits. Unlike integer quantization, FP8 provides a wider dynamic range at the cost of reduced precision in the mantissa, making it particularly suitable for the diverse value distributions encountered in transformer activations. Modern accelerators (e.g., NVIDIA H100) provide native FP8 support, enabling up to 2\times throughput improvement over BF16 for matrix multiplications. To accelerate the rollout phase, we quantize both the weights and activations of the rollout model \theta_{\mathrm{old}} to obtain a low-bit counterpart \hat{\theta}_{\mathrm{old}}=Q(\theta_{\mathrm{old}}) . Because quantization affects the entire forward pass, the logits computed by the quantized

Read the complete text