S-GRPO: Unified Post-Training for Large Vision-Language Models DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; 2026;

Yuming Yan, Kai Tang, Sihong Chen, Ke Xu, Dan Hu, Qun Yu, Pengfei Hu · 2026

Read S-GRPO: Unified Post-Training for Large Vision-Language Models DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; 2026; by Yuming Yan, Kai Tang, Sihong Chen, Ke Xu, Dan Hu, Qun Yu, Pengfei Hu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-16557.txt.

Introduction

Yan and colleagues unify post-training for large vision-language models with S-GRPO. Supervised fine-tuning forces one expert path and can cause catastrophic forgetting of general skills. Reinforcement learning explores many paths but can collapse at cold start in sparse-reward visual tasks. S-GRPO blends imitation guidance into multi-trajectory exploration, and injects the verified expert path with maximal reward when a group fails fully. That balance speeds convergence, preserves general skills, and beats single-method training on domain adaptation.

Catalog record

CREATORYuming Yan, Kai Tang, Sihong Chen, Ke Xu, Dan Hu, Qun Yu, Pengfei Hu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE42287
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.16557

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

S-GRPO: Unified Post-Training for Large Vision-Language Models DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; 2026; Yuming Yan, Kai Tang, Sihong Chen, Ke Xu, Dan Hu, Qun Yu, Pengfei Hu Abstract . Current post-training methodologies for adapting Large Vision-Language Models (LVLMs) generally fall into two distinct paradigms: Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL). Despite their prevalence, both approaches suffer from significant inherent inefficiencies when applied in isolation. SFT forces the model’s generation along a single, deterministic expert trajectory, often inducing catastrophic forgetting of general multimodal capabilities due to distributional shifts. Conversely, pure RL explores multiple generated trajectories but frequently encounters optimization collapse—a "cold-start" problem where an unaligned model fails to spontaneously sample any domain-valid trajectories in sparse-reward visual tasks. In this paper, we propose Supervised Group Relative Policy Optimization (S-GRPO), a unified post-training framework that seamlessly integrates the definitive guidance of imitation learning into the multi-trajectory exploration of preference optimization. Tailored for direct-generation visual tasks, S-GRPO introduces Conditional Ground-Truth Trajectory Injection (CGI). When a binary verifier detects a complete exploratory failure within a sampled group of trajectories, CGI dynamically injects the verified ground-truth trajectory into the candidate pool. By assigning a deterministic maximal reward to this injected anchor, S-GRPO enforces a strictly positive signal within the group-relative advantage estimation. This mechanism reformulates the supervised learning objective as a high-advantage component of the policy gradient, compelling the model to dynamically balance between exploiting the expert trajectory and exploring novel visual concepts. Theoretical analysis and empirical results demonstrate that S-GRPO gracefully bridges the gap between SFT and RL, drastically accelerates convergence, and achieves superior domain adaptation while flawlessly preserving the base model’s general-purpose capabilities. Figure 1 . Conceptual comparison of post-training paradigms. Left: SFT limits learning to a single deterministic trajectory. Middle: Standard GRPO explores multiple trajectories but suffers from vanishing gradients ( A\approx 0 ) when all generated samples fail under strict reward models. Right: Our proposed S-GRPO conditionally injects the ground-truth trajectory into the candidate group. This guarantees a positive advantage signal ( A>0 ), effectively resolving the cold-start problem and preventing optimization collapse in a unified single stage. 1. Introduction Large Vision-Language Models (LVLMs) ( 4 ; 39 ; 2 ; 41 ; 53 ; 7 ; 40 ; 16 ; 27 ; 26 ; 31 ; 3 ; 8 ) have demonstrated remarkable general-purpose capabilities by unifying visual perception with natural language understanding. Recently, the focus of alignment research has shifted towards adapting these foundational models to highly specialized domains, such as medical image diagnostics, complex geometric problem-solving, and fine-grained industrial visual inspection. In these scenarios, the model must not only perceive visual tokens but also adhere to rigorous logical constraints and domain-specific rubrics. To achieve this domain adaptation, prevailing post-training techniques typically fall into two distinct categories: Supervised Fine-Tuning (SFT-like methods) ( 29 ; 17 ; 22 ) and Reinforcement Learning (RL-like methods) ( 35 ; 46 ; 49 ; 47 ; 43 ; 11 ; 42 ; 48 ; 44 ) . Although both paradigms are widely adopted, as shown in Figure 1 , each suffers from profound structural limitations resulting from how they manage the autoregressive generation process. SFT ( 29 ) utilizes maximum likelihood estimation (MLE) to force the model along a single, deterministic ground-truth trajectory . While highly efficient for injecting specific domain knowledge, this heavy reliance on strict teacher-forcing introduces a significant distributional shift. This shift often triggers the "Alignment Tax" ( 25 ) —a form of catastrophic forgetting where the model’s broad, general-purpose multimodal competencies are aggressively overwritten by the narrow target distribution. In stark contrast, RL algorithms—such as Group Relative Policy Optimization ( 35 ) (GRPO)—attempt to refine the policy by sampling multiple diverse trajectories and optimizing them against a reward function. However, directly applying RL to an unaligned base model exposes the severe "cold-start" problem in exploration ( 42 ) . In specialized visual domains evaluated by strict binary verifiers, the base model struggles to spontaneously generate highly accurate trajectories. When using group-based RL, if all sampled trajectories in a group are incorrect, the reward variance collapses to zero, leading to vanishing gradients and stalled learning. Consequently, practitioners face a fundamental dilemma: SFT limits optimization to a single trajectory causing catastrophic forgetting, while pure RL explores multiple trajectories but suffers from optimization collapse due to reward sparsity. We argue that the dichotomy between SFT and RL is artificial. Mathematically, SFT can be viewed as an RL process where the singular ground-truth trajectory is exclusively assigned a maximal reward. This perspective suggests that the transition from supervised learning to reinforcement learning should be a continuous spectrum rather than a disjointed sequence of stages. If we could integrate this strong, unambiguous supervision signal directly into the multi-trajectory online exploration phase of RL, we could theoretically synthesize the strengths of both paradigms. By treating the ground-truth trajectory as an active participant in the preference comparison process—an "anchor" with a guaranteed high reward—we can leverage the contrastive nature of policy gradients to seamlessly unify the definitive expert imitation of SFT with the autonomous multi-trajectory self-exploration of RL within a single optimization landscape. Based on this insight, we propose Supervised Group Relative Policy Optimization (S-GRPO), a unified post-training framework tailored for LVLMs. S-GRPO introduces a dynamic mechanism: Conditional Ground-Truth Trajectory Injection (CGI) . During online sampling, if a simple binary verifier detects that the policy has failed to generate any valid trajectories within an exploratory group, S-GRPO conditionally injects the verified ground-truth trajectory into the candidate pool. The relative advantage is then computed against this mixed group. The injected trajectory acts as a high-advantage anchor, guiding the gradient steeply toward the correct multimodal distribution. As the model improves and begins generating correct trajectories autonomously, the CGI mechanism gracefully phases out, allowing training to seamlessly transition from supervised bootstrapping to pure self-exploratory reinforcement learning. Our main contributions are summarized as follows: • Unified Post-Training Framework : We formalize S-GRPO, an algorithm that elegantly integrates the objectives of SFT and preference optimization, eliminating the need to choose between isolated paradigms and mitigating catastrophic forgetting in LVLMs. • Mechanism Design : We propose the Conditional Ground-Truth Trajectory Injection (CGI) strategy, which dynamically solves the exploration cold-start problem in binary-verifier settings by guaranteeing positive learning signals within multi-trajectory updates. • Empirical Performance : We demonstrate that S-GRPO significantly accelerates convergence in sparse-reward visual domains and achieves superior domain-specific performance while preserving broad general capabilities compared to other methods. 2. Related Works 2.1. Large Vision Language Models (LVLMs) Large Vision-Language Models (LVLMs) ( 4 ; 39 ; 2 ; 41 ; 53 ; 7 ; 40 ; 16 ; 27 ; 26 ; 31 ; 3 ; 8 ; 36 ) have driven remarkable advancements in cross-modal intelligence. This evolution unfolded across three key phases: early foundational architectures (e.g., CLIP ( 32 ) , ALBEF ( 21 ) ) bridged the modality gap for zero-shot transfer; subsequent models like LLaVA ( 28 ) elevated general capabilities via visual instruction tuning; and current state-of-the-art models—such as the InternVL series ( 41 ; 53 ; 40 ) , Qwen-VL family ( 4 ; 39 ; 2 ; 31 ; 3 ) ,

Read the complete text