Guidance Contrastive Token Credit Assignment for Discrete Policy Optimization

Shufan Li, Konstantinos Kallidromitis, Akash Gokul, Yuta Kyuragi, Aditya Grover · 2026

Read Guidance Contrastive Token Credit Assignment for Discrete Policy Optimization by Shufan Li, Konstantinos Kallidromitis, Akash Gokul, Yuta Kyuragi, Aditya Grover (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-29198.txt.

Introduction

Shufan Li and colleagues offer Guidance Contrastive Policy Optimization, giving discrete policy learning per-token credit instead of uniform sample rewards. Group methods such as GRPO and DAPO broadcast one advantage across all tokens, missing individual contributions. GCPO contrasts predictions under good and bad prompts, giving token gains proportional to the difference between those predictions. The method highlights relevant regions, such as visual areas matching textual prompts in picture generation and keywords in reasoning traces. Across picture and reasoning benchmarks, GCPO outperforms its GRPO and DAPO norms.

Catalog record

CREATORShufan Li, Konstantinos Kallidromitis, Akash Gokul, Yuta Kyuragi, Aditya Grover
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE38685
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.29198

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Guidance Contrastive Token Credit Assignment for Discrete Policy Optimization Shufan Li, Konstantinos Kallidromitis, Akash Gokul, Yuta Kyuragi, Aditya Grover Abstract Group-advantage-based reinforcement learning methods, such as GRPO and DAPO, have demonstrated strong performance across diverse domains, including mathematical reasoning and text-to-image generation. However, their reliance on sample-level rewards introduces a key limitation as uniform credit assignment across all tokens fails to capture fine-grained, token-level contributions. To address this issue, we propose Guidance Contrastive Policy Optimization (GCPO), a novel algorithm that enables per-token credit assignment by contrasting model predictions under positive and negative prompts. Rather than uniformly broadcasting sample-level advantages, GCPO assigns token-level advantages proportional to the difference between these contrastive predictions, allowing more precise and informative learning signals. Empirically, we find that GCPO emphasizes semantically relevant regions—such as visual areas aligned with textual prompts in text-to-image generation, and critical keywords within reasoning traces for chain-of-thought tasks. Through extensive experiments, GCPO consistently outperforms GRPO and DAPO baselines on both text-to-image generation and chain-of-thought reasoning benchmarks, demonstrating its effectiveness as a general and scalable optimization strategy for discrete policy learning. Code will be available at https://github.com/jacklishufan/gcpo 1 Introduction Reinforcement Learning (RL) has proven to be a highly effective post-training approach for generative models in a wide range of scenarios including math reasoning, coding, and text-to-image synthesis 29 ; 6 ; 24 ; 7 ; 46 ; 50 ; 20 ; 15 ; 51 . Recently, Group Relative Policy Optimization (GRPO) 30 and its variants have been widely adopted to improve models, such as large language models (LLMs), across many domains and problem scales. In contrast to prior works like PPO 29 , which use a learnable value model and generalized advantage estimation (GAE) 28 to obtain per-token supervision signals through the actor-critic framework, a key innovation of GRPO is to replace GAE with a simpler advantage estimator based on group-normalized per-sample rewards. While this design removes the requirement of an extra value network and improves efficiency, scalability, and stability of training, it broadcasts a uniform sample-level rewards to all tokens in a sequence, which potentially neglects important intra-token differences. Intuitively, not all tokens are equally important. In chain-of-thought (CoT) 39 reasoning, some words contribute to the substance of the reasoning traces (e.g., a math calculation) and other words are mere fillers (e.g., punctuation or linking verbs). In text-to-image synthesis, some image regions are directly associated with the entities described in the text prompt while other regions have less association with the prompt. An ideal post-training algorithm should be aware of such differences and weigh tokens differently. Figure 1 : GCPO enables fine-grained token credit assignment via contrastive guidance. GCPO assigns per-token advantages by contrasting the likelihood of each token in a sampled sequence under positive (conditional) and negative (unconditional) prompts. Tokens in semantically critical image regions show high divergence between the two and are amplified, while background regions are down-weighted. The resulting per-token score are used to scale the per-sample advantage in GRPO training. This observation has motivated a growing body of work on token credit assignment, which studies how token-level contributions can be estimated and incorporated into policy optimization objectives. Existing approaches typically construct token-level importance weights or advantages using heuristics derived from quantities such as token confidence, gradient information, entropy, or statistical significance tests 42 ; 37 ; 33 . More broadly, these methods explore different mechanisms for decomposing sequence-level rewards into finer-grained optimization signals for autoregressive generation, while recent works have also extended token credit assignment to multimodal settings by emphasizing visually grounded tokens 12 . In this work, we propose guidance contrastive policy optimization (GCPO), a variant of GRPO that assigns per-token importance weighting by contrasting positive and negative predictions from the same policy. GCPO is largely motivated by classifier-free guidance 11 , a widely adapted inference technique for text-to-image models. CFG augments the model’s original prediction by the differences between its prediction with a positive prompt (i.e. user input) and a negative prompt (e.g. empty string). CFG has been shown to improve image quality and text–image alignment in diffusion-based visual generation models, and has become a standard component in modern text-to-image systems 11 ; 18 ; 26 ; 25 . We observe that in discrete image generation, where an image is represented as a sequence of discrete tokens as opposed to continuous latents, not all tokens are affected by CFG equally. In particular, regions that are directly associated with the prompt will have a larger difference between the predictions given a positive and negative prompt, while regions that do not contain entities specified by the prompt are less sensitive. Based on this observation, we build a token credit assignment scheme that assigns high credit to tokens more sensitive to CFG as opposed to broadcasting the sample-level reward uniformly. Specifically, this credit assignment is computed using the KL divergence between per token predictions with positive prompts and with negative prompts. This allows the learning process to focus on key regions in the sampled image, improving training stability and achieving better downstream performance. While we draw inspiration from the text-to-image domain, we do not constrain GCPO to visual synthesis. Although less commonly studied, classifier-free guidance has been explored in language modeling as an inference-time control mechanism, where it has been shown to improve reasoning 27 . More broadly, related contrastive inference methods in LLMs demonstrate that differences between positive and negative conditioning signals can provide meaningful steering signals for generation 21 ; 22 , suggesting that such signals may also be useful for policy learning. Motivated by this observation, we extend GCPO to multimodal chain-of-thought reasoning in vision-language models (VLMs) by incorporating analogous importance weighting over text tokens. Compared with text-to-image use cases, extending GCPO to VLMs introduces two challenges. First, unlike text-to-image models that naturally support CFG by incorporating unconditional generation tasks via prompt dropping during training, instruction-tuned LLMs always expect a prompt. We explore several options of negative prompts and discovered that simply augmenting the original prompt with the extra instruction “please generate a wrong answer" at the end works well. Second, while existing works demonstrated CFG can work for VLM inference in certain cases, the common inference protocol of VLMs do not employ CFG because it does not reliably improve model’s performance in a predicted manner. Luckily, through empirical observation we find that CFG-inspired token weighting in GCPO still improves the model’s performance even though the rollout and sampling process does not explicit employ CFG as an inference technique. Another challenge common in both image and text generation is finding a good normalization technique for importance weights. Since the unnormalized KL divergence lies in the range of (0,\infty) , naively applying a softmax normalization or min-max normalization will concentrate the probability mass to only a few tokens. Most notably, we observe that the first token in the response commonly exhibits a large KL divergence. A common approach to stabilize these weights is clipping and temperature based scaling. However, we observe that the KL scale is highly dynamic across different tasks, prompts, and sequence lengths, making hyperparameter tuning challenging. To remove the need of task-specific hyperparameter tuning while ensuring a smooth token credit assignment, we employ a rank-based normalization technique that normalizes the per-token KL divergence based

Read the complete text