Beyond Mode Elicitation: Diversity-Preserving Reinforcement Learning via Latent Diffusion Reasoner Haoqiang Kang, Yizhe Zhang, Nikki Lijing Kuang, Yi-An Ma, Lianhui Qin Abstract Recent reinforcement learning (RL) methods improve LLM reasoning by optimizing discrete Chain-of-Thought (CoT) generation; however, exploration in token space often suffers from diversity collapse as policy entropy decreases due to mode elicitation behavior in discrete RL. To mitigate this issue, we propose Latent Diffusion Reasoning with Reinforcement Learning (LaDi-RL), a framework that conducts exploration directly in a continuous latent space, where latent variables encode semantic-level reasoning trajectories. By modeling exploration via guided diffusion, multi-step denoising distributes stochasticity and preserves multiple coexisting solution modes without mutual suppression. Furthermore, by decoupling latent-space exploration from text-space generation, we show that latent diffusion–based optimization is more effective than text-space policy optimization alone, while a complementary text policy provides additional gains when combined with latent exploration. Experiments on code generation and mathematical reasoning benchmarks demonstrate consistent improvements in both pass@ 1 1 and pass@ k k over discrete RL baselines, with absolute pass@ 1 1 gains of +9.4% on code generation and +5.7% on mathematical reasoning, highlighting diffusion-based latent RL as a principled alternative to discrete token-level RL for reasoning. 1 Introduction Figure 1 : Average pass@ k performance over all tested benchmarks. LaDi-RL preserves solution diversity and converts LaDiR’s pass@ k gains into strong pass@ 1 improvements, whereas vanilla GRPO exhibits diversity collapse at large k . Reinforcement learning (RL) has become a dominant paradigm for improving the reasoning ability of large language models (LLMs), particularly through optimizing the generation of Chain-of-Thought (CoT) trajectories ( 33 ; 128 ; 90 ) . However, this process is often plagued by a significant side effect: diversity collapse ( 96 ; 22 ; 129 ; 141 ; 37 ) , which constrains the exploration space and impedes further RL optimization ( 98 ) . This issue is empirically reflected in metrics such as pass@ k ( 37 ; 19 ; 129 ; 18 ; 9 ) , where RL-tuned models underperform the original base model at large K as the vanilla GRPO performance shown in Figure 1 . This indicates the mode elicitation behavior in discrete RL causes the model to collapse onto dominant modes during generation at the expense of diversity ( 129 ) . Recent analyses on LLMs ( 21 ; 16 ) and Diffusion Language Models ( 30 ; 142 ) attribute this collapse to a fundamental entropy mechanism of softmax-based policies: correct tokens are typically assigned high probability by the policy under softmax normalization, increasing their logits, as the reward automatically suppresses all other tokens, driving entropy down—if without explicit regularization. This motivates latent reasoning methods ( 34 ; 51 ; 8 ; 144 ; 139 ; 102 ) that do not rely on softmax over discrete tokens, which may circumvent this issue. In particular, latent diffusion reasoners ( 51 ) exhibit large improvement in diversity and strong gains in pass@ k , as diffusion models ( 41 ) capture the full support of the underlying distribution ( 94 ; 52 ; 95 ) and allow multiple modes to coexist—each mode contributes to the learned score field without suppressing others ( 52 ; 105 ) . This naturally raises the question: can the diversity advantage of diffusion models be leveraged to enhance RL exploration for LLM reasoning? In this paper, we answer the above question in the affirmative and propose Latent Diffusion Reasoning with Reinforcement Learning (LaDi-RL) , a framework that models reasoning as a latent diffusion policy optimized via reinforcement learning, together with a complementary text policy. Each reasoning process is represented by a fixed-length block of continuous latent tokens that encode a semantic reasoning trajectory. A latent diffusion model iteratively denoises these tokens conditioned on the input query, and the resulting latent representation conditions a text policy that generates the final answer. Reinforcement learning is applied directly to the latent diffusion process, with rewards computed from the generated answers and attributed to the corresponding latent trajectories. By decoupling exploration in latent reasoning space from answer text generation, LaDi-RL preserves the diversity benefits of latent diffusion . This design enables structured and flexible exploration for reasoning by shifting reinforcement learning from discrete token space to a continuous latent space. First, our method preserves the base latent diffusion model’s inherent diversity after RL training , as shown in Figure 1 , by distributing stochasticity across multi-step diffusion, mitigating the diversity collapse of discrete RL. Second, exploration in latent space operates at the semantic level , where neighboring latents represent meaningful and flexible variations of entire reasoning trajectories, allowing RL to explore diverse trajectories to high reward regions rather than mere syntactic token perturbations. Lastly, by decoupling exploration in latent and text spaces, we empirically find that latent diffusion–based optimization is more effective than text-space policy optimization alone , while a complementary text policy provides additional gains when combined with latent exploration. We evaluate our method on code generation and mathematical reasoning benchmarks, where LaDi-RL consistently improves both accuracy and diversity compared to discrete-space RL baselines. It achieves absolute pass@ 1 improvements of +9.4% and +5.7% respectively while simultaneously surpassing the base model’s pass@ k ceiling. These results demonstrate that diffusion-based latent exploration offers a principled alternative to discrete RL for LLM reasoning by preserving diversity while improving accuracy. 2 Preliminary 2.1 Group Relative Policy Optimization Group Relative Policy Optimization (GRPO) ( 33 ) is a reinforcement learning algorithm that has been widely adopted for optimizing large generative models, including large language models ( 128 ) and diffusion-based generative models ( 61 ; 121 ) . Formally, given a group of rollout trajectories \{\mathbf{o}_{g}\}_{g=1}^{G} sampled from the current policy \pi_{\theta} , where the trajectories share the same conditioning input (e.g., prompt or context), GRPO computes a normalized advantage for each trajectory directly from scalar rewards: \hat{A}_{g}=\frac{R_{g}-\mathrm{mean}(\{R_{g}\}_{g=1}^{G})}{\mathrm{std}(\{R_{g}\}_{g=1}^{G})} where R_{g} denotes the outcome reward of trajectory \mathbf{o}_{g} . The GRPO objective is defined as: \displaystyle\mathcal{J}_{\mathrm{GRPO}}(\theta)=\mathbb{E}_{\{\mathbf{o}_{g}\}_{g=1}^{G}\sim\pi_{\theta_{\mathrm{old}}}}\Bigg[\frac{1}{G}\sum_{g=1}^{G}\min\!\Big(r_{g}(\theta)\hat{A}_{g}, (1) \displaystyle\mathrm{clip}\big(r_{g}(\theta),1-\epsilon,1+\epsilon\big)\hat{A}_{g}\Big)\Bigg]-\beta\,D_{\mathrm{KL}}\!\left(\pi_{\theta}\,\|\,\pi_{\theta_{\mathrm{ref}}}\right). where r_{g}(\theta)=\frac{\pi_{\theta}(\mathbf{o}_{g})}{\pi_{\theta_{\mathrm{old}}}(\mathbf{o}_{g})} and \pi_{\theta_{\mathrm{ref}}} is a fixed reference policy, \epsilon is the clipping threshold, and \beta controls KL regularization. Figure 2 : Comparison between standard discrete CoT reasoning and latent continuous CoT reasoning. Left: conventional autoregressive generation produces reasoning directly in token space. Right: reasoning is represented as a fixed-size latent block, enabling continuous modeling and decoupling reasoning from surface text generation. GRPO for Language Models. For LLMs, the policy \pi_{\theta}(o_{t}\mid o_{ to indicate the beginning of the latent block, followed by a fixed number of latent tokens Z=\{z_{1},\dots,z_{B}\} , and a token to mark its end. The pretrained LLM then generates the final answer autoregressively conditioned on both the question and the latent block. The latent tokens are produced by compressing a short reference CoT (typically fewer than 1k tokens) using a variational autoencoder (VAE), yielding a fixed-size latent representation (e.g., B=64 ). Unlike the original LaDiR formulation, which applies blockwise diffusion over multiple latent segments, we employ a single sufficiently large latent block, which we find adequate for reconstructing short CoTs while simplifying the model. Training Objective. LaDiR is trained using a joint objective that aligns latent reasoning with downstream text generation. Let Z^{\mathrm{enc}} denote the latent tokens encoded from a reference CoT. We train a conditional latent generator using a flow-matching objective: \mathcal{L}_{\mathrm{FM}}=\mathbb{E}\left[\|v_{\theta}(Z_{t},Q,t)-(Z^{\mathrm{enc}}-\epsilon)\|_{2}^{2}\right] , where Z_{t}=(1-t)\epsilon+tZ^{\mathrm{enc}} . Conditioned on the question Q and latent block Z , the LLM is simultaneously trained to generate the target answer Y using a standard autoregressive cross-entropy loss: \mathcal{L}_{\mathrm{CE}}=-\mathbb{E}_{(Q,Z,Y)}\sum_{t}\log p_{\theta}(y_{t}\mid y_{... ), and (2) the answer text (formal solution with final answer). This structure achieves remarkable compression while maintaining complete mathematical rigor: LaDi-RL: VAE Latent (CoT) — 287 tokens First, use the formula \frac{1}{z}=\frac{\overline{z}}{|z|^{2}} to simplify both terms. Since |z|=4 , we have |z|^{2}=16 . Next, write z=a+bi with a^{2}+b^{2}=16 . Expand both fractions and combine to find the real part. The real part will be a linear combination of a and b : Ra+Sb for some coefficients R and S . To maximize this subject to a^{2}+b^{2}=16 , parameterize using a=4\cos\theta and b=4\sin\theta . The maximum of R\cos\theta+S\sin\theta is \sqrt{R^{2}+S^{2}} by Cauchy-Schwarz. Calculate the coefficients carefully and factor out common terms to simplify the final arithmetic. LaDi-RL: Answer Text — 243 tokens Solution: Using \frac{1}{z}=\frac{\overline{z}}{|z|^{2}}=\frac{\overline{z}}{16} : \frac{75+117i}{z}+\frac{96+144i}{\overline{z}}=\frac{(75+117i)\overline{z}}{16}+\frac{(96+144i)z}{16} Let z=a+bi where a^{2}+b^{2}=16 . Then: \displaystyle(75+117i)(a-bi) \displaystyle=(75a+117b)+i(117a-75b) \displaystyle(96+144i)(a+bi) \displaystyle=(96a-144b)+i(144a+96b) The real part of the sum is: \frac{(75a+117b)+(96a-144b)}{16}=\frac{171a-27b}{16}=\frac{9(19a-3b)}{16} To maximize 19a-3b subject to a^{2}+b^{2}=16 , use a=4\cos\theta , b=4\sin\theta : 19(4\cos\theta)-3(4\sin\theta)=4(19\cos\theta-3\sin\theta) The maximum is 4\sqrt{19^{2}+3^{2}}=4\sqrt{370} . Therefore, the maximum real part is: \frac{9\cdot 4\sqrt{370}}{16}=\frac{9\sqrt{370}}{4} Wait—let me recalculate. Actually, by Cauchy-Schwarz, 81\cos\theta-108\sin\theta\leq\sqrt{81^{2}+108^{2}}=\sqrt{6561+11664}=\sqrt{18225}=135 . Thus the maximum is \frac{4\cdot 135}{4}=\boxed{540} . The LaDi-RL approach achieves a 83.7% token reduction compared to vanilla GRPO (530 total tokens vs. 3247 tokens) while discovering the correct answer. The VAE latent provides high-level strategic guidance, while the answer text executes the computation with appropriate detail. E.2 Code Generation: MBPP Diversity Analysis We examine code generation capabilities using Problem 163 from the Mostly Basic Programming Problems (MBPP) ( 4 ) benchmark. We focus on solution diversity—a critical metric for exploring the solution space and enabling robust policy learning. MBPP Problem 163 Write a function to calculate the sum of the positive integers of n+(n-2)+(n-4)+\dots (until n-x\leq 0 ). Function signature: def sum_series(n: int) -> int Test cases: • assert sum_series(6) == 12 • assert sum_series(10) == 30 • assert sum_series(9) == 25 E.2.1 Vanilla GRPO: Limited Diversity Vanilla GRPO with long chain-of-thought reasoning (avg. 2847 tokens per sample) tends to converge toward similar solution structures across multiple samples. We present three representative solutions from a sample of 10 generations: Vanilla GRPO 1 (Pattern frequency: 7/10 samples) Chain-of-Thought (truncated): Okay, so I need to write a function that calculates the sum n+(n-2)+(n-4)+\ldots until the term becomes \leq 0 . Let me think about this step by step. First, I should understand the pattern. We’re starting with n and subtracting 2 each time. So if n=6 , the sequence is 6,4,2 , and we stop because 2-2=0 . The sum is 6+4+2=12 . Wait, the problem says ”until n-x\leq 0 ”. So we continue as long as the term is positive. Let me verify with the test cases. For n=6 : 6+4+2=12 . Yes, that matches. For n=10 : 10+8+6+4+2=30 . That works. For n=9 : 9+7+5+3+1=25 . Also correct. So the pattern is clear. I’ll use a while loop to keep adding terms as long as they’re positive… [continues for 2700+ more tokens discussing edge cases, alternative implementations, etc.] Final Code: ⬇ 1 def sum_series ( n ): 2 total = 0 3 current = n 4 while current > 0: 5 total += current 6 current -= 2 7 return total Vanilla GRPO 2 (Pattern frequency: 2/10 samples) Chain-of-Thought (truncated): Let me approach this mathematically. The series n+(n-2)+(n-4)+\ldots is an arithmetic sequence with first term a=n and common difference d=-2 . We need to find how many terms there are. If n is even, the last term is 2. If n is odd, the last term is 1. So the number of terms is \lceil n/2\rceil . For the sum of an arithmetic series: S=\frac{k(a+l)}{2} where k is the number of terms and l is the last term… [continues for 2400+ more tokens deriving the closed form] Final Code: ⬇ 1 def sum_series ( n ): 2 if n <= 0: 3 return 0 4 num_terms = ( n + 1) // 2 5 if n % 2 == 0: 6 last_term = 2 7 else : 8 last_term = 1 9 return num_terms * ( n + last_term ) // 2 Vanilla GRPO 3 (Pattern frequency: 1/10 samples) Chain-of-Thought (truncated): I could use a functional programming approach with Python’s built-in functions. Let me generate the sequence first, then sum it using sum() . The sequence can be generated using range(n, 0, -2) . This starts at n and decrements by 2 each time, stopping before reaching 0 or going negative… [continues for 2600+ more tokens discussing Python idioms] Final Code: ⬇ 1 def sum_series ( n ): 2 return sum ( range ( n , 0, -2)) E.2.2 LaDi-RL: Enhanced Diversity LaDi-RL demonstrates significantly improved solution diversity through its VAE-based sampling mechanism. The VAE latent CoT (avg. 297 tokens) provides strategic guidance, while the answer text contains the actual code implementation. The latent space allows exploration of structurally distinct algorithmic approaches: LaDi-RL Solution 1: VAE Latent (289 tokens) To solve this problem, observe that we’re summing an arithmetic sequence: n,n-2,n-4,\ldots stopping when the term becomes \leq 0 . Use a simple iterative approach with a while loop. Initialize a running total and current value. In each iteration, add the current value to the total and decrement by 2. Continue while the current value is positive. This approach is straightforward and easy to verify for correctness. LaDi-RL Solution 1: Code (47 tokens) ⬇ 1 def sum_series ( n ): 2 total = 0 3 while n > 0: 4 total += n 5 n -= 2 6 return total LaDi-RL Solution 2: VAE Latent (312 tokens) For an arithmetic series with first term a=n and common difference d=-2 , we can derive a closed-form solution. Determine the number of terms: if n is even, we have n/2 terms; if n is odd, we have (n+1)/2 terms. The sum formula for an arithmetic series is S=\frac{k(a+l)}{2} where k is the number of terms, a is the first term, and l is the last term. For even n , last term is 2; for odd n , last term is 1. Substitute into the formula to get the closed-form solution. LaDi-RL Solution 2: Code (93 tokens) ⬇ 1 def sum_series ( n ): 2 if n <= 0: 3 return 0 4 num_terms = ( n + 2) // 2 5 last_term = 1 if n % 2 == 1 else 2 6 return num_terms * ( n + last_term ) // 2 LaDi-RL Solution 3: VAE Latent (268 tokens) Python’s range function can generate the sequence directly. Use range(n, 0, -2) to produce n,n-2,n-4,\ldots,2 or 1 . Then apply the built-in sum function to compute the total. This is the most concise and Pythonic solution. LaDi-RL Solution 3: Code (31 tokens) ⬇ 1 def sum_series ( n ): 2 return sum ( range ( n , 0, -2)) LaDi-RL Solution 4: VAE Latent (325 tokens) Use a recursive approach. The base case is when n\leq 0 , which returns 0. The recursive case adds n to the sum of the series starting at n-2 . This demonstrates a functional programming style and is elegant, though potentially less efficient for large n due to recursion depth. LaDi-RL Solution 4: Code (52 tokens) ⬇ 1 def sum_series ( n ): 2 if n <= 0: 3 return 0 4 return n + sum_series ( n - 2) LaDi-RL Solution 5: VAE Latent (341 tokens) Recognize that this is a sum of either all even numbers up to n (if n is even) or all odd numbers up to n (if n is odd). For even n : sum = 2+4+\ldots+n=2(1+2+\ldots+n/2)=2\cdot\frac{(n/2)(n/2+1)}{2}=\frac{n(n+2)}{4} For odd n : sum = 1+3+\ldots+n=((n+1)/2)^{2} Use these closed formulas directly for O(1) time complexity. LaDi-RL Solution 5: Code (78 tokens) ⬇ 1 def sum_series ( n ): 2 if n <= 0: 3 return 0 4 if n % 2 == 0: 5 return n * ( n + 2) // 4 6 else : 7 return (( n + 1) // 2) ** 2 Qualitative Analysis: The MBPP-163 problem demonstrates LaDi-RL’s efficiency advantages on moderately complex tasks. Vanilla GRPO generates extremely verbose reasoning (2654-2931 tokens on average) that exhaustively explores implementation alternatives, verifies test cases multiple times, and discusses micro-optimizations at length. While this thoroughness occasionally yields insights, it primarily introduces redundancy. LaDi-RL’s VAE latent (268-341 tokens) distills the essential strategic choices—iterative vs. closed-form vs. functional approaches—without the exploratory overhead. The five generated solutions span the full spectrum of reasonable implementations: simple iteration, mathematical closed-form, Pythonic built-ins, recursion, and parity-based optimization. This diversity emerges naturally from the latent sampling process rather than through exhaustive deliberation, achieving an 87.7% token reduction while maintaining high pass@ 1 0 rates. E.3 Challenging Problem: MBPP-427 To further demonstrate LaDi-RL’s advantages on more complex tasks, we examine MBPP Problem 427 ( 4 ) , which requires multi-step algorithmic reasoning: MBPP Problem 427 Write a function to find the largest palindromic number in a given array of positive integers. Function signature: def find_max_palindrome(nums: list) -> int Test cases: • assert find_max_palindrome([1, 232, 5545, 909, 15]) == 5545 • assert find_max_palindrome([1, 2, 3, 4, 5]) == 5 • assert find_max_palindrome([11, 232, 1221, 333]) == 1221 Vanilla GRPO (3124 tokens) Chain-of-Thought (truncated): Alright, I need to find the largest palindromic number in an array. Let me break this down step by step. First, what is a palindrome? It’s a number that reads the same forwards and backwards. So 121, 1221, 5545 are all palindromes. Wait, is 5545 a palindrome? Let me check: 5-5-4-5. Reversed: 5-4-5-5. Yes, that’s the same! Now, the task is to find the LARGEST palindromic number. So I need to: 1. Check each number to see if it’s a palindrome 2. Keep track of the maximum palindrome found How do I check if a number is a palindrome? I could convert it to a string and check if the string equals its reverse. In Python, that would be: str(n) == str(n)[::-1] . Alternatively, I could reverse the number mathematically by extracting digits and rebuilding it. Let me think about which approach is better… The string approach is simpler and more readable. The mathematical approach might be slightly faster for very large numbers, but for this problem, clarity is probably more important than micro-optimizations. Let me verify with the test cases: - For [1, 232, 5545, 909, 15], I need to check each: - 1: ”1” reversed is ”1” correct palindrome - 232: ”232” reversed is ”232” correct palindrome - 5545: ”5545” reversed is ”5445”… wait no, ”5545” reversed is ”5455”… hmm, let me be more careful. The string ”5545” has characters [’5’,’5’,’4’,’5’]. Reversed: [’5’,’4’,’5’,’5’], which gives ”5455”. That’s not equal to ”5545”, so 5545 is NOT a palindrome! But the test case says the answer should be 5545. Let me re-read the problem… Oh wait, maybe I misread. Let me check the test case again… [continues for 2900+ more tokens exploring edge cases, reconsidering the problem, implementing helper functions, etc.] Final Code: ⬇ 1 def find_max_palindrome ( nums ): 2 def is_palindrome ( n ): 3 s = str ( n ) 4 return s == s [::-1] 5 6 max_pal = -1 7 for num in nums : 8 if is_palindrome ( num ): 9 if num > max_pal : 10 max_pal = num 11 12 return max_pal if max_pal != -1 else None LaDi-RL Solution 1: VAE Latent (294 tokens) To find the largest palindromic number, we need two components: (1) a helper function to check if a number is a palindrome, and (2) a main function to find the maximum among palindromic numbers. For checking palindromes, convert the number to a string and compare it with its reverse. This is simple and handles all edge cases naturally. For finding the maximum, filter the input array to keep only palindromes, then return the maximum. Handle the case where no palindromes exist by returning an appropriate value. LaDi-RL Solution 1: Code (81 tokens) ⬇ 1 def find_max_palindrome ( nums ): 2 def is_palindrome ( n ): 3 s = str ( n ) 4 return s == s [::-1] 5 6 palindromes = [ n for n in nums if is_palindrome ( n )] 7 return max ( palindromes ) if palindromes else -1 LaDi-RL Solution 2: VAE Latent (318 tokens) Use a single-pass algorithm with early optimization. Instead of collecting all palindromes and then finding the max, track the maximum palindrome as we iterate through the array. Check palindromes using string reversal for clarity. Initialize the maximum to negative infinity or None to handle empty results. This approach has O(n) time complexity and O(1) space complexity (excluding the space for string conversion during palindrome checking). LaDi-RL Solution 2: Code (98 tokens) ⬇ 1 def find_max_palindrome ( nums ): 2 max_palindrome = float ( ’-inf’ ) 3 4 for num in nums : 5 if str ( num ) == str ( num )[::-1]: 6 max_palindrome = max ( max_palindrome , num ) 7 8 return max_palindrome if max_palindrome != float ( ’-inf’ ) else -1 LaDi-RL Solution 3: VAE Latent (305 tokens) Implement a mathematical approach to check palindromes without string conversion. Reverse the number by extracting digits using modulo and division operations. While this is more complex, it demonstrates algorithmic thinking and avoids string overhead. Compare the original number with its mathematically reversed version. Combine this with a max-tracking approach similar to Solution 2 for efficiency. LaDi-RL Solution 3: Code (147 tokens) ⬇ 1 def find_max_palindrome ( nums ): 2 def is_palindrome ( n ): 3 if n < 0: 4 return False 5 original = n 6 reversed_num = 0 7 while n > 0: 8 reversed_num = reversed_num * 10 + n % 10 9 n //= 10 10 return original == reversed_num 11 12 result = -1 13 for num in nums : 14 if is_palindrome ( num ) and num > result : 15 result = num 16 return result LaDi-RL Solution 4: VAE Latent (281 tokens) Use Python’s functional programming features. Apply filter() to extract palindromic numbers, then max() to find the largest. This declarative style is concise and leverages Python’s built-in functions for readability. Use a lambda function for the palindrome check to keep everything compact. Handle the empty case by providing a default value to max() . LaDi-RL Solution 4: Code (74 tokens) ⬇ 1 def find_max_palindrome ( nums ): 2 is_pal = lambda n : str ( n ) == str ( n )[::-1] 3 palindromes = filter ( is_pal , nums ) 4 return max ( palindromes , default =-1) Qualitative Analysis: For this challenging problem requiring both palindrome detection and maximum-finding logic, vanilla GRPO exhibits significant inefficiency, producing a 3124-token chain-of-thought that includes false starts, misinterpretations of test cases, and extensive deliberation over implementation choices. The exploratory nature, while thorough, leads to substantial cognitive overhead. In contrast, LaDi-RL’s VAE latent provides focused strategic guidance (281-318 tokens) that decomposes the problem into clear subcomponents: palindrome checking and maximum selection. The four diverse solutions demonstrate different algorithmic perspectives—list comprehension with filtering, single-pass tracking, mathematical digit reversal, and functional programming—each with distinct space-time tradeoffs. This structural diversity, combined with the dramatic token efficiency (375-465 total tokens vs. 3124 tokens, an 85-87% reduction), demonstrates LaDi-RL’s ability to maintain solution quality while exploring the algorithmic design space more effectively than vanilla GRPO’s verbose exploration. E.4 LiveCodeBench V6 To evaluate performance on competitive programming challenges, we examine a medium-difficulty dynamic programming problem from LiveCodeBench V6 ( 48 ) , representing the type of algorithmic reasoning required in technical interviews and programming contests: LiveCodeBench V6—Dynamic Programming) You are given an array of positive integers nums and an integer k . In one operation, you can choose any element and increase or decrease it by 1. Return the minimum number of operations needed to make all elements equal, where you can perform at most k operations in total. Function signature: def min_operations(nums: List[int], k: int) -> int Constraints: • 1\leq len(nums) \leq 10^{5} • 1\leq nums[i] \leq 10^{9} • 0\leq k\leq 10^{9} Example test cases: • min_operations([1, 2, 3], 5) == 2 (make all 2) • min_operations([1, 10, 100], 50) == -1 (impossible) • min_operations([5, 5, 5], 0) == 0 (already equal) Vanilla GRPO (4287 tokens) Chain-of-Thought (truncated): Okay, this is an interesting problem. Let me think step by step about what we’re trying to optimize here. We want all elements to be equal, and we can increase or decrease each element by 1 in each operation. The constraint is that we have at most k operations total. We need to find the minimum number of operations if it’s possible, or return -1 if impossible. First observation: if we’re going to make all elements equal, they should converge to some target value t . The question is: what should t be? If we choose target t , then the total number of operations needed is: \text{ops}(t)=\sum_{i=1}^{n}|nums[i]-t| We want to minimize this, subject to the constraint that ops(t)\leq k . Now, which value of t minimizes the sum of absolute deviations? This is a classic problem in statistics—the median! The median minimizes the sum of absolute deviations from a set of points. Wait, but there’s a constraint. We need ops(t)\leq k . So we can’t just use the median blindly. Let me think more carefully… Actually, I think the approach should be: 1. Try all possible target values t 2. For each t , calculate ops(t)=\sum|nums[i]-t| 3. Check if ops(t)\leq k 4. Among all valid targets, find the one with minimum ops(t) But wait, trying all possible values of t is inefficient. The range of nums could be up to 10^{9} , so that’s way too many values to try. Let me reconsider. Actually, the optimal target must be one of the values in nums or close to them. Why? Because the cost function ops(t) is piecewise linear in t , with breakpoints at each element of nums . The minimum will occur at one of these breakpoints. Actually, thinking more carefully: the sum of absolute deviations \sum|nums[i]-t| is minimized at the median. But here we have a constraint k . So the problem is: Minimize \sum|nums[i]-t| subject to \sum|nums[i]-t|\leq k . Wait, that doesn’t make sense. If we’re minimizing the same thing we’re constraining, the answer is just the minimum value if it’s \leq k , otherwise -1. Let me re-read the problem… ”Return the minimum number of operations needed to make all elements equal, where you can perform at most k operations in total.” Oh! So we want to find the minimum number of operations to make all elements equal, and if that minimum exceeds k , return -1. The constraint k is an upper bound, not something we optimize against. So the problem simplifies to: 1. Find the target value t that minimizes \sum|nums[i]-t| 2. Calculate \text{min\_ops}=\sum|nums[i]-t| 3. If \text{min\_ops}\leq k , return min_ops ; otherwise return -1. [continues for 3900+ more tokens discussing edge cases, optimizations, alternative median-finding algorithms, etc.] Final Code: ⬇ 1 from typing import List 2 3 def min_operations ( nums : List [ int ], k : int ) -> int : 4 if len ( nums ) == 1: 5 return 0 6 7 # Sort to find median 8 sorted_nums = sorted ( nums ) 9 n = len ( sorted_nums ) 10 11 # Median is the value that minimizes sum of absolute deviations 12 median = sorted_nums [ n // 2] 13 14 # Calculate total operations needed 15 total_ops = sum ( abs ( num - median ) for num in nums ) 16 17 # Check if possible within k operations 18 if total_ops <= k : 19 return total_ops 20 else : 21 return -1 LaDi-RL Solution 1: VAE Latent (327 tokens) The key insight is that to minimize the total operations to make all elements equal, we should choose the target value that minimizes the sum of absolute deviations. This target is the median of the array. Steps: 1. Sort the array to find the median 2. For an array of length n , the median is at index n//2 after sorting 3. Calculate the sum of absolute differences between each element and the median 4. If this sum is \leq k , return it; otherwise return -1 Time complexity: O(n\log n) for sorting. Space complexity: O(1) if sorting in-place. LaDi-RL Solution 1: Code (98 tokens) ⬇ 1 def min_operations ( nums : List [ int ], k : int ) -> int : 2 if len ( nums ) == 1: 3 return 0 4 5 median = sorted ( nums )[ len ( nums ) // 2] 6 total_ops = sum ( abs ( x - median ) for x in nums ) 7 8 return total_ops if total_ops <= k else -1 LaDi-RL Solution 2: VAE Latent (294 tokens) Alternative approach: use quickselect to find the median in O(n) average time instead of O(n\log n) sorting. The median-finding problem can be solved more efficiently using the quickselect algorithm, which has expected linear time complexity. After finding the median, compute the sum of deviations and compare against k . This optimization matters for large arrays where sorting overhead becomes significant. LaDi-RL Solution 2: Code (142 tokens) ⬇ 1 import random 2 3 def min_operations ( nums : List [ int ], k : int ) -> int : 4 def quickselect ( arr , k ): 5 pivot = random . choice ( arr ) 6 left = [ x for x in arr if x < pivot ] 7 mid = [ x for x in arr if x == pivot ] 8 right = [ x for x in arr if x > pivot ] 9 10 if k < len ( left ): 11 return quickselect ( left , k ) 12 elif k < len ( left ) + len ( mid ): 13 return mid [0] 14 else : 15 return quickselect ( right , k - len ( left ) - len ( mid )) 16 17 median = quickselect ( nums , len ( nums ) // 2) 18 total_ops = sum ( abs ( x - median ) for x in nums ) 19 return total_ops if total_ops <= k else -1 LaDi-RL Solution 3: VAE Latent (311 tokens) Consider an edge case optimization: if all elements are already equal, return 0 immediately without further computation. Use Python’s statistics.median for clean, readable code. While it internally sorts, it’s a standard library function that’s well-optimized and clearly expresses intent. This solution prioritizes code clarity over micro-optimizations, which is often appropriate for interview settings where correctness and readability matter most. LaDi-RL Solution 3: Code (87 tokens) ⬇ 1 from statistics import median 2 3 def min_operations ( nums : List [ int ], k : int ) -> int : 4 if len ( set ( nums )) == 1: 5 return 0 6 7 target = int ( median ( nums )) 8 total_ops = sum ( abs ( x - target ) for x in nums ) 9 return total_ops if total_ops <= k else -1 Qualitative Analysis: This competitive programming problem from LiveCodeBench V6 demonstrates the limitations of vanilla GRPO on algorithmic challenges. The 4287-token chain-of-thought reveals extensive deliberation about the median property, verification through examples, discussion of even vs. odd array lengths, and exploration of various median-finding algorithms. While this thoroughness eventually leads to the correct insight, it represents substantial computational waste. LaDi-RL’s VAE latents (294-327 tokens) directly encode the critical insight—that the median minimizes sum of absolute deviations—without the meandering exploration. The three solutions demonstrate algorithmic diversity (simple sorting, quickselect optimization, and standard library usage) while achieving 92-93% token reduction (425-469 total tokens vs. 4287 tokens). For competitive programming contexts where rapid prototyping and algorithmic intuition are essential, LaDi-RL’s ability to distill strategic insights into compact latent representations provides a significant advantage over vanilla GRPO’s verbose deliberative process.