Hölder Policy Optimisation
Yuxiang Chen Dingli Liang Yihang Chen Ziqin Gong Chenyang Le Zhaokai Wang, [2pt] Jiachen Zhu Lingyu Yang Jianghao Lin Weinan Zhang Jun Wang · 2026
Read Hölder Policy Optimisation by Yuxiang Chen Dingli Liang Yihang Chen Ziqin Gong Chenyang Le Zhaokai Wang, [2pt] Jiachen Zhu Lingyu Yang Jianghao Lin Weinan Zhang Jun Wang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-12058.txt.
Introduction
Yuxiang Chen and ten colleagues widen group relative policy optimization with the Hölder mean. GRPO scores gains across groups of sampled paths, but fixed ways of pooling token odds limit it: some pooling modes crash training, others give weak results. HölderPO pools token odds by the Hölder mean, where a bigger p sharpens gradients on rare signals and a smaller p caps gradient spread. A slow annealing plan for p through training beats the trade-off, reaching 54.9 percent mean accuracy on math tests and 93.8 percent on ALFWorld.
Catalog record
| CREATOR | Yuxiang Chen Dingli Liang Yihang Chen Ziqin Gong Chenyang Le Zhaokai Wang, [2pt] Jiachen Zhu Lingyu Yang Jianghao Lin Weinan Zhang Jun Wang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 106058 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.12058
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Hölder Policy Optimisation Yuxiang Chen Dingli Liang Yihang Chen Ziqin Gong Chenyang Le Zhaokai Wang, [2pt] Jiachen Zhu Lingyu Yang Jianghao Lin Weinan Zhang Jun Wang Abstract Group Relative Policy Optimisation (GRPO) enhances large language models by estimating advantages across a group of sampled trajectories. However, mapping these trajectory-level advantages to policy updates requires aggregating token-level probabilities within each sequence. Relying on a fixed aggregation mechanism for this step fundamentally limits the algorithm’s adaptability. Empirically, we observe a critical trade-off: certain fixed aggregations frequently suffer from training collapse, while others fail to yield satisfactory performance. To resolve this, we propose HölderPO , a generalised policy optimisation framework unifying token-level probability aggregation via the Hölder mean. By explicitly modulating the parameter p p , our framework provides continuous control over the trade-off between gradient concentration and variance bounds. Theoretically, we prove that a larger p p concentrates the gradient to amplify sparse learning signals, whereas a smaller p p strictly bounds gradient variance. Because no static configuration can universally resolve this concentration-stability trade-off, we instantiate the framework with a dynamic annealing algorithm that progressively schedules p p across the training lifecycle. Extensive evaluations demonstrate superior stability and convergence over existing baselines. Specifically, our approach achieves a state-of-the-art average accuracy of 54.9 % 54.9\% across multiple mathematical benchmarks, yielding a substantial 7.2 % 7.2\% relative gain over standard GRPO and secures an exceptional 93.8 % 93.8\% success rate on ALFWorld. Figure 1: HölderPO unifies token-level aggregation under a single parameter p . The objective at the top generalises GRPO by replacing its arithmetic mean over token-level importance ratios with the Hölder mean of order p\in\mathbb{R} , recovering GRPO ( p=1 ) and GMPO/GSPO ( p\to 0 ) as special cases. The bar chart reports accuracy on AIME24 (blue, sparse signal) and MATH500 (red, dense signal), with dashed lines marking GRPO baselines. Bottom: the token weight distribution W(p) , with each panel ordering tokens from small (left) to large (right) importance ratio. 1 Introduction Reinforcement Learning (RL) has emerged as a key technique for advancing the alignment and complex reasoning capabilities of Large Language Models (LLMs) ( 23 ; 28 ) . Recently, Group Relative Policy Optimisation (GRPO) has emerged as a highly effective and compute-efficient algorithm, largely driving the success of reasoning models like DeepSeek-R1 ( 30 ) . GRPO operates by estimating advantages across a group of sampled trajectories, substantially reducing training overhead by eliminating the need for an external critic model. However, mapping these trajectory-level advantages to policy updates requires aggregating token-level probabilities within each sequence. As the demand for solving long-horizon reasoning tasks grows, the fundamental mechanics of this fixed aggregation step have come under scrutiny ( 22 ) . Existing algorithms rigidly rely on static aggregation functions: standard GRPO ( p=1 ) defaults to the Arithmetic Mean, while recent variants like GMPO ( 53 ) and GSPO ( 54 ) ( p\rightarrow 0 ) attempt to mitigate variance by employing the Geometric Mean. Despite their empirical success, these fixed aggregation mechanisms implicitly impose a static optimisation landscape, limiting their adaptability across long-horizon reasoning tasks of varying signal density — the regime in which the trade-off we identify becomes acute. Through empirical investigation, we observe a critical trade-off: certain fixed aggregations frequently suffer from training collapse, while others fail to yield satisfactory performance. Specifically, on dense-signal tasks (where supervision is distributed across many tokens, e.g., MATH ( 14 ) ), standard GRPO ( p=1 ) disproportionately over-weights minor token-level errors, inducing high-variance gradient updates that can lead to training collapse. Conversely, on sparse-signal tasks (where correct reasoning is concentrated in rare, high-magnitude tokens, e.g., AIME ( 17 ) ), GSPO ( p\rightarrow 0 ) overly smooths the probability ratios, suppressing the effective use of these rare “aha moments”. Figure 1 visualises this divergence: AIME24 accuracy peaks at p=3 while MATH500 peaks at p=-1 , with the bottom row showing how the underlying token weight distribution W(p) deforms across the p -axis. Essentially, there is no “silver bullet” among static mean functions; the optimal probability aggregation is not a constant, but rather a function of task signal density and the model’s training progression. To address these fundamental limitations, we propose HölderPO , a generalised policy optimisation framework unifying token-level probability aggregation via the adaptable Hölder mean ( p -norm). By explicitly modulating the parameter p , the framework provides continuous control over the trade-off between gradient concentration and variance bounds. Theoretically, we prove a two-sided trade-off in p : a larger p concentrates the gradient weight distribution on a small subset of tokens, amplifying the effective use of rare informative learning signals at the cost of looser variance bounds. Conversely, a smaller p strictly tightens the variance of the policy gradient estimator, ensuring training stability at the cost of weakening the response to those same sparse signals. Because no static configuration can simultaneously realise both endpoint advantages, we instantiate the HölderPO framework with a dynamic annealing algorithm. By progressively scheduling p from a higher positive value to a negative value during training, this algorithm seamlessly transitions the model from aggressive signal amplification in the early stages to variance-controlled convergence in the later stages. Extensive empirical evaluations across a comprehensive suite of complex reasoning and decision-making benchmarks strongly validate our claims. Built upon the Qwen2.5-Math-7B base ( 42 ) , our ablation studies first confirm the task-specific sensitivity of p : sparse-signal tasks strictly favour higher p values for aggressive signal amplification, whereas dense-signal tasks benefit from lower (possibly negative) p values for gradient stability. Crucially, when explicitly setting p=3 , our approach effectively breaks the existing performance ceiling on the highly challenging AIME benchmark, surpassing the previous 43.3\% accuracy record to achieve 46.7\% . Building on these insights, by employing our dynamic annealing algorithm, HölderPO unifies these advantages without incurring additional computational overhead. Consequently, our approach achieves a state-of-the-art average accuracy of 54.9% across five mathematical benchmarks (AIME, AMC, MATH, Minerva ( 18 ) , and OlympiadBench ( 13 ) ), a 7.2\% relative gain over standard GRPO that surpasses concurrent token-aggregation methods including PMPO ( 52 ) . Beyond mathematical reasoning, this dynamic adaptability extends to open-world agentic tasks, securing an exceptional 93.8% success rate on the ALFWorld benchmark ( 31 ) , a 28.8\% relative gain over GRPO ( 72.8\% ). In summary, our main contributions are as follows: • The HölderPO Framework: We propose HölderPO, a generalised policy optimisation framework that dynamically unifies various mean-based probability aggregations through the adaptable Hölder parameter p . • Theoretical Foundation: We theoretically characterise the two-sided role of p in long-horizon reasoning: a larger p concentrates gradient weight to amplify sparse learning signals, whereas a smaller p strictly bounds gradient variance to ensure training stability. No fixed p realises both endpoint advantages simultaneously, motivating dynamic scheduling. • Empirical Breakthroughs and SOTA Performance: Empirically, explicitly employing a large p=3 breaks the existing performance ceiling on the highly challenging AIME benchmark. Furthermore, instantiating the framework with a dynamic p -annealing algorithm achieves state-of-the-art results, securing a 54.9\% average accuracy across five mathematical benchmarks and an exceptional 93.8\% success rate on ALFWorld agentic tasks. 2 Related Work Reinforcement Learning for Complex Reasoning. Reinforcement Learning