Do Post-Training Algorithms Actually Differ? A Controlled Study Across Model Scales Uncovers Scale-Dependent Ranking Inversions

Xiaoyi Li · 2026

Read Do Post-Training Algorithms Actually Differ? A Controlled Study Across Model Scales Uncovers Scale-Dependent Ranking Inversions by Xiaoyi Li (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-19335.txt.

Introduction

Li asks whether the many post-training methods for language models truly differ in effect. A shared testbed, OXRL, runs 51 methods under one setup for fair checks. The study covers eight methods at four model sizes and three test fields. Rankings flip with scale: the worst small-scale method tops the chart at 7 billion weights. Twenty variants of DPO add no real gain over the plain form. Score spreads shrink by over 30 times outside the training field. Scale, the study finds, dwarfs method choice.

Catalog record

CREATORXiaoyi Li
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE42574
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.19335

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Do Post-Training Algorithms Actually Differ? A Controlled Study Across Model Scales Uncovers Scale-Dependent Ranking Inversions Xiaoyi Li Abstract Post-training alignment has produced dozens of competing algorithms—DPO, SimPO, KTO, GRPO, and others—yet practitioners lack controlled comparisons to guide algorithm selection. We present oxRL , a unified framework implementing 51 post-training algorithms with identical infrastructure, enabling the first large-scale apples-to-apples evaluation. Our study spans 8 algorithms across 4 model scales (0.5B–7B), 3 evaluation domains , and a 20-variant DPO taxonomy (100 runs at 1.5B, 5 seeds each), totaling ∼ \sim 240 training runs on H100 GPUs. Three headline findings emerge. (1) Algorithm rankings are unstable across scale : at 1.5B, online RL (SGRPO) tops all methods at 58.0% ± \pm 0.57 on GSM8K; by 7B, the worst small-scale method (SimPO) becomes the best (85.8%), a complete ranking inversion driven by model scale rather than LoRA regularization (confirmed via 2 × \times 2 factorial). (2) Loss function modifications yield negligible gains : none of 20 DPO variants significantly outperform vanilla DPO after Bonferroni correction; the sole significant outlier, SimPO, is worse ( − - 11.5 pp, p < 10 − 4 p<10^{-4} ). (3) Algorithm leverage is task-specific : the 19.3 pp GSM8K spread collapses to 0.54 pp on MATH ( 36 × 36\times ) and 0.47 pp on general-domain benchmarks ( 41 × 41\times ), confirming that algorithm choice matters primarily within the training distribution. These findings yield a hierarchy of leverage for practitioners: model scale ( ∼ {\sim} 50 pp) ≫ \gg training paradigm ( ∼ {\sim} 10 pp) ≫ \gg online vs. offline ( ∼ {\sim} 9 pp) ≫ \gg loss function ( ∼ {\sim} 1 pp). We release all code, configs, and evaluation data as a living community benchmark. 1 Introduction Post-training of large language models (LLMs)—aligning a pretrained model with human preferences or task objectives—has become the decisive stage separating a capable base model from a useful one ( Ouyang et al. 2022 ; Touvron et al. 2023 ) . The past two years have produced an explosion of post-training algorithms: online RL methods like PPO ( Schulman et al. 2017 ) and GRPO ( Shao et al. 2024 ) , offline preference methods like DPO ( Rafailov et al. 2023 ) , and a growing family of variants—SimPO ( Meng et al. 2024 ) , KTO ( Ethayarajh et al. 2024 ) , ORPO ( Hong et al. 2024 ) —each claiming improvements. DeepSeek-R1 ( DeepSeek-AI 2025 ) demonstrated that pure RL post-training can elicit sophisticated reasoning, further expanding the design space. The practitioner’s dilemma. Each algorithm paper reports results on different base models, datasets, and evaluation suites, making cross-method comparison unreliable. The few existing comparative studies ( Xu et al. 2024 ) cover only 2–3 methods and often confound algorithmic differences with implementation details. Three critical dimensions remain unexplored: how rankings change across model scale , which DPO variant modifications actually matter (over 20 exist ( Rafailov et al. 2023 ) ), and the compute-performance tradeoff between online and offline methods. Our approach. We address these gaps through the first large-scale, controlled empirical comparison of post-training algorithms, resting on three pillars: 1. Unified framework. All 51 algorithms share the same model loading, data pipeline, distributed training (DeepSpeed ZeRO-3), and evaluation harness, eliminating codebase-as-confound. 2. Controlled design. We fix the model family (Qwen 2.5), training data, optimizer, and LR schedule, varying only the loss function. Where algorithms require additional components (reference models, rollouts), these are standardized. 3. Multi-scale evaluation. 8 algorithms across 4 scales (0.5B, 1.5B, 3B, 7B) on GSM8K, plus 20 DPO variants at 1.5B with 5 seeds each. At 3B, we train under both full FT and LoRA, providing a 2 \times 2 factorial with 7B to disentangle scale from LoRA effects. Key findings. • Rankings invert across scale. At 1.5B, SGRPO (online RL) tops all methods at 58.0% \pm 0.57, outperforming SFT by 3.6 pp and DPO by 8.9 pp. By 7B, SimPO—worst at small scale—becomes best (85.8%), while SFT collapses to near-baseline. A 2 \times 2 factorial confirms scale, not LoRA, drives the inversion. • Zero of 20 DPO variants beat vanilla DPO. In 100 runs with Bonferroni correction, SimPO is the sole significant outlier ( - 11.5 pp, p<10^{-4} )—and it is worse . • Algorithm leverage is task-specific. The 19.3 pp GSM8K spread collapses to 0.54 pp on MATH ( 36\times ) and 0.47 pp on general-domain benchmarks ( 41\times ); no method degrades general capabilities. • A hierarchy of leverage. Scale ( {\sim} 50 pp) \gg training paradigm ( {\sim} 10 pp) \gg online vs. offline ( {\sim} 9 pp) \gg loss function ( {\sim} 1 pp). • A methodological warning. We discover a hidden determinism bug in PyTorch’s DistributedSampler that silently eliminates seed-dependent variance, likely affecting other distributed training studies. We release oxRL as a standardized benchmark protocol for post-training algorithms—analogous to GLUE for NLU or HELM for evaluation, but targeting the training algorithm axis. 1 1 1 Code and data: https://github.com/warlockee/oxRL . We provide four artifacts: (1) a codebase where any new algorithm can be compared by implementing a single loss function; (2) a prescribed evaluation protocol with statistical tests; (3) reproducible YAML configs for every algorithm \times scale combination; and (4) all raw evaluation outputs for independent verification. 2 Related Work RLHF and online RL. The RLHF pipeline ( Ouyang et al. 2022 ; Touvron et al. 2023 ) trains a reward model and optimizes the policy using PPO ( Schulman et al. 2017 ) with a KL penalty. GRPO ( Shao et al. 2024 ) simplifies this by eliminating the value network through group-normalized rewards. We evaluate three GRPO loss variants (SGRPO, GSPO, CISPO), showing that the token-vs-sentence-level loss distinction exhibits scale dependence. DPO and its variants. Rafailov et al. 2023 showed the RLHF objective can be reparameterized to bypass reward modeling. This spawned a large family: IPO ( Azar et al. 2024 ) , SimPO ( Meng et al. 2024 ) , KTO ( Ethayarajh et al. 2024 ) , ORPO ( Hong et al. 2024 ) , and others. Our work provides the first controlled evaluation of 20 DPO variants under identical conditions. Comparative studies. Xu et al. 2024 compare DPO and PPO, and Ivison et al. 2024 disentangle data, procedures, and loss functions, but both cover only 2–3 algorithms. Spangher et al. 2025 benchmark 17 algorithms across 3,500 runs but at a single scale—they cannot detect the ranking inversions we observe between 3B and 7B. Wu et al. 2025 demonstrate GRPO–DPO theoretical equivalence; our empirical results partially contradict this (SGRPO outperforms DPO by 8.9 pp at 1.5B). Saeidi et al. 2025 evaluate DPO variants without multi-seed runs or multiple comparison correction; our 100-run Bonferroni-corrected sweep provides stronger statistical evidence. Tajwar et al. 2024 and Tan et al. 2025 address related dimensions (on-policy data, RL scaling laws) that our work complements. No prior work explores the DPO variant space at our scale or examines ranking stability across model sizes. Our 0/20 result for DPO variants parallels the finding of Lucic et al. 2018 that most GAN variants do not outperform the original under controlled evaluation—a pattern that may be general to families of loss

Read the complete text