Hard Negative Sample–Augmented DPO Post-Training for Small Language Models

Haocheng Lu [email protected], Minjun Zhu [email protected], Henry Yu [email protected] · 2025

Read Hard Negative Sample–Augmented DPO Post-Training for Small Language Models by Haocheng Lu [email protected], Minjun Zhu [email protected], Henry Yu [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-19728.txt.

Introduction

Haocheng Lu, Minjun Zhu, and Henry Yu improve small language models at math by studying their mistakes in detail. Standard post-training grades each attempted solution as simply right or wrong, yet wrong chain-of-thought solutions often look convincing while hiding one subtle logical, algebraic, or numerical flaw. Their pipeline first fine-tunes on worked solutions, then brings in a compact verifier called MathVerifier.

The verifier scores each attempt along six error dimensions, producing interpretable wrongness and absurdity ratings. Those scores mine hard negatives, solutions that are nearly right but structurally flawed, and weight the preference pairs used in direct preference optimization. On a 1.5-billion-parameter Qwen2.5 model, the guided method beat plain fine-tuning and unweighted optimization, especially on problems where answers come out numerically close but logically broken.

Catalog record

CREATORHaocheng Lu [email protected], Minjun Zhu [email protected], Henry Yu [email protected]
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE43 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-19728

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Hard Negative Sample–Augmented DPO Post-Training for Small Language Models Haocheng Lu [email protected], Minjun Zhu [email protected], Henry Yu [email protected] Abstract Large language models (LLMs) continue to struggle with mathematical reasoning, and common post-training pipelines often reduce each generated solution to a binary outcome: correct or incorrect. This view is limiting in practice, because failures in chain-of-thought (CoT) reasoning are frequently structured —solutions may look convincing while containing subtle logical, algebraic, or numerical flaws. Meanwhile, reinforcement learning from human feedback (RLHF) variants that rely on large reward models or LLM-as-a-judge signals can be expensive, difficult to scale, and unstable to iterate. We propose a lightweight and pragmatic post-training pipeline that targets such structured errors under realistic compute budgets. Starting from supervised fine-tuning (SFT) on MetaMathQA-style CoT data, we introduce a compact MathVerifier that decomposes a candidate solution into a six-dimensional error profile and aggregates it into interpretable wrongness and absurdity scores. These verifier signals serve two roles: (i) mining hard negatives that are near-correct yet structurally flawed, and (ii) defining per-sample importance weights that emphasize the most informative preference pairs. We integrate both into an offline Direct Preference Optimization (DPO) objective via a verifier-guided weighted formulation. Experiments on a 1.5B-parameter Qwen2.5 model show that verifier-guided, weighted DPO yields more targeted improvements than vanilla SFT and unweighted DPO, particularly on problems where solutions are numerically close to correct but logically inconsistent, while avoiding the overhead of training a large reward model or relying on external judges. Keywords— mathematical reasoning, chain-of-thought, verifier-guided training, hard negative mining, direct preference optimization (DPO), small language models 1 Introduction We address the limitation of current math evaluation and post-training pipelines, which still treat solutions in a purely binary way—“correct” or “incorrect”. However, in reinforcement learning (RL) for large language models, sampling is central: we want to focus training on the most informative data. Relying on human annotators to sort through model outputs and identify such examples is prohibitively expensive and slow. Moreover, for mathematical reasoning models, it is not enough to simply find wrong answers; we care how they are wrong, whether they are wildly off, subtly flawed, or superficially convincing but logically inconsistent. The dominant paradigm today is to use an LLM-as-a-judge, but this introduces a high inference cost, error propagation from the judge model, and instability due to subjective or inconsistent judgments. To address this, we propose an automatic pipeline for mining hard negatives in mathematical reasoning: model outputs that are confidently presented but structurally or numerically incorrect. Our pipeline combines lightweight programmatic checks to detect and rank such hard negatives without relying on a costly external judge. We then apply Direct Preference Optimization (DPO) on pairs of preferred vs. hard-negative solutions. DPO is particularly well-suited here because (i) it directly optimizes a preference objective without training an explicit reward model, (ii) it works purely with offline data, and (iii) naturally leverages the contrast between strong but flawed solutions and their better alternatives. By coupling hard-negative mining with DPO, our framework provides a scalable way to sharpen mathematical reasoning in LLM while significantly reducing dependence on human labels and LLM-as-a-judge annotations. 2 Related Work Verifiers and process supervision for mathematical reasoning. A growing line of work improves mathematical reasoning in LLMs by learning explicit verifiers or process reward models. Lightman et al. propose process supervision for mathematical problem solving and introduce the PRM800K dataset, showing that step-level reward models can substantially improve performance on MATH-style benchmarks. [ 12 ] Building on this idea, Math-Shepherd trains a process reward model from automatically constructed supervision, and uses it both to rerank candidate solutions and as a reward model for step-by-step PPO, yielding large gains on GSM8K and MATH. [ 19 ] More recently, VeriThinker treats verification itself as an auxiliary task: the model is fine-tuned to judge the correctness of chain-of-thought (CoT) solutions, which leads to shorter reasoning chains with comparable or even higher accuracy. [ 2 ] In parallel, DeepSeekMath-V2 integrates self-verification into a specialized mathematical model, emphasizing strict step-wise checking and highlighting the high false-positive rate of current verifiers that often accept logically flawed proofs. [ 16 ] Other work such as MathQ-Verify focuses on verifying the well-posedness of math questions themselves, filtering out inconsistent or ill-formed problems before training. [ 17 ] Compared to these approaches, typically involving heavy-weight PRMs or specialized architectures, our work assumes only a lightweight math-verifier and uses it primarily as a tool for mining informative hard negatives rather than as a full-fledged reward model. LLM-as-a-judge and its limitations. Many alignment and evaluation pipelines adopt the LLM-as-a-judge paradigm, where a stronger model grades or compares candidate responses from a target model. This approach has been widely used for automatic evaluation and for constructing preference data at scale, but recent analyzes show that judge models can suffer from substantial biases and instability, including self-preference, position bias, and sensitivity to superficial format differences. [ 9 , 21 , 4 ] These issues are particularly problematic for mathematical reasoning, where correctness is discrete and small logical errors can be masked by fluent, confident text; empirical studies report high false-positive rates when generic LLM judges are asked to validate math solutions or proofs. Our framework deliberately avoids using an external LLM judge both for evaluation and for preference-label generation, and instead relies on programmatic checks and a task-specific math-verifier to decide which solutions are plausible yet incorrect. Preference-based post-training and hard-negative mining. Direct Preference Optimization (DPO) has emerged as a simple and effective alternative to traditional RLHF, recasting preference learning as a supervised objective in preference pairs without training an explicit reward model or performing online rollouts. [ 15 ] Subsequent work has proposed numerous variants and extensions that modify the reference policy, regularization, or loss structure, and several studies emphasize that the quality and diversity of preference pairs—especially the presence of strong hard negatives—are often more critical than the choice of objective itself. In parallel, preference-based methods have been explored in unlearning and safety settings, where negative preference optimization is used to “push down” undesired behaviors while maintaining utility on normal data. [ 23 , 3 ] Our work is complementary to these lines: we use standard DPO as the optimization backbone, and focus on an automatic pipeline for constructing high-value preference pairs in the specific context of mathematical reasoning, where the negative samples are confident but structurally or numerically incorrect solutions mined by our verifier. Semantic similarity, NLI-style reasoning, and ranking-based evaluation. To go beyond purely syntactic or final-answer checks, our math-verifier incorporates semantic signals inspired by natural language inference (NLI) and ranking-based evaluation. NLI corpora such as SNLI and MultiNLI provide large-scale datasets of sentence pairs labeled as entailment, contradiction, or neutral, and have driven progress on models that reason about high-level semantic relations between natural language statements. [ 1 , 20 ] In a complementary direction, Wang et al. argue that directly using similarity scores as absolute metrics for embeddings can be misleading, and propose EvalRank, a ranking-based intrinsic evaluation method that better correlates with downstream performance. [ 18 ] We borrow these perspectives by treating candidate solutions as objects to be ranked rather than independently scored: our verifier combines programmatic answer checking with semantic consistency and NLI-style signals to rank

Read the complete text