Stabilizing Iterative Self-Training with Verified Reasoning via Symbolic Recursive Self-Alignment

Xinyu Zhang · 2026

Read Stabilizing Iterative Self-Training with Verified Reasoning via Symbolic Recursive Self-Alignment by Xinyu Zhang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-21558.txt.

Introduction

Zhang tackles a failure mode in self-training, where a model learns from its own solutions over many rounds. Checking only final answers lets lucky guesses with flawed reasoning slip into the training data. Zhang's Verified Self-Improvement framework checks each reasoning step instead, using a computer algebra library and consistency rules. On grade-school math problems, it rejected about a third of correct-answer solutions as lucky guesses. Accuracy climbed from 80.5 to 91.0 percent across five rounds, while unverified training collapsed.

Catalog record

CREATORXinyu Zhang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE28486
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.21558

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Stabilizing Iterative Self-Training with Verified Reasoning via Symbolic Recursive Self-Alignment Xinyu Zhang Abstract Recursive self-improvement—where a model iteratively trains on its own outputs—promises sustained capability growth but faces a fundamental obstacle: recursive drift . As models train on self-generated data across multiple iterations, errors in intermediate reasoning compound, leading to mode collapse and performance degradation. We propose Neuro-Symbolic Recursive Self-Alignment ( NSRSA ) , which stabilizes iterative self-training by embedding a symbolic verification subsystem that gates training data quality at the reasoning step level . Unlike outcome-only filtering (which admits “lucky guesses” with flawed reasoning), NSRSA verifies each arithmetic operation via sympy , checks logical flow consistency across reasoning steps, and enforces domain constraints. We evaluate NSRSA on GSM8K using Qwen3-4B-Thinking across 5 self-training iterations under five conditions: no verification, outcome verification, majority voting, full NSRSA symbolic verification, and NSRSA with DPO. Our filtering analysis shows that NSRSA rejects approximately 34% of correct-answer solutions that pass outcome verification, eliminating “lucky guesses” with flawed reasoning from the training set. We further demonstrate that constructing DPO preference pairs from NSRSA verification teaches the model to distinguish sound from flawed reasoning (reward accuracy 46% → \to 63%). NSRSA provides an extensible framework that demonstrates how external symbolic verification can make recursive self-improvement measurable and reliable within domains where automated verification is available. 1 Introduction The prospect of recursive self-improvement (RSI)—where an AI system iteratively enhances its own capabilities by training on self-generated data—has been a central aspiration in AI research. Recent work on self-training ( Zelikman et al. 2022 ; Singh et al. 2024 ) , self-rewarding language models ( Yuan et al. 2024 ) , and bootstrapped reasoning ( Hosseini et al. 2024 ) demonstrates that models can meaningfully improve by learning from their own outputs. However, a fundamental challenge limits RSI in practice: recursive drift . When models train on self-generated data iteratively, errors in intermediate reasoning steps compound across iterations. A model may produce a correct final answer through flawed reasoning (a “lucky guess”), and when this solution enters the training set, the model learns to replicate the flawed reasoning pattern. Over multiple iterations, this compounds into systematic degradation—a phenomenon closely related to model collapse in recursive data generation ( Shumailov et al. 2024 ) . The core insight of this work is that the granularity of verification determines the depth of stable recursion . Prior self-training approaches typically filter by outcome only: is the final answer correct? This admits solutions where the reasoning chain contains arithmetic errors, logical inconsistencies, or constraint violations that happened to cancel out. Training on such solutions propagates flawed reasoning patterns. We propose Neuro-Symbolic Recursive Self-Alignment ( NSRSA ) , which embeds a symbolic verification subsystem into the self-training loop. Before any self-generated solution enters the training set, NSRSA verifies: 1. Answer correctness : The final answer matches the ground truth. 2. Arithmetic verification : Every parseable arithmetic expression in the chain-of-thought is evaluated with sympy ( Meurer et al. 2017 ) and confirmed correct. 3. Logical flow : Variable assignments are tracked across reasoning steps to detect hallucinated intermediate values. 4. Constraint satisfaction : Domain constraints (non-negativity for counts, integrality for discrete quantities) are enforced. By requiring every reasoning step to be symbolically sound, NSRSA eliminates lucky guesses from the training data, enabling the model to learn from genuinely correct reasoning chains. This produces more stable self-improvement over deeper recursion. Contributions. (1) We introduce NSRSA , a neuro-symbolic framework for stabilizing recursive self-improvement through step-level symbolic verification. (2) We empirically demonstrate that NSRSA performs substantially more selective filtering than outcome-only verification ( \sim 52% vs. \sim 78% acceptance rate), removing solutions with correct answers but flawed reasoning. (3) We show that verification-based DPO preference pairs teach models to prefer sound reasoning over lucky answers. (4) We provide a complete, reproducible pipeline for iterative self-training with symbolic verification. 2 Related Work Self-training for reasoning. STaR ( Zelikman et al. 2022 ) bootstraps reasoning by training on self-generated rationales that lead to correct answers. ReST-EM ( Singh et al. 2024 ) extends this to an EM-style framework with multiple sampling. V-STaR ( Hosseini et al. 2024 ) trains verifiers alongside generators. Self-rewarding language models ( Yuan et al. 2024 ) use the model itself as a judge. These methods filter primarily by outcome correctness; NSRSA adds step-level symbolic verification to improve training data quality. Process supervision. Lightman et al. 2024 demonstrate that process-level supervision (verifying each reasoning step) outperforms outcome supervision for training math reasoning models. Uesato et al. 2022 compare process and outcome feedback. While these works require human-annotated step labels, NSRSA achieves step-level verification automatically through symbolic computation. Neuro-symbolic reasoning. TORA ( Gou et al. 2024 ) integrates tool use (including symbolic computation) into the reasoning process. Generative theorem provers ( Polu & Sutskever 2020 ) use formal verification. NSRSA differs in that symbolic verification is applied as a filter on training data rather than during inference, making it complementary to these approaches. Model collapse. Shumailov et al. 2024 show that training on model-generated data leads to progressive quality degradation. Huang et al. 2024 demonstrate that LLMs struggle to self-correct without external feedback. NSRSA addresses this by providing external symbolic feedback that prevents error propagation across iterations. 3 Method: Neuro-Symbolic Recursive Self-Alignment 3.1 Iterative Self-Training Loop NSRSA operates as a recursive self-training loop (Figure 1 ). Starting from a base model M_{0} , each iteration i proceeds as: 1. Generate : For each training problem x_{j} , sample N candidate solutions \{y_{j}^{(1)},\ldots,y_{j}^{(N)}\} from M_{i-1} with temperature T>0 . 2. Verify : Apply the verification filter \mathcal{V} to select solutions: \mathcal{D}_{i}=\{(x_{j},y_{j}^{(k)}):\mathcal{V}(y_{j}^{(k)},a_{j})=\text{True}\} , where a_{j} is the ground-truth answer. 3. Train : Fine-tune M_{i-1} on \mathcal{D}_{i} to produce M_{i} . 4. Evaluate : Measure task performance, reasoning quality, and diversity. The critical design choice is the verification filter \mathcal{V} . We compare three strategies: Figure 1: NSRSA pipeline. At each iteration, the model generates multiple solutions per problem. The symbolic verification subsystem checks answer correctness, arithmetic validity (via sympy ), logical flow consistency, and domain constraints. Only solutions passing all checks enter the training set. 3.2 Verification Strategies No Verification (baseline). Randomly sample one solution per problem regardless of correctness. This represents unfiltered self-training. Outcome Verification. Keep all solutions whose final extracted answer matches the ground truth. This is the standard approach in STaR-style methods. NSRSA (Symbolic Verification). A solution passes NSRSA only if all four checks pass: 3.2.1 Check 1: Answer Correctness Extract the final numeric answer using pattern matching (after #### markers for GSM8K) and verify it matches the ground truth within tolerance \epsilon=10^{-6} . 3.2.2 Check 2: Arithmetic Verification Parse the chain-of-thought to extract arithmetic expressions of the form “ A\odot B=C ” where \odot\in\{+,-,\times,\div\} . For each expression, evaluate the left-hand side using sympy.sympify() and verify | LHS -C|<\epsilon . We require \geq 80\% of parseable expressions to be arithmetically correct: \text{ArithRate}(y)=\frac{|\{e\in\text{Expr}(y):|\text{sympy}(e.\text{lhs})-e.\text{rhs}|<\epsilon\}|}{|\text{Expr}(y)|}\geq\tau_{\text{arith}} (1) where \tau_{\text{arith}}=0.8 by default. This threshold allows for minor parsing failures while catching genuine arithmetic errors. Parser coverage analysis. A key subtlety of the arithmetic check is its behavior when no expressions are parseable: if |\text{Expr}(y)|=0 , the pass rate defaults to 1.0 (a vacuous pass). This means the

Read the complete text