SARL: Label-Free Reinforcement Learning by Rewarding Reasoning Topology
Yifan Wang, Bolian Li, David Cho, Ruqi Zhang, Fanping Sui, Ananth Grama, [0.45em] Purdue University · 2026
Read SARL: Label-Free Reinforcement Learning by Rewarding Reasoning Topology by Yifan Wang, Bolian Li, David Cho, Ruqi Zhang, Fanping Sui, Ananth Grama, [0.45em] Purdue University (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-27977.txt.
Introduction
Wang and five co-authors propose SARL, a way to train reasoning models without labeled answers. Instead of scoring only the final answer, SARL maps the steps of each response and rewards sound reasoning structure. On math tests it beats earlier label-free methods and even some methods trained with true answers, with large gains on AIME25. On open-ended WildBench tasks it also leads prior label-free methods and a preference-based rival.
Catalog record
| CREATOR | Yifan Wang, Bolian Li, David Cho, Ruqi Zhang, Fanping Sui, Ananth Grama, [0.45em] Purdue University |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 41748 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.27977
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SARL: Label-Free Reinforcement Learning by Rewarding Reasoning Topology Yifan Wang, Bolian Li, David Cho, Ruqi Zhang, Fanping Sui, Ananth Grama, [0.45em] Purdue University Abstract SARL: Label-Free Reinforcement Learning by Rewarding Reasoning Topology Yifan Wang 1 Bolian Li 1 David Cho 1 Ruqi Zhang 1 Fanping Sui 2 Ananth Grama 1 1 Purdue University 2 Texas Instruments Reinforcement learning has become central to improving large reasoning models, but its success still relies heavily on verifiable rewards or labeled supervision. This limits its applicability to open ended domains where correctness is ambiguous and cannot be verified. Moreover, reasoning trajectories remain largely unconstrained, and optimization towards final answer can favor early exploitation over generalization. In this work, we ask whether general reasoning ability can be improved by teaching models how to think (the structure of reasoning) rather than what to produce (the outcome of reasoning) and extend traditional RLVR to open ended settings. We introduce structure aware reinforcement learning (SARL), a label free framework that constructs a per response Reasoning Map from intermediate thinking steps and rewards its small world topology, inspired by complex networks and the functional organization of the human brain. SARL encourages reasoning trajectories that are both locally coherent and globally efficient, shifting supervision from destination to path. Our experiments on Qwen3-4B show SARL surpasses ground truth based RL and prior label free RL baselines, achieving the best average gain of 9.1% under PPO and 11.6% under GRPO on math tasks and 34.6% under PPO and 30.4% under GRPO on open ended tasks. Beyond good performance, SARL also exhibits lower KL divergence, higher policy entropy, indicating a more stable and exploratory training and generalized reasoning ability. Keywords: Structure aware reinforcement learning, label free RLVR, LLM post-training * Date: 2026-03-29 Code: https://github.com/cacayaya/SARL Contact: [email protected] 1 Introduction Recently, Large Reasoning Models (LRMs) have demonstrated transformative capabilities in tackling complex, reasoning-intensive tasks, especially in mathematics and code generation [ 5 , 29 ] . Central to this progress is the Chain-of-Thought (CoT) paradigm, where models generate explicit, step-by-step reasoning steps before producing a final answer [ 27 , 25 ] . These reasoning capabilities are typically catalyzed by supervised fine-tuning (SFT) and, more crucially, by reinforcement learning (RL) [ 15 , 18 ] . By rewarding correct final outcomes, RL allows models to autonomously explore and refine their internal logic, leading to the emergence of sophisticated “thinking” behaviors [ 10 , 23 ] . However, the current success of reasoning models is largely tethered to the Reinforcement Learning from Verifiable Rewards (RLVR) framework [ 5 , 18 ] . RLVR relies heavily on tasks with objective ground truths and automated verifiers to provide reward signals. This dependency creates a “bottleneck of verifiability”: in domains where ground truth is ambiguous, expensive to label, or non-existent, such as open-ended strategic planning or complex philosophical inquiry, the RL signal largely vanishes. Furthermore, current reward modeling predominantly focuses on outcome-level rewards, which treat the reasoning process as a “black box” and offer limited control over intermediate steps. While Process Reward Models (PRMs) [ 10 , 13 , 7 ] have been introduced to provide more granular feedback, they are still constrained by the cost of step-level supervision and by limited scalability to tasks without clear, rule-based verification. However, to some extent, PRMs still optimize reasoning primarily through its contribution to final-answer correctness, offering limited control over how intermediate thoughts are structured or organized. This raises a central question: Can complex reasoning be optimized not through supervision derived from ground-truth answers, but by directly shaping how the model thinks? A compelling answer to this question lies in the macro-scale functional architecture of the human brain [ 20 ] . Neuroscientific evidence suggests that human high-level cognition is facilitated by the small-world organization of the brain’s connectome [ 3 ] . This topology is characterized by dense local clusters responsible for specialized information processing and a few strategic long-range shortcuts that minimize the path length between disparate functional modules [ 26 ] . Such a structure allows the brain to achieve an optimal balance between functional segregation and global integration, which enables efficient relational reasoning and creative insight even in the absence of external feedback [ 24 ] . In this biological framework, a thinking process is essentially a traversal across this small-world network. Here, the quality of reasoning is determined by its navigational efficiency and topological coherence [ 17 ] . Inspired by this architecture, we hypothesize that reasoning models can benefit from structural priors that encourage more efficient exploration in latent conceptual space. In this work, we introduce Structure Aware Reinforcement Learning (SARL), a label free training framework that improves reasoning by optimizing the topology of the reasoning process itself rather than relying on ground truth labels. SARL constructs a per response Reasoning Map from intermediate thinking steps and rewards its small world structure, encouraging reasoning trajectories that are both locally coherent and globally efficient. Our main contributions are as follows: • We propose SARL, a label-free RL framework that optimizes the structure of reasoning rather than its outcome, by rewarding the small-world topology of per-response Reasoning Maps. • We show that SARL is effective in both mathematical and open ended reasoning, outperforming prior label free baselines and matching or surpassing ground truth based RL in several settings. • We find that SARL induces a more stable training regime, with lower policy drift and higher entropy, indicating more exploration and generalization ability. Figure 1: Overview of Structure Aware Reinforcement Learning (SARL). Left: SARL replaces outcome based supervision with a label free structure reward computed from the model’s reasoning process. Middle: the reward is motivated by the small world prior, which balances local specialization and global integration. Right: construction of the SARL Reasoning Map and Structure Reward. 2 Related Work Reasoning Structure in Human Cognition and Topology of LRMs Reasoning. The human brain’s connectome exhibits a small-world topology [ 26 , 3 ] : dense local clustering within specialized functional modules coexists with short global path lengths across the entire network, enabling simultaneous functional segregation and rapid cross-module integration [ 24 ] . This structural property is not merely architectural; it underlies the brain’s capacity to navigate abstract cognitive spaces efficiently, coordinating between specialized processing units at minimal energetic cost [ 17 , 20 ] . Recent empirical work has begun uncovering analogous structural signatures in Large Reasoning Models (LRMs). 14 extract reasoning graphs from the hidden-state trajectories of LRMs and find that distilled reasoning models exhibit substantially higher cyclicity, diameter, and small-world index than base models, with these structural advantages scaling with task difficulty and correlating positively with accuracy. From a geometric perspective, 33 model chain-of-thought reasoning as smooth flows in the model’s representation space, showing that logical transitions induce structured geometric trajectories governed by the semantics of intermediate propositions. At the level of trace analysis, 4 introduce graph-based structural metrics for CoT traces and demonstrate that the fraction of abandoned reasoning branches, rather than trace length or self-review frequency, is the strongest predictor of correctness, providing causal evidence that structural organization shapes reasoning outcomes. 21 further show that topological data analysis features of reasoning traces carry substantially higher predictive power for reasoning quality than standard graph-connectivity