Multi-Path Collaborative Reasoning via Reinforcement Learning

Jindi Lv, Yuhao Zhou, Zheng Zhu, Xiaofeng Wang, Guan Huang, Jiancheng Lv · 2025

Read Multi-Path Collaborative Reasoning via Reinforcement Learning by Jindi Lv, Yuhao Zhou, Zheng Zhu, Xiaofeng Wang, Guan Huang, Jiancheng Lv (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-01485.txt.

Introduction

Jindi Lv and co-authors widen how language models reason through hard tasks. Step-by-step thought chains help a lot. But a single chain tends to lock onto one path. It rarely weighs other fair paths in full.

The team lets many paths grow and aid each other. Reinforcement learning guides that joint search. Soft abstract tokens let paths share sense in a smooth space. Joint search should find better answers on tasks with more than one good route. It may also cut the risk of one early slip ruining a whole chain.

Catalog record

CREATORJindi Lv, Yuhao Zhou, Zheng Zhu, Xiaofeng Wang, Guan Huang, Jiancheng Lv
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE48 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-01485

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Multi-Path Collaborative Reasoning via Reinforcement Learning Jindi Lv, Yuhao Zhou, Zheng Zhu, Xiaofeng Wang, Guan Huang, Jiancheng Lv Abstract Chain-of-Thought (CoT) reasoning has significantly advanced the problem-solving capabilities of Large Language Models (LLMs), yet conventional CoT often exhibits internal determinism during decoding, limiting exploration of plausible alternatives. Recent methods attempt to address this by generating soft abstract tokens to enable reasoning in a continuous semantic space. However, we find that such approaches remain constrained by the greedy nature of autoregressive decoding, which fundamentally isolates the model from alternative reasoning possibilities. In this work, we propose Multi-Path Perception Policy Optimization (M3PO), a novel reinforcement learning framework that explicitly injects collective insights into the reasoning process. M3PO leverages parallel policy rollouts as naturally diverse reasoning sources and integrates cross-path interactions into policy updates through a lightweight collaborative mechanism. This design allows each trajectory to refine its reasoning with peer feedback, thereby cultivating more reliable multi-step reasoning patterns. Empirical results show that M3PO achieves state-of-the-art performance on both knowledge- and reasoning-intensive benchmarks. Models trained with M3PO maintain interpretability and inference efficiency, underscoring the promise of multi-path collaborative learning for robust reasoning. (a) (b) (c) (d) Figure 1 : Comparative analysis of reasoning paradigms. Figures (a) and (b) present results on knowledge and STEM benchmarks, while Figures (c) and (d) evaluate Soft Thinking without training on MATH-500 and GSM8k. Soft Thinking shows degraded performance, indicating extraneous inference noise. In contrast, M3PO consistently improves performance on knowledge- and reasoning-intensive benchmarks (with up to a 9.5% average gain on five knowledge datasets), demonstrating the superiority of multi-path reasoning paradigm. 1 Introduction The advent of Chain-of-Thought (CoT) ( 51 ; 21 ; 14 ) has markedly improved the reasoning capabilities of Large Language Models (LLMs) ( 29 ; 50 ; 43 ) on complex tasks. By generating explicit intermediate steps in natural language, CoT decomposes problems and guides models step-by-step toward the answer ( 4 ) . However, the discrete token decoding underlying standard CoT induces internal determinism, which limits the exploration of plausible alternatives ( 57 ) . Recent efforts seek to transcend discrete token decoding by introducing continuity into the reasoning process. Latent reasoning ( 15 ; 39 ; 41 ) propagates hidden states instead of tokens to enable differentiable reasoning, but such representations often deviate from the pretrained embedding space, requiring costly alignment and limiting compatibility ( 38 ) . To address this, Soft Thinking ( 61 ; 52 ) performs soft aggregation within the input embedding space, preserving model compatibility while simulating a smooth semantic flow. Nevertheless, we reveal that Soft Thinking inherently lacks the ability to represent diverse semantic trajectories in parallel ( 52 ) . As shown in Figure 2 , during consecutive decoding, each subsequent distribution consistently follows the dominant semantic token from the previous step, resulting in a coherent but exclusive progression along a single trajectory. Although soft aggregation improves information capacity, it primarily reinforces the dominant path while introducing extraneous noise over time, as evidenced in Figures 1(c) and 1(d) . This phenomenon stems from the greedy nature of LLMs in autoregressive generation, where hidden states evolve at each step toward the most confident semantic direction ( 52 ) . Such locally optimal decisions are amplified over long reasoning chains, resulting in a cumulative bias toward dominant paths and exclusion of alternatives. Thus, when a reasoning trajectory begins with a flawed premise, the lack of timely corrective feedback allows erroneous logic to propagate unimpeded through subsequent steps ( 53 ) . In this work, we propose M3PO , a novel M ulti- P ath P erception P olicy O ptimization framework for reasoning robustness. M3PO leverages the inherent independence among policy rollouts to create natural multi-path sources. Building on this perspective, we design a collaborative mechanism that explicitly incorporates cross-path interactions into the policy update. Through reward-guided policy optimization, M3PO progressively internalizes more consistent and reliable reasoning patterns. This paradigm promotes robust policy optimization by allowing trajectories to benefit from cross-path insights. Specifically, the parameter-free gate facilitates step-level information exchange, providing trajectories with opportunities to gently refine their reasoning chain and reduce the persistence of local biases. In particular, by maintaining and refining multiple hypotheses in parallel, M3PO aligns with human-like cognition ( 9 ; 10 ; 2 ) , striking an adaptive balance between exploration and exploitation. Empirically, as shown in Figures 1(a) and 1(b) , M3PO achieves superior performance over the standard CoT approach across both knowledge- and reasoning-intensive benchmarks. The advantage is particularly pronounced on knowledge-oriented tasks, where it delivers an average improvement of 9.5%. These results validate the distinct advantage of our proposed multi-path reasoning paradigm. We highlight the main contributions of this paper below: • We hypothesize parallel rollouts in reinforcement learning as a natural source of reasoning diversity, eliminating the need for curated auxiliary datasets. • We design a multi-path collaborative learning mechanism that integrates cross-path interactions into policy updates, enabling implicit exploration-exploitation balance. • Our framework achieves state-of-the-art reasoning performance (up to +9.5% accuracy over conventional CoT) on standard benchmarks without additional parameters. Figure 2 : An illustrative case of probability distribution in vanilla Soft Thinking method. Each step aligns closely with the top-1 token from the prior step, leading to rapid concentration along a single semantic path. 2 Related Works 2.1 Continuous Space Reasoning Recent works have explored continuous space reasoning as an alternative to explicit CoT generation ( 31 ; 12 ; 48 ; 46 ; 45 ) . Several studies demonstrate that Transformers can perform multi-hop reasoning implicitly through their internal hidden states, without generating intermediate textual steps ( 55 ; 36 ) . Deng et al. ( 8 ; 7 ) further exploit this idea by using hidden dynamics directly for reasoning, bypassing symbolic output altogether. Geiping et al. ( 11 ) extend this paradigm with a depth-recurrent architecture that reuses Transformer layers during inference, increasing computational depth per token and enabling internal processing. Another line focuses on explicit continuous reasoning through latent representations ( 8 ; 41 ) . COCONUT ( 15 ) enables reasoning in the hidden state space by replacing discrete tokens with continuous vectors, eliminating the need for human-readable chains. CODI ( 39 ) learns to align recurrent state updates via self-distillation, forming structured reasoning trajectories in the latent space. Although conceptual advances, they often struggle to scale to larger models and maintain performance on complex reasoning tasks. Figure 3 : Overview of the M3PO framework. For a given question, multiple trajectories are first generated in parallel (Rollout). During thinking, these paths dynamically interact via a novel collaborative mechanism to refine intermediate reasoning steps (Collaborative Learning). The optimized trajectories are then used for policy updates via group-relative advantage estimation, closing the loop between multi-path exploration and policy learning (Policy Optimization). More recently, training-free approaches such as Soft Thinking ( 61 ; 13 ; 52 ) and Mixture-of-Inputs (MoI) ( 62 ) have emerged, leveraging output distributions to align hidden states with input embeddings, facilitating iterative reasoning in continuous space. Building on this, HRPO ( 59 ) introduces a hybrid policy that fuses discrete token embeddings with soft embeddings through a gating mechanism, enabling richer internal state evolution during reasoning. Despite these innovations, Soft

Read the complete text