SCPRM: A Schema-aware Cumulative Process Reward Model for Knowledge Graph Question Answering

Jiujiu Chen, Yazheng Liu, Sihong Xie, Hui Xiong · 2026

Read SCPRM: A Schema-aware Cumulative Process Reward Model for Knowledge Graph Question Answering by Jiujiu Chen, Yazheng Liu, Sihong Xie, Hui Xiong (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-02819.txt.

Introduction

Jiujiu Chen and three co-authors build a reward model that grades the steps of reasoning over a knowledge graph, for question answering. Grading steps helps guide search, but a common fault is risk compensation: a wrong step gets offset by a later right one, so a flawed path scores well. Graphs worsen the risk, since many paths link the same entities.

Their model judges each step in light of the path so far, and adds a schema distance between the current step and the target hidden in the question, giving cumulative, forward-looking rewards. Plugged into a tree search, it drives multi-hop reasoning over medical and legal graphs and a general question set. Scores improved modestly on average over strong reward and search combinations, in fields where a flawed path is costly.

Catalog record

CREATORJiujiu Chen, Yazheng Liu, Sihong Xie, Hui Xiong
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE32 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.02819

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SCPRM: A Schema-aware Cumulative Process Reward Model for Knowledge Graph Question Answering Jiujiu Chen, Yazheng Liu, Sihong Xie, Hui Xiong Abstract Large language models excel at complex reasoning, yet evaluating their intermediate steps remains challenging. Although process reward models provide step-wise supervision, they often suffer from a risk compensation effect, where incorrect steps are offset by later correct ones, assigning high rewards to flawed reasoning paths. This issue is further exacerbated in knowledge graph (KG) reasoning, as there may exist multiple paths between the start and end entities in the KGs, and a risky step can make the reasoning path flawed. Those limitations are problematic in risk-sensitive tasks such as medical and legal KG reasoning. To address the issues, we propose a Schema-aware Cumulative Process Reward Model (SCPRM) that evaluates reasoning paths by conditioning on the reasoning prefix , and incorporating schema distance between current reasoning step and the implicit target parsed from the query, which provides cumulative and future rewards to guide the path explorations. We further integrate SCPRM into Monte Carlo Tree Search (MCTS) as SCPRM-MCTS to conduct multi-hop reasoning on KGs for question answering (QA) tasks. Across medical and legal KGQA and CWQ, SCPRM-MCTS improves the performance of Hits@k by an average of 1.18% over strong baselines, demonstrating more accurate and risk-sensitive reasoning evaluation. 1 Introduction As Large Language Models (LLMs) evolve, they have transitioned from simple question answering to complex multi-step reasoning, largely driven by the emergence of Chain-of-Thought (CoT) prompting 38 . However, the inherently linear structure of CoT makes LLMs vulnerable to error propagation, as a single mistake can derail the entire reasoning trajectory and lead to an incorrect final answer 33 . While outcome-based reward models can verify final answers 10 ; 25 , they cannot distinguish whether the model arrived at the correct answer through logic or hallucination 42 ; 16 ; 40 . To mitigate this, researchers have shifted towards Process Reward Models (PRMs) that evaluate intermediate steps combined with search algorithm, e.g., Tree-of-Thoughts 41 . Figure 1: Comparison of reward models for reasoning based on medical KGQA. Red rewards indicate the wrong ranks of traversed paths on the KG. Currently, existing PRMs have shown strong performance in mathematical reasoning and code generation tasks 45 ; 17 , but those tasks are typically tolerant of minor intermediate risky steps, as later correct steps can recover the final answer. Consequently, vanilla PRMs usually aggregate rewards additively across the trajectory, making them relatively insensitive to localized biases or risky decisions. This assumption breaks down in Knowledge Graph Question Answering (KGQA). The complex KG topology allows for multiple reasoning paths between the start entity and the answer, but the logically valid paths are strictly limited. This is particularly severe in risk-sensitive applications such as medical or legal KGQA (illustrated in Figure 1 ). For example, a path traversing entities related to symptoms involve logically irrelevant or incorrect information to the query, but still coincidentally reach the correct final answer. Vanilla PRMs assign such paths with high rewards by averaging risk and correctness, allowing risky steps to be offset by later rewards. In risk-sensitive reasoning tasks, such compensation is unacceptable. Once a high-risk step occurs, the remaining trajectory should be substantially penalized, even if it reaches a technically correct endpoint 46 . Beyond risk sensitivity, KGQA requires adherence to implicit constraints expressed in the query 4 ; 18 . While conditional reward models 46 can prevent additive reward compensation, they introduce a severe length bias. Multiplying step-wise probabilities ranging from 0 to 1 monotonically decreases the total reward, so the conditional reward models naturally tie the total reward to trajectory length, disproportionately penalizing longer reasoning path. This length sensitivity severely weakens logical exploration in KGQA, where multi-hop paths are necessary for complex queries. To address these challenges, we propose the Schema-aware Cumulative Process Reward Model (SCPRM), designed specifically for risk-sensitive multi-hop reasoning in KGQA. Inspired by the A^{*} search algorithm 12 , SCPRM decouples reasoning evaluation into two dimensions: cumulative reward and future reward. Unlike prior A^{*} -style methods that rely on arbitrary heuristics or hand-crafted rule-based evaluators, SCPRM is a parameterized model with base LLM and heads for reward outputs, which is different from the design of next-token prediction. The cumulative reward utilizes step-wise conditional probabilities to capture risk propagation, ensuring that a highly risky step mathematically results in a lower reward since the step. Simultaneously, the future reward is computed by a schema-aware estimator, which extracts reasoning logic from the query to measure the schema distance between the current step and the target, and ensures logically consistence in multi-hop paths without length bias. We evaluate SCPRM by integrating it with MCTS on KGQA tasks, and the contributions are summarized as follows: • Non-Compensatory PRM Architecture: We introduce SCPRM, which leverages conditional probabilities to model cumulative path rewards reversed by the past cost in A^{*} algorithm. This ensures risky steps impose persistent and multiplicative penalties, effectively vetoing risky trajectories that vanilla PRMs would artificially rescue. • Schema-Aware Future Estimation: We propose a learnable future reward module that utilize logical schema parsed from the query. By computing schema distance between the query constraints and current reasoning, SCPRM learns the query logic by overcoming the inherent length bias of conditional reward models. • Strong Performance on Risk-Sensitive KGQA: Extensive experiments on medical and legal KGQA and Complex WebQuestions (CWQ) datasets demonstrate SCPRM with different small base LLMs (e.g., Llama3.1-8B) achieves average 1.18 % improvement in Hits@k compared to closed-source powerful LLMs like GPT-4o-mini. 2 Related Work Reward models are learned evaluators to score candidate outputs according to human or task preferences 27 , and play an important role in alignment and reasoning optimization. Early reward functions were largely heuristic or rule-based, relying on manually designed criteria such as formatting constraints or keyword matching, which limited their applicability to open-ended generation 11 ; 13 . Subsequent work introduced Outcome Reward Models (ORMs) that only supervise the final answer of the reasoning trajectory 6 . Later studies also connected outcome supervision to value estimation for decoding, demonstrating that final-answer rewards can guide search toward more promising reasoning trajectories 43 . However, ORMs cannot localize intermediate reasoning errors due to the sparse feedback. To address this limitation, recent work has shifted toward Process Reward Models (PRMs), which assign rewards to intermediate reasoning steps rather than only final outputs. Process supervision was shown to reduce reasoning errors and outperform outcome-only supervision on challenging math benchmarks 34 ; 20 . Subsequent methods improved scalability by replacing expensive human step annotations with automatically generated supervision, e.g., Monte Carlo rollouts or ranking-based Q-value formulations that model dependencies across steps 36 ; 47 ; 19 . Beyond mathematics, PRMs have been extended to code generation using execution feedback as stepwise supervision 26 ; 2 ; 17 , and medical and legal reasoning 35 ; 44 ; 15 ; 5 . However, in risk-sensitive domains such as medicine and law, existing PRMs often suffer from risk compensation, where high rewards on some steps can offset critical errors. This limitation motivates the need for new process reward models that explicitly account for stepwise risk and logical safety. Knowledge graph question answering (KGQA) aims to answer natural

Read the complete text