PEPS: Quantum-Inspired Reinforcement Learning for Coherent Reasoning Traces in LLMs Venkat Margapuri, Garik Kazanjian, Naren Kosaraju Abstract Large Language Models (LLMs) often struggle with maintaining coherent multi-step reasoning traces, particularly in tasks that require a structured logical flow. This work introduces a quantum-inspired approach to address the challenge by incorporating a fidelity-based reward derived from Projected Entangled Pair States (PEPS) into Proximal Policy Optimization. Unlike prior approaches that use direct supervision or contrastive objectives, the proposed method guides learning through structural consistency, offering a novel approach to enforce global coherence in generated reasoning traces. The proposed framework is evaluated using multiple coherence-determining metrics on diverse datasets such as GSM8K, StrategyQA, and EntailmentBank spanning arithmetic, intuitive, and entailment-based reasoning. Results show that the proposed quantum-inspired approach offers significant improvements over supervised, contrastive, and pretrained baseline approaches, highlighting the effectiveness of quantum-inspired fidelity as a foundation to improve reasoning trace coherence in LLMs. I Introduction In recent times, large language models (LLMs) have made substantial progress in their ability to perform diverse natural language tasks such as open-domain question answering [ 1 ] [ 2 ] , code generation [ 3 ] [ 4 ] , summarization [ 5 ] , and dialogue systems [ 6 ] . Despite the progress, their ability to perform coherent multi-step reasoning is limited even when the final answers are correct. The logical gaps in the underlying chains of thought pose challenges for applications in the fields of scientific discovery, legal analysis, and multi-hop factual verification where transparency, factual grounding, and coherent reasoning are essential. These shortcomings are often masked by fluent surface-level language but the fundamental pitfall with the current LLM alignment techniques remains. LLM alignment techniques such as supervised fine-tuning (SFT) [ 7 ] , Reinforcement Learning from Human Feedback (RLHF) [ 8 ] , Direct Preference Optimization (DPO) [ 9 ] , and Constitutional AI [ 10 ] optimize scalar reward signals or preference-based objectives that reflect end-task performance. While these approaches are effective at improving fluency, they don’t always emphasize logical consistency and step-wise coherence within multi-step reasoning traces. To address the challenges with the current alignment approaches, this study proposes a novel quantum-inspired trace-alignment approach that precisely focuses on improving the internal consistency of multi-step reasoning traces. At the core of the proposed approach is a quantum-inspired Projected Entangled Pair States (PEPS) [ 11 ] model which draws from quantum many-body physics to capture entanglement across distributed systems. The PEPS formalism is used to represent reasoning traces as structured tensor networks, where each step in the trace is encoded as a projection entangled with the rest of the trace. By contracting the tensor network, the model computes the fidelity score, a scalar value that reflects the global consistency of the reasoning trace. The fidelity score serves as a holistic, structure-aware measure to determine the alignment of the individual steps as they form a coherent multi-step argument. However, the fidelity score is a non-differentiable sequence-level scalar signal that cannot be optimized directly using token-level objectives typically employed in approaches such as SFT. Therefore, this study employs a reinforcement learning (RL) formulation to incorporate the fidelity score into the model’s behavior. Precisely, Proximal Policy Optimization (PPO) [ 12 ] is used as the policy optimization algorithm due to its robustness to noisy and high-variance rewards, and effectiveness in LLM fine-tuning. PPO allows the LLM to iteratively adapt its generation policy in response to the PEPS-guided feedback (fidelity score) while preserving training stability and avoiding policy collapse. This work contributes to the research on quantum-inspired LLM alignment by applying PEPS, a tensor network formalism, to guide multi-step reasoning through RL. The contributions of the work are: • The introduction of a quantum-inspired use case of PEPS to model multi-step reasoning as a structured tensor network to enable holistic coherence scoring via trace-level fidelity. • The development of a PPO-based RL framework that optimizes multi-step reasoning using PEPS-based fidelity as a sequence-level reward signal, enabling direct optimization of logical consistency beyond token-level objectives. • The introduction of a comprehensive reasoning-aware evaluation framework that combines structural metrics with semantic similarity metrics to assess internal coherence and validity of multi-step reasoning traces. II Related Work Quantum-Inspired Tensor Networks for LLMs: Recent advances in quantum-inspired modeling highlight the utility of tensor networks to enhance structured reasoning and drive architectural novelty. Kong et al. [ 13 ] propose a Quantum Weighted Tensor Hybrid Network (QWTHN) that combines quantum neural circuits with tensor representations to achieve superior parameter-efficient fine-tuning compared to traditional Low-Rank Adaptation (LoRA) [ 14 ] , demonstrating improved generalization. Likewise, the work by Aizpurua, Jahromi, Singh, and Orus [ 15 ] integrates tensor network disentanglers and matrix product operators into transformer blocks, effectively decomposing dense weight matrices into structured, entangled components that capture richer contextual dependencies. Complementing these architectural innovations, Chen et al. [ 16 ] propose QuanTA, a quantum-inspired tensor adaptation method that supports high-rank parameter updates using tensor operations inspired by quantum circuits. It enables expressive fine-tuning without additional inference overhead and outperforms LoRA variants on reasoning tasks. Ali, Delgado, and de Leceta [ 17 ] survey the practical adoption of quantum-inspired tensor network methods such as matrix product states MPS and PEPS in industrial and optimization contexts, highlighting their scalability and interpretability for AI applications. Krupp et al. [ 18 ] evaluate the quality of LLM-generated tips in quantum computing education, finding that such tips can match or outperform expert-created ones in terms of conceptual helpfulness. Their findings reinforce the need for coherence-aware evaluation metrics such as tensor network–based fidelity scoring. These bodies of research collectively establish a solid foundation for leveraging tensor network principles such as entanglement and fidelity as a means of improving both structure and reasoning in language models. RL with Non-Traditional Rewards: RL with LLMs has extended beyond the conventional supervised loss functions to incorporate non-traditional reward signals that reflect human preferences, factuality, and coherence. The foundational studies by Ziegler et al. [ 19 ] and Stiennon et al. [ 20 ] in RLHF introduce reward models trained on human comparisons to fine-tune LLMs using PPO, resulting in outputs more aligned with human preferences. Although RLHF is effective, it is limited by the scalability and sujectivity of human feedback. The study by Scheurer et al. [ 21 ] proposes replacing human feedback with an LLM-based feedback generated by the LLM trained using imitation learning. Bai et al. [ 22 ] propose Constitutional AI, a framework that fine-tunes LLMs using artificial intelligence (AI) generated feedback based on a predefined set of ethical principles that reflect human preferences. Both [ 21 ] and [ 22 ] demonstrate that LLMs can generate feedback on their own outputs, enabling scalable fine-tuning without relying on direct human supervision. Shmied et al. [ 23 ] highlight the ”knowing-doing” gap in LLMs where models exhibit correct reasoning trace, yet fail to act reliably on that knowledge. They propose RL fine-tuning on chain-of-thought outputs to align reasoning with action performance. Building on these advances, this study introduces a PEPS-based fidelity critic that evaluates the internal consistency of multi-step reasoning traces and provides a structured, scalable reward signal to guide RL (PPO) without relying on human annotations. Multi-Step Reasoning Evaluation: The evaluation of multi-step reasoning traces in LLMs extends beyond measuring final-answer accuracy. Comprehensive assessment requires capturing properties such as logical consistency, contextual relevance, and semantic fluency, which recent studies address through different techniques. Entailment-based approaches by Canburu et al. [ 24 ] assess groundedness and local validity by evaluating whether intermediate steps are logically supported by the input context. Logic-driven benchmarks introduced by Alfageeh et al. [ 25 ] evaluate reasoning behavior by mapping natural language prompts to formal logic representations. Ling et al. [ 26 ] propose a decompositional verification framework that decomposes chain-of-thought reasoning into stepwise natural language deductions, enabling logical validation of intermediate inference. While these approaches offer valuable insights, they are differ in scope, typically focusing on isolated properties such as factuality, validity, or structural consistency. The study by Lee and Hockenmaier [ 27 ] brings conceptual clarity to the evaluation space by proposing a taxonomy that categorizes reasoning quality into four dimensions: groundedness, validity, coherence, and utility. This study adopts the categorical taxonomy in its evaluation protocol to ensure that local step quality and global trace coherence are captured in a principled and annotation-free manner. III Methodology This study proposes a two-stage training pipeline to guide LLMs toward coherent multi-step reasoning. In the first stage, a quantum-inspired fidelity functional based on a PEPS tensor is trained to evaluate the structural coherence of reasoning traces. In the second stage, the pretrained fidelity functional is used as a reward signal to fine-tune a causal LLM using PPO-based RL. The two stages are illustrated in Figure 1 , and the following subsections formalize each of the stages. Fig. 1: Overview of the Quantum-Inspired Two-Stage Training Pipeline Stage 1: PEPS-based Fidelity Functional Training III-A Entangled Tensor Representations of Reasoning Traces Let \mathcal{X} and \mathcal{R} denote the space of natural language queries and multi-step reasoning traces, respectively. For a given input query x\in\mathcal{X} , the LLM generates a reasoning trace \mathbf{r}=(s_{1},s_{2},s_{3},\dots,s_{t})\in\mathcal{R} , where each s_{t} corresponds to a human interpretable intermediate reasoning step. The latent structure across these steps is formalized by embedding each s_{t} into a high-dimensional real Hilbert space \mathcal{H}\cong\mathbb{R}^{d} , where \mathcal{H} is the semantic embedding space and \mathbb{R}^{d} is the d -dimensional space of real-valued sentence embeddings. This mapping is performed using a sentence-level encoder \Phi:\mathcal{V}\rightarrow\mathcal{H} , where \mathcal{V} denotes the set of all valid token sequences over a fixed vocabulary. Each embedded step is denoted as r_{t}=\Phi(s_{t})\in\mathcal{H} . The full reasoning trace is lifted into a tensor product space that encodes the semantic representation of the full reasoning trace across multiple reasoning steps. Each reasoning step r_{t} is embedded into a local representation in \mathcal{H} , and the full trace, \psi , is mapped into the joint space \mathcal{H}^{\otimes T} as: \psi=\Phi(\mathbf{r})=r_{1}\otimes r_{2}\otimes\cdots\otimes r_{T}\in\mathcal{H}^{\otimes T} (1) The tensor product captures the joint space of reasoning steps but not their correlations. A PEPS-based contraction integrates local semantics into a globally entangled representation, with coherence assessed through a fidelity functional, \mathcal{F}_{\mathcal{T}} . III-B PEPS Tensor Networks for Structural Coherence The fidelity functional, \mathcal{F}_{\mathcal{T}} , is parameterized by a PEPS tensor, \mathcal{T}\in\mathbb{R}^{D\times D\times d\times D\times D} where the physical dimension d corresponds to the semantic embedding space of each reasoning step r_{t}\in\mathbb{R}^{d} , and the four bond dimensions D model latent pairwise interactions between adjacent reasoning steps. Shared across all positions in the reasoning trace, the PEPS tensor \mathcal{T} , serves as a latent contraction operator that integrates local semantics into a globally entangled structure, enabling coherence assessment through structured tensor contraction. Each semantic embedding r_{t}\in\mathbb{R}^{d} is incorporated into the PEPS-based tensor network through a mode-3 contraction (along semantic dimension) with the shared tensor \mathcal{T} , resulting in a structured chain of entangled representations across the reasoning steps: \mathcal{C}^{[t]}(\mathcal{T},r_{t})=\sum_{k=1}^{d}\mathcal{T}_{\alpha\beta k\gamma\delta}(r_{t})_{k} (2) where \mathcal{C}^{[t]}(\mathcal{T},r_{t})\in\mathbb{R}^{D\times D\times D\times D} is the contracted rank-4 PEPS tensor obtained upon the mode-3 contraction. The indices \alpha,\beta,\gamma,\delta\in\{1,\dots,D\} represent the four latent variables that mediate pairwise interactions between adjacent reasoning steps, and the index k\in\{1,\dots,d\} corresponds to the physical (semantic) dimension. For a given entangled reasoning trace, \psi , its fidelity, \mathcal{F}_{\mathcal{T}}(\psi) , is computed by accumulating the \ell_{2} -norms of the contracted tensors: \mathcal{F}_{\mathcal{T}}(\psi)=\prod_{t=1}^{T}\left\|\mathcal{C}^{[t]}(\mathcal{T},r_{t})\right\|_{2} (3) The multiplicative form of the fidelity functional, \mathcal{F}_{\mathcal{T}} , ensure that local misalignments propagate globally, enforcing strict coherence across reasoning steps. If all local contractions yield high norms, the fidelity remains high, reflecting a globally consistent trace. However, by penalizing inconsistent steps that yield a low norm, the corresponding term proportionally reduces the overall score. This multiplicative aggregation mechanism appropriately amplifies the effect of inconsistencies, suitably rewards consistent alignment, and guides the model toward globally structured reasoning. III-C Self-Supervised and Contrastive PEPS Tensor Training Prior to RL-based fine-tuning using PPO, the PEPS tensor \mathcal{T} is learned through a two-stage optimization process using a large corpus of structurally coherent, unlabeled multi-step reasoning traces, \mathcal{D}_{\text{traces}} . The first stage consists of self-supervised learning (SSL) wherein a given reasoning trace \mathbf{r}^{(i)}\in\mathcal{D}_{\text{traces}} is embedded as \psi^{(i)}=\Phi(\mathbf{r}^{(i)}) and its coherence is evaluated using the fidelity functional, \mathcal{F}_{\mathcal{T}}(\psi^{(i)}) . The training objective maximizes the log-fidelity, implemented as minimizing the negative log-fidelity loss, \mathcal{L}_{\text{SSL}} : \mathcal{L}_{\text{SSL}}(\mathcal{T})=-\frac{1}{N}\sum_{i=1}^{N}\log\mathcal{F}_{\mathcal{T}}(\psi^{(i)}+\varepsilon),\quad\varepsilon>0 (4) Log-fidelity is chosen for optimization rather than raw fidelity to improve numerical stability during training. The use of log-fidelity converts the multiplicative aggregation of step-wise contractions into an additive form, mitigating the risk of numerical underflow that might arise from multiplying multiple values less than one. As a result, it provides smoother gradients especially for longer traces. Although this formulation is used for numerical stability, it naturally admits a quantum-inspired interpretation that parallels energy-based formulations in quantum many-body systems. The negative log-fidelity can be viewed as an energy potential over the latent Hilbert space of reasoning traces, where structurally coherent traces correspond to low-energy configurations and incoherent traces are assigned higher energy values. After SSL, the PEPS tensor \mathcal{T} is further refined using contrastive fine-tuning to enhance its ability to distinguish structurally coherent reasoning traces from corrupted or incoherent ones. This stage improves the discriminative sensitivity of the fidelity functional \mathcal{F}_{\mathcal{T}} by explicitly optimizing it to assign higher scores to valid reasoning traces and lower scores to perturbed variants. Each original (coherent) reasoning trace from \mathcal{D}_{\text{traces}} is paired with a non-coherent counterpart generated by augmenting the original reasoning trace using random shuffling of steps along with the substitution of semantically unrelated statements. The contrastive objective minimizes the relative fidelity of the non-coherent trace compared to its coherent counterpart using the loss function \mathcal{L}_{\text{contrast}}(\mathcal{T}) , defined as: \mathcal{L}_{\text{contrast}}(\mathcal{T})=-\log\frac{\mathcal{F}_{\mathcal{T}}(\psi^{+})}{\mathcal{F}_{\mathcal{T}}(\psi^{+})+\mathcal{F}_{\mathcal{T}}(\psi^{-})} (5) where \psi^{+} and \psi^{-} are the entangled representations of original (coherent) trace and corrupted trace respectively. By explicitly contrasting between coherent and non-coherent reasoning traces, the contrastive learning complements the SSL objective to improve the overall fidelity critic before fine-tuning with PPO in Stage 2. Stage 2: Fidelity-Guided PPO Fine-Tuning After SSL, the PEPS tensor \mathcal{T} is fixed and used as a fidelity-based reward critic to fine-tune a causal LLM \pi_{\theta} using PPO. Given an input query x , the LLM samples a reasoning trace \psi\sim\pi_{\theta}(\cdot\mid x) and receives a composite reward, R(\psi) , that integrates structural coherence and novelty rewards. The composite reward is expressed as: R(\psi)=\lambda_{f}\cdot\log\mathcal{F}_{\mathcal{T}}(\psi)+\lambda_{r}\cdot\mathcal{H}_{\text{novelty}}(\psi),\quad\lambda_{f}+\lambda_{r}=1 (6) where: • \mathcal{F}_{\mathcal{T}}(\psi) (eq. 3 ) is the PEPS fidelity score measuring global structural coherence of the reasoning trace. • \mathcal{H}_{\text{novelty}}(\psi) is the n -gram-based novelty reward, given by the proportion of unique n -grams to the total number of n -grams, given by: \mathcal{H}_{\text{novelty}}(\psi)=\frac{\left|\left\{g_{i}\right\}_{i=1}^{K}\right|}{K} (7) where g_{i} is the i -th n -gram, and K=L-n+1 where L is the total number of tokens across the steps of the reasoning trace \psi . This composite reward is intentionally curated to reinforce complementary aspects of high-quality reasoning. The first term, \log\mathcal{F}_{\mathcal{T}}(\psi) , serves as the structural core of the reward, quantifying the global coherence of the reasoning trace to ensure that the sequence of reasoning steps adheres to entanglement-informed patterns of logical consistency learned in Stage 1. The second term, \mathcal{H}_{\text{novelty}}(\psi) , encourages informational diversity by rewarding reasoning traces that exhibit low semantic redundancy, thereby promoting progression and variation across the reasoning trace. Collectively, these two terms provide a well-calibrated reward that supports structural coherence, semantic relevance, and step-level diversity in a unified and interpretable manner. The LLM is fine-tuned using PPO which updates the LLM by maximizing a clipped surrogate objective: \displaystyle\mathcal{L}_{\text{PPO}}(\theta)=\mathbb{E}_{\psi}\Big[ \displaystyle\min\big(\rho(\psi)A(\psi), (8) \displaystyle\text{clip}(\rho(\psi),1-\epsilon,1+\epsilon)A(\psi)\big)\Big]+\beta\mathcal{H}[\pi_{\theta}] where \theta denotes the parameters of the LLM policy \pi_{\theta} , \mathbb{E}_{\psi} represents the expectation over sampled reasoning traces \psi\sim\pi_{\theta}(\cdot\mid x) , \rho(\psi)=\frac{\pi_{\theta}(\psi)}{\pi_{\theta_{\text{old}}}(\psi)} is the likelihood ratio between the current and previous policy, and A(\psi)=R(\psi)-\hat{V}(x) is the advantage function based on the difference between the obtained reward, R(\psi) , and a learned value baseline, \hat{V}(x) , approximated by an exponential running mean. The hyperparameter \epsilon controls the clipping range for the policy update, and \beta weights the contribution of the entropy term. The policy entropy is defined as: \mathcal{H}[\pi_{\theta}]=-\sum_{t}\sum_{w\in\mathcal{V}}\pi_{\theta}(w_{t}\mid h_{t})\log\pi_{\theta}(w_{t}\mid h_{t}) (9) where \mathcal{V} is the vocabulary and h_{t} denotes the decoder state at time step t . Two auxiliary terms are incorporated into the PPO objective to regularize optimization and prevent the model from collapsing: • Kullback-Leibler (KL) penalty, \mathcal{L}_{\text{KL}} , to ensure that the updated policy \pi_{\theta} does not deviate excessively from a reference policy \pi_{\text{ref}} , defined as: \mathcal{L}_{\text{KL}}=D_{\text{KL}}\big(\pi_{\theta}(\cdot\mid x)\,\|\,\pi_{\text{ref}}(\cdot\mid x)\big) (10) • Supervised auxiliary loss, \mathcal{L}_{\text{sup}} , to encourage the model to stay consistent with gold reasoning traces y^{*} by minimizing the negative log likelihood of the correct tokens, defined as: \mathcal{L}_{\text{sup}}=\mathbb{E}_{x,y^{*}}\left[-\sum_{t}\log\pi_{\theta}(y^{*}_{t}\mid y^{*}_{