Enhancing Reasoning Accuracy In Large Language Models During Inference Time
Vinay Sharma [email protected], Manish Jain [email protected] · 2026
Read Enhancing Reasoning Accuracy In Large Language Models During Inference Time by Vinay Sharma [email protected], Manish Jain [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-21301.txt.
Introduction
Sharma and Jain compare three inference-time strategies for improving language model reasoning, with no extra training. Self-consistency samples the model many times and takes the most frequent answer. Dual-model agreement trusts only reasoning that two models share. Self-reflection has the model critique and revise its own work. All use chain-of-thought prompting. Self-consistency with nucleus sampling gives the largest gains, about nine to fifteen points over greedy decoding. Dual-model checks suit higher risk uses. Self-reflection adds only marginal gains for smaller models.
Catalog record
| CREATOR | Vinay Sharma [email protected], Manish Jain [email protected] |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 16751 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.21301
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Enhancing Reasoning Accuracy In Large Language Models During Inference Time Vinay Sharma [email protected], Manish Jain [email protected] Abstract Large Language Models (LLMs) often exhibit strong linguistic abilities while remaining unreliable on multi-step reasoning tasks, particularly when deployed without additional training or fine-tuning. In this work, we study inference-time techniques to improve the reasoning accuracy of LLMs. We systematically evaluate three classes of inference-time strategies: (i) self-consistency via stochastic decoding, where the model is sampled multiple times using controlled temperature and nucleus sampling and the most frequent final answer is selected; (ii) dual-model reasoning agreement, where outputs from two independent models are compared and only consistent reasoning traces are trusted; and (iii) self-reflection, where the model critiques and revises its own reasoning. Across all evaluated methods, we employ Chain-of-Thought (CoT) [ 5 ] prompting to elicit explicit intermediate reasoning steps before generating final answers. In this work, we provide a controlled comparative evaluation across three inference-time strategies under identical prompting and verification settings. Our experiments on LLM [ 1 ] show that self-consistency with nucleus sampling and controlled temperature value yields the substantial gains, achieving a 9%–15% absolute improvement in accuracy over greedy single-pass decoding, well-suited for low-risk domains, offering meaningful gains with minimal overhead. The dual-model approach provides additional confirmation for model reasoning steps thus more appropriate for moderate-risk domains, where higher reliability justifies additional compute. Self-reflection offers only marginal improvements, suggesting limited effectiveness for smaller non-reasoning models at inference time. 1 1 footnotetext: * Equal contribution 1 Introduction Large Language Models (LLMs) [ 1 ] have demonstrated impressive performance across a wide range of natural language tasks, including text generation, summarization, translation, question answering, etc. Their success is largely driven by large-scale pretraining on diverse corpora, enabling strong linguistic fluency and pattern recognition. However, despite these capabilities, LLMs often struggle with tasks that require reliable multi-step reasoning, logical consistency, or structured problem solving. Improving reasoning performance in LLMs has therefore become an active area of research. Prior work has explored a variety of approaches, including architectural modifications, specialized training objectives, reinforcement learning [ 3 ] , and instruction tuning. While effective, such methods typically require access to model weights, additional training data, or substantial computational resources. This motivates the study of inference-time techniques that aim to improve reasoning reliability without modifying model parameters. Inference-time strategies operate solely at the level of prompting, decoding, or output aggregation. Recent studies have shown that sampling-based methods, such as self-consistency [ 6 ] , can improve reasoning accuracy by exploiting diversity in model outputs. Other approaches leverage multiple models to cross-validate reasoning, or encourage models to critique and revise their own outputs through self-reflection [ 4 ] . In this work, we present a systematic empirical study of inference-time reasoning techniques for improving the accuracy of LLMs on multi-step reasoning tasks. 2 Dataset Details Logical Reasoning Improvement Dataset: All inference-time reasoning strategies were evaluated using the Logical Reasoning Improvement Dataset, publicly available on huggingface. The dataset was curated and released by garage-bAInd/Open-Platypus [ 2 ] . This dataset is particularly appropriate for our experimental setting for several reasons. 1. Dataset ’Input’ text columns contains data from diverse domains like Science, Mathematics, Coding etc. thus allowing us to measure the effectiveness of inference-time techniques independently of specific domains. 2. The problems are deterministic and have unambiguous ground-truth answers, which enables reliable quantitative evaluation of accuracy improvements across decoding and verification strategies. Dataset Link: https://huggingface.co/datasets/garage-bAInd/Open-Platypus Dataset Overview: The dataset is organized into several columns, each serving a specific purpose: input: The input text or question that requires logical reasoning. This column provides the initial statement or problem that needs solving. output: The correct answer or solution to the logical reasoning question. This column contains the expected outcome or response. instruction: Additional instructions or guidelines for solving the logical reasoning question. This column provides any specific guidance or steps required to arrive at the correct answer. data source: The source or origin of the logical reasoning question. This column specifies where the question was obtained from. 3 Methodology 3.1 Self Consistency + Controlled Temperature and Nucleus Sampling Technique Overview: Self-consistency is an inference-time strategy designed to improve the reliability of Large Language Models (LLMs) on reasoning-intensive tasks without modifying model parameters or requiring additional training. Instead of relying on a single deterministic generation, the model is sampled multiple times under controlled stochastic decoding, and the final prediction is obtained by aggregating these independent reasoning paths. In our implementation, self-consistency is combined with controlled temperature sampling and nucleus (top-p) sampling to encourage diverse yet semantically plausible reasoning trajectories. Specifically, the same question is posed to the model multiple times (n=6), each time allowing stochastic variation in token generation. Each run produces a full step-by-step reasoning trace followed by a final answer. These final answers are then extracted and then using a majority voting mechanism the most consistent answer across samples is determined. This approach operationalizes the hypothesis that correct reasoning paths are more likely to recur across multiple stochastic generations than incorrect ones, even when individual generations are imperfect. Controlled Stochastic Decoding: To induce meaningful diversity across reasoning paths, we configure the reasoning model with a moderate temperature (T=0.8) and nucleus sampling (top p=0.9). Temperature controls the randomness of token selection by flattening the output distribution, while nucleus sampling restricts token choices to the smallest set whose cumulative probability mass exceeds p. Together, these mechanisms ensure that: 1. The model explores alternative reasoning trajectories rather than repeating a single deterministic path. 2. Generated responses remain coherent and grounded, avoiding degenerate or low-probability outputs. Answer Extraction and Majority Voting: Each reasoning output contains both intermediate steps and a final answer. To ensure fair aggregation, we employ a low-temperature model (T=0.1) to extract only the final answer from each generated response. This normalization step removes variability introduced by formatting, verbosity, or reasoning style. Once extracted, the candidate answers are passed to a majority voting module, implemented as an LLM-based judge operating under near-deterministic settings. Rather than performing a simple string match, this judge selects the most semantically consistent answer across candidates. This design choice improves robustness to minor lexical variations (e.g., “2” vs. “two”) and aligns better with human notions of answer equivalence. Verification and Evaluation: For evaluation purposes, the final aggregated answer is compared against the ground-truth label using an independent verifier model. Figure 1: Self Consistency With Controlled Temperature and Nucleus Sampling 3.2 Dual-Model Reasoning with Cross-Model Verification Technique Overview: Dual-Model Reasoning is an inference-time strategy that improves reasoning reliability by cross-validating solutions generated by two independent language models with different architectures, training data, and deployment characteristics. In our implementation, two models are prompted with the same reasoning instruction to independently solve the same problem. Each model produces a structured, step-by-step solution along with a final answer. A independent verifier model then evaluates whether both reasoning outputs reach the same correct conclusion. Agreement between models is treated as a strong signal of correctness, while disagreement flags the response as unreliable. This approach is motivated by the observation that different models tend to fail in different ways, making cross-model agreement a powerful indicator of reasoning correctness. Cross-Model Agreement Verification: After both models generate their solutions, a verifier model is