Knowing When to Quit: A Principled Framework for Dynamic Abstention in LLM Reasoning

Hen Davidov, Nachshon Cohen, Oren Kalinsky, Yaron Fairstein, Guy Kushilevitz, Ram Yazdi, Patrick Rebeschini · 2026

Read Knowing When to Quit: A Principled Framework for Dynamic Abstention in LLM Reasoning by Hen Davidov, Nachshon Cohen, Oren Kalinsky, Yaron Fairstein, Guy Kushilevitz, Ram Yazdi, Patrick Rebeschini (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-18419.txt.

Introduction

Davidov and six colleagues tackle wasted compute from long, wrong chain-of-thought answers in language models. Abstention withholds outputs likely to be wrong, but most methods decide only before or after writing. The authors define mid-writing quitting as a clear action in penalty-based reinforcement learning, with a reward knob trading compute against information. Abstaining when value falls below that knob beats natural baselines by proof, and tests on math reasoning and toxic-output dodging confirm better accuracy when the model chooses to answer.

Catalog record

CREATORHen Davidov, Nachshon Cohen, Oren Kalinsky, Yaron Fairstein, Guy Kushilevitz, Ram Yazdi, Patrick Rebeschini
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE106247
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.18419

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Knowing When to Quit: A Principled Framework for Dynamic Abstention in LLM Reasoning Hen Davidov, Nachshon Cohen, Oren Kalinsky, Yaron Fairstein, Guy Kushilevitz, Ram Yazdi, Patrick Rebeschini Abstract LLMs utilizing chain-of-thought reasoning often waste substantial compute by producing long, incorrect responses. Abstention can mitigate this by withholding outputs unlikely to be correct. While most abstention methods decide to withhold outputs before or after generation, dynamic mid-generation abstention considers early termination of unpromising reasoning traces at each token position. Prior work has explored empirical variants of this idea, but principled guidance for the abstention rule remains lacking. We present a formal analysis of dynamic abstention for LLMs, modeling abstention as an explicit action within a regularized reinforcement learning framework. An abstention reward parameter controls the trade-off between compute and information. We show that abstaining when the value function falls below this reward strictly outperforms natural baselines under general conditions. We further derive a principled and efficient method to approximate the value function. Empirical results on mathematical reasoning and toxicity avoidance tasks support our theory and demonstrate improved selective accuracy over existing methods. 2 1 Introduction LLMs have demonstrated remarkable proficiency across a broad spectrum of reasoning-intensive domains, including mathematical problem solving, scientific question answering, program synthesis, and symbolic logic ( 30 ; 45 ) . A primary catalyst for these advancements is their capacity to generate long-form natural language responses that explicitly articulate intermediate reasoning steps, a paradigm best exemplified by Chain-of-Thought (CoT) prompting ( 21 ; 43 ) . However, while such verbose reasoning substantially enhances task performance, it increases the computational cost of inference. Crucially, this computational expenditure does not guarantee accuracy. Models frequently generate incorrect responses while still incurring the full latency and cost of long-form generation. This inefficiency is compounded by the empirical observation that incorrect reasoning traces tend to be longer than correct ones ( 17 ; 49 ) . This creates two distinct problems: incorrect answers are delivered to users, and significant computational resources are wasted generating them. A natural solution is to allow abstention: withholding low-quality outputs or deferring to a fallback mechanism such as a human expert or a more capable model ( 44 ) . The landscape of abstention methods (surveyed more broadly in Appendix A ) can be understood through the lens of when the abstention decision is made, and how well each approach addresses the two problems above. At one extreme lies output processing : deciding whether to abstain after the complete response has been generated ( 20 ; 22 ; 5 ) . We focus on reasoning tasks with verifiable rewards: domains such as mathematical problem-solving, code generation, and symbolic reasoning where correctness can be checked deterministically. In these settings, the challenge is not whether we can identify errors, but whether we can do so before expending resources on full CoT generation. At the opposite extreme, input processing attempts to predict failure based solely on the prompt ( 20 ; 47 ; 7 ) . This approach addresses the efficiency problem effectively - if abstention occurs, no computation is wasted. However, input processing may have limited accuracy in selecting which prompts to abstain from answering ( 20 ) . Even under deterministic decoding, the prompt alone oftentimes provides little information about which specific reasoning path the model will take or where it might fail. This limitation is compounded when LLMs are deployed with stochastic sampling, a common practice that improves performance on reasoning tasks ( 15 ) . Recently, 2 and 46 proposed observing partial generations and making abstention decisions at fixed token positions . While observing part of the output improves upon input-only methods, the point at which a trace’s fate becomes clear varies across problems. 46 address this with an empirical dynamic rule that can terminate generation at varying token positions. While their technique demonstrated empirical promise, it requires segmenting traces into reasoning chunks and using an external model to label intermediate answer correctness. However, a correct intermediate answer may later be revised incorrectly, and vice versa. What matters for abstention is whether the final answer will be correct. To directly target final-answer correctness, we work within the KL-regularized reinforcement learning framework ( 18 ; 50 ) . Since correctness is verified only for the final answer, we model rewards as sparse and binary. The reward for a finalized answer is 1 if it is correct, and 0 otherwise. To accommodate abstention, we augment the action space with an abstention token \bot and associate it with an abstention reward r_{\bot} . This r_{\bot} quantifies the utility of abstention, whether that means deferral to a human expert or routing to a more capable model. Within this framework, we analyze a natural decision rule, dynamic value-thresholding . We abstain when the value function falls below r_{\bot} . In our sparse reward setting, the value function reduces to the expected terminal reward from the current state, giving it a natural interpretation as the model’s confidence in eventual success. If this falls below the fallback reward, continuing is wasteful. This threshold r_{\bot} gives practitioners direct control over the abstention rate: lower r_{\bot} yields higher selective accuracy, while higher r_{\bot} yields greater compute savings. We formalize when and why this rule outperforms alternatives. Throughout, we say a policy dominates another policy if it achieves at least the same expected reward, and it strictly dominates a policy if it achieves strictly higher reward. We obtain the following results, assuming perfect knowledge of the value function: • Dominance over no abstention (Proposition 4.2 ): Given any base LLM, dynamic value-thresholding dominates non-abstaining. it strictly dominates if abstention occurs with positive probability. • Dominance over fixed-position abstention (Corollaries 4.5 and 4.6 ): Given any base LLM, dynamic value-thresholding dominates input processing based on value-thresholding. Further, it dominates fixed-position abstention at position k>1 when low-value states cannot recover before k . Finally, dominance is strict if the dynamic rule has a positive probability of abstaining after the k -th token position. • Optimality (Proposition 4.7 ): Given a base LLM that dominates all non-abstaining LLMs, and assuming zero KL regularization, dynamic value-thresholding dominates all abstention-enabled policies. • Improvement bound (Proposition 4.9 ): Given any base LLM, and assuming zero KL regularization and Lipschitz continuity, dynamic abstention improves over no abstention at a rate that is upper-bounded linearly in the abstention rate. We address the estimation of the value function in Section 5 . We show that for binary rewards and zero KL regularization, the value function equals the probability of trajectory success conditioned on the current state. As a result, parallel binary cross-entropy training at all non-terminal token positions recovers this quantity under realizability, as proven by Proposition B.2 . In Appendix B.8 we extend this result to general bounded rewards using an MSE loss. We utilize the binary cross entropy loss to train a two-layer MLP probe on the LLM’s hidden states. Compared to the 7B and 4B-parameter models tested in Section 6 , the two-layer MLP adds roughly 10^{4} parameters, making probe inference and training overhead negligible. In Section 6 , we measure the performance of dynamic value-thresholding compared to natural baselines, examine robustness to imperfect value estimation, show cross-dataset generalization of our value

Read the complete text