High Accuracy, Less Talk ( HALT \mathrm{HALT} ): Reliable LLMs through Capability-Aligned Finetuning
Tim Franzmeyer, Archie Sravankumar, Lijuan Liu, Yuning Mao, Rui Hou, Sinong Wang, Jakob Foerster, Luke Zettlemoyer, Madian Khabsa · 2025
Read High Accuracy, Less Talk ( HALT \mathrm{HALT} ): Reliable LLMs through Capability-Aligned Finetuning by Tim Franzmeyer, Archie Sravankumar, Lijuan Liu, Yuning Mao, Rui Hou, Sinong Wang, Jakob Foerster, Luke Zettlemoyer, Madian Khabsa (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-04051.txt.
Introduction
Tim Franzmeyer and co-authors teach a language model to talk less in a 2025 paper. Such models answer each prompt in full, even when they lack the lore to be right. Those bold wrongs, called made-up facts, erode trust in the tool.
Their tune trains the model to give only what it can stand by and to hold back the rest. Training text is built by splitting the model own drafts into lone facts and steps, then marking which are false by true lore. The tuned model thus learns its own edge. Users gain a model that says less and is right more, a fair trade in high-stakes work.
Catalog record
| CREATOR | Tim Franzmeyer, Archie Sravankumar, Lijuan Liu, Yuning Mao, Rui Hou, Sinong Wang, Jakob Foerster, Luke Zettlemoyer, Madian Khabsa |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 52 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-04051
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
High Accuracy, Less Talk ( HALT \mathrm{HALT} ): Reliable LLMs through Capability-Aligned Finetuning Tim Franzmeyer, Archie Sravankumar, Lijuan Liu, Yuning Mao, Rui Hou, Sinong Wang, Jakob Foerster, Luke Zettlemoyer, Madian Khabsa Abstract Large Language Models (LLMs) currently respond to every prompt. However, they can produce incorrect answers when they lack knowledge or capability – a problem known as hallucination. We instead propose post-training an LLM to generate content only when confident in its correctness and to otherwise (partially) abstain. Specifically, our method, HALT \mathrm{HALT} , produces capability-aligned post-training data that encodes what the model can and cannot reliably generate. We generate this data by splitting responses of the pretrained LLM into factual fragments (atomic statements or reasoning steps), and use ground truth information to identify incorrect fragments. We achieve capability-aligned finetuning responses by either removing incorrect fragments or replacing them with “Unsure from Here” – according to a tunable threshold that allows practitioners to trade off response completeness and mean correctness of the response’s fragments. We finetune four open-source models for biography writing, mathematics, coding, and medicine with HALT \mathrm{HALT} for three different trade-off thresholds. HALT \mathrm{HALT} effectively trades off response completeness for correctness, increasing the mean correctness of response fragments by 15% on average, while resulting in a 4% improvement in the F1 score (mean of completeness and correctness of the response) compared to the relevant baselines. By tuning HALT \mathrm{HALT} for highest correctness, we train a single reliable Llama3-70B model with correctness increased from 51% to 87% across all four domains while maintaining 53% of the response completeness achieved with standard finetuning. † † affiliation: University of Oxford † † affiliation: Meta GenAI † † affiliation: Meta FAIR † † contribution: Work done at Meta † † date: June 3rd, 2025 † † correspondence: Tim Franzmeyer [email protected] , Madian Khabsa [email protected] 1 Introduction Most current language models attempt to respond to the majority of prompts, regardless of how complex they are or how much domain knowledge is required to answer them. While this behavior is desirable for creative tasks (e.g., poem writing), it is undesired when factual correctness is crucial. This phenomenon, often referred to as hallucination, poses risks in high-stakes applications such as medicine or law. We propose \mathrm{HALT} , which finetunes an LLM to only respond with information it is confident about, and to (partially) abstain otherwise. For example, when asked to write a person’s biography, the LLM would list only facts it is confident in and would omit others. Similarly, when presented with a complex math problem, it would show only the reasoning steps in which it has high confidence, terminating its response with “Unsure from here” if it cannot proceed – potentially forgoing a final answer. Naturally, restricting an LLM to provide only high-confidence information reduces the total number of correct statements, as it withholds any information it deems uncertain – even if some of that information might ultimately be correct. To balance this trade-off, \mathrm{HALT} allows practitioners to adjust a confidence threshold. This threshold controls whether the LLM should respond more eagerly (and risk occasional errors) or more conservatively (and risk omitting correct information). \mathrm{HALT} ’s key insight is to finetune the pretrained LLM only on correct content it is capable of generating , i.e., content within the bounds of the knowledge and reasoning capabilities obtained during pretraining, as we illustrate in Figure 1 . Further, \mathrm{HALT} finetunes the LLM to output “Unsure from here” when uncertain about the remainder of the generation. Figure 1: Comparison between Standard and HALT Finetuning for LLMs: Standard finetuning trains the LLM on responses that potentially exceed the pretrained LLM’s capabilities , which results in incorrect outputs at test time. HALT finetuning trains the model only on content within the pretrained LLM’s capability limits , omitting unknowns or replacing them with “Unsure from here” . HALT finetuning improves response correctness as the LLM is trained to generate responses according to its capabilities, which may result in partially incomplete responses when the pretrained LLM’s capabilities are insufficient. \mathrm{HALT} is motivated by the observation that LLMs possess internal calibration ( Tian et al. 2023b ; Kadavath et al. 2022 ) , suggesting they can produce responses aligned with their internal confidence in the relevant facts or reasoning steps. \mathrm{HALT} ’s approach of generating capability-aligned finetuning samples according to the pretrained LLM’s capabilities is based on recent work ( Lin et al. 2023 ; Zhou et al. 2024 ) which shows that LLMs do not acquire novel capabilities during finetuning but only learn to effectively utilize knowledge and reasoning capabilities obtained during pretraining. Moreover, \mathrm{HALT} ’s approach of finetuning only on content that the LLM is capable of generating is supported by recent work ( Gekhman et al. 2024 ; Kang et al. 2024 ) showing that finetuning LLMs on concepts unknown to them increases hallucination. While \mathrm{HALT} requires additional model generations and post-processing for each pretrained LLM it is applied to, we believe this is reasonable, as recent work ( Zhou et al. 2024 ) found that merely 1000 finetuning examples can be sufficient to finetune generally capable models. Unlike previous methods ( Yadkori et al. 2024 ; Farquhar et al. 2024 ; Cheng et al. 2024 ; Brahman et al. 2024 ; Feng et al. 2024 ; Zhang et al. 2024a ; Kang et al. 2024 ) , \mathrm{HALT} requires no additional computations at test time, as it does not rely on post-processing or sampling. Instead, it produces a finetuned model that attempts the best possible answer within its capability. Also, unlike earlier work, \mathrm{HALT} extends to prompts involving not only knowledge retrieval but also complex reasoning. Given a pretrained LLM and a finetuning dataset, for each prompt, HALT generates a response aligned to the capabilities of the LLM. For a given prompt, the \mathrm{HALT} pipeline (1) generates an initial response via few-shot prompting of the pretrained LLM, (2) splits it into factual fragments , (3) assesses correctness of individual fragments via an Evaluator LLM with access to ground truth information (e.g. the ground truth response), and (4) post-processes the fragments to arrive at the final \mathrm{HALT} finetuning response – which only contains fragments that the pretrained LLM is capable of generating. Figure 2 provides an example of the pipeline. For the second step of decomposing the response into fragments, we make the assumption that responses either consist of independent fragments or causally dependent fragments , i.e., fragments that build on top of each other in a logical sequence. This distinction is made for each prompt based on its domain, e.g., math samples are assumed to consist of causally dependent fragments, while biographies are assumed to consist of independent ones. For responses composed of independent fragments, we rely on prior work ( Song et al. 2024 ) and use a finetuned LLM to extract atomic statements, i.e., independently verifiable statements. For example, “Barack Obama was born in Hawaii” is an atomic statement that could be extracted from his biography. In step 3, the correctness of each fragment is assessed independently using the ground-truth-conditioned evaluator LLM (e.g. conditioned on the respective Wikipedia article). The \mathrm{HALT} finetuning response (4) is then composed of all correct fragments, or set to “Unsure