CounterRefine: Answer-Conditioned Counterevidence Retrieval for Inference-Time Knowledge Repair in Factual Question Answering

Tianyi Huang, Ying Kai Deng · 2026

Read CounterRefine: Answer-Conditioned Counterevidence Retrieval for Inference-Time Knowledge Repair in Factual Question Answering by Tianyi Huang, Ying Kai Deng (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-16091.txt.

Introduction

Huang and Deng repair wrong QA answers by hunting counterevidence at run time. Many factual errors come from commitment, not missing access. Their CounterRefine layer treats the first answer as a testable guess. Answer-shaped queries fetch evidence for and against the draft. A guarded step then keeps or revises the answer. Revisions pass only through strict deterministic validation checks. SimpleQA gains reach 5.8 correct-rate points over one-pass RAG. Only 5.6 percent of outputs change in the full trace. Helpful flips outnumber harmful ones by 180 to 8.

Catalog record

CREATORTianyi Huang, Ying Kai Deng
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE24603
LICENSEhttp://creativecommons.org/licenses/by-sa/4.0/
RIGHTSCC BY SA
SOURCEarXiv

Source: https://arxiv.org/abs/2603.16091

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

CounterRefine: Answer-Conditioned Counterevidence Retrieval for Inference-Time Knowledge Repair in Factual Question Answering Tianyi Huang, Ying Kai Deng Abstract In factual question answering, many errors are not failures of access but failures of commitment: the system retrieves relevant evidence, yet still settles on the wrong answer. We present CounterRefine , a lightweight inference-time repair layer for retrieval-grounded QA. CounterRefine first produces a short answer from retrieved evidence, then gathers additional support and conflicting evidence with follow-up queries conditioned on that draft answer, and finally applies a restricted refinement step that outputs either KEEP or REVISE , with proposed revisions accepted only if they pass deterministic validation. In effect, CounterRefine turns retrieval into a mechanism for testing a provisional answer rather than merely collecting more context. On the full SimpleQA benchmark, CounterRefine improves a matched GPT-5 Baseline-RAG by 5.8 points and reaches a 73.1% correct rate, while exceeding the reported one-shot GPT-5.4 score by roughly 40 points. These findings suggest a simple but important direction for knowledgeable foundation models: beyond accessing evidence, they should also be able to use that evidence to reconsider and, when necessary, repair their own answers. 1 Introduction Large language models are increasingly used as knowledge interfaces, but a central difficulty remains unresolved: a model can retrieve relevant material and still commit to the wrong fact. In short-form factual QA, these errors are unforgiving. A wrong year, nearby entity, or almost-right title is still fully incorrect. This makes factual QA a useful setting for studying a broader question that matters for knowledgeable foundation models: if some knowledge is wrong, can we repair it at inference time rather than only during pre-training or parameter editing? Retrieval-augmented generation (RAG) improves factuality by grounding generation in external evidence rather than relying only on parametric memory ( 7 ; 6 ) . Yet retrieval alone does not eliminate candidate-selection errors. A first-pass retriever is optimized for topic relevance , not necessarily for candidate discrimination . However, once a draft answer exists, the retrieval problem changes. The most useful next query is often not the original question, but the question conditioned on that candidate answer. If the draft year is wrong, adding that year to the query can yield a snippet that directly falsifies it. If the draft entity is too broad or ambiguous, answer-conditioned re-querying can reveal a more discriminative title, list entry, or sentence. We build on this observation with CounterRefine , a retrieval-based short-answer system that first drafts an answer from web snippets and then performs one answer-conditioned counterevidence pass. The second stage is intentionally narrow. It is not tasked with solving the question from scratch. Instead, it decides whether to KEEP the draft or REVISE it in light of additional evidence, and a deterministic validator blocks unsupported or ill-formed rewrites. The result is a simple repair layer that can sit on top of an existing retrieval pipeline. Our empirical evidence is centered on full-benchmark official evaluation. On the full 4,326-question SimpleQA benchmark with the official grader ( 20 ; 14 ) , CounterRefine improves a matched Baseline-RAG from 63.7% to 67.7% correct and from 64.1% to 68.1% F1 on Claude Sonnet 4.6. On GPT-5, correct rate improves from 67.3% to 73.1% and F1 from 58.6% to 72.1%. We also evaluate transfer on 100- and 300-example slices from HotpotQA distractor development ( 21 ) , where CounterRefine improves exact match on both Claude Sonnet 4.6 and GPT-5. Qualitative examination shows that most successful revisions repair entity confusions, dates, and numeric values, while the remaining failures are dominated by relation confusion and event mismatch. Our contributions are: (1) we formalize answer-conditioned counterevidence retrieval as a simple keep-or-revise layer for factual QA; (2) we show that this layer improves over a matched retrieval baseline under the official SimpleQA evaluation pipeline and also transfers to HotpotQA exact match; and (3) we provide a targeted qualitative analysis of where the method helps and where its remaining failure modes arise. 2 Related Work Retrieval-grounded factual QA. Retrieval-augmented generation improves knowledge-intensive NLP by grounding generation in external documents ( 7 ) . Dense Passage Retrieval further improved open-domain QA with learned neural retrieval ( 6 ) . More recent work has explored adaptive retrieval and critique, for example by teaching models when to retrieve and how to reflect on retrieved evidence ( 2 ) . Other work has targeted robustness to noisy retrieval ( 22 ) or refreshed model knowledge through search-engine augmentation ( 17 ) . Our method differs in focus: rather than redesigning the retriever or the full RAG stack, we add a minimal second retrieval phase to stress-test the first answer. Verification and self-correction. A broad line of work improves factuality by checking or revising an initial answer. Chain-of-Verification generates verification questions before producing a final answer ( 3 ) . CRITIC uses tool-interactive critique to validate and amend outputs ( 5 ) . RARR retrieves evidence and revises unsupported claims while trying to preserve the original text ( 4 ) . SelfCheckGPT detects likely hallucinations through disagreement across samples ( 9 ) . CounterRefine is closest in spirit to this family, but it is narrower and cheaper: one extra retrieval pass, one extra model call, and a constrained short-answer decision rather than broad long-form editing. Knowledge repair and model editing. Wrong knowledge can also be addressed by editing model parameters directly. ROME and MEMIT show that factual associations in language models can be modified through targeted weight updates ( 10 ; 11 ) . Those methods operate on the model’s stored knowledge itself. CounterRefine is complementary: it performs answer-level repair at inference time instead of changing model parameters, using external evidence to challenge and revise a candidate answer. Benchmarks and conflicting evidence. Factuality has been studied through claim verification datasets such as FEVER ( 16 ) , factual QA benchmarks such as TruthfulQA ( 8 ) , long-form factuality metrics such as FACTSCORE ( 12 ) , and broader surveys of LLM factuality ( 19 ) . SimpleQA is particularly relevant here because it isolates short-form factual precision in a benchmark of 4,326 questions with a single indisputable answer ( 20 ) . Recent work has also emphasized that retrieval errors are not limited to irrelevant documents; systems often face ambiguous, noisy, or conflicting evidence ( 18 ) . Our setting is simpler than that literature’s multi-document conflict benchmarks, but the motivating issue is related: even in short-answer QA, retrieved snippets can weakly support multiple nearby candidates. Answer-conditioned re-querying is one lightweight way to make those conflicts visible before finalizing an answer. 3 Method Figure 1: Schematic overview of CounterRefine . Starting from a draft answer produced by Baseline-RAG , CounterRefine collects additional evidence intended to support, check, or contradict that candidate, evaluates the draft answer against the resulting evidence set, and then passes the result to a constrained keep-vs.-revise gate. The final answer is either the original draft or a revised candidate accepted under evidence-based validation using the same evidence set. For readability, the figure abstracts away some implementation details from the text (question-type-conditioned query selection and per-query snippet aggregation) but it is aligned with the three-stage pipeline described in Section 3. 3.1 Overview We refer to the first-pass retrieval system

Read the complete text