Calibrated Confidence Estimation for Tabular Question Answering
Lukas Voss · 2026
Read Calibrated Confidence Estimation for Tabular Question Answering by Lukas Voss (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-12491.txt.
Introduction
Lukas Voss compares five confidence estimation methods for tabular question answering. All models prove severely overconfident on structured data, far more than on text questions. Self-evaluation methods score clearly below perturbation methods on both benchmarks. His Multi-Format Agreement exploits format variation across Markdown, HTML, JSON, and CSV. It costs 20 percent less than sampling baselines. It cuts calibration error by 44 to 63 percent. Paired with self-consistency, it lifts AUROC markedly.
Catalog record
| CREATOR | Lukas Voss |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 68564 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.12491
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Calibrated Confidence Estimation for Tabular Question Answering Lukas Voss Abstract Large language models (LLMs) are increasingly deployed for tabular question answering, yet calibration on structured data is largely unstudied. This paper presents the first systematic comparison of five confidence estimation methods across five frontier LLMs and two tabular QA benchmarks. All models are severely overconfident (i.e., smooth ECE 0.35–0.64 versus 0.10–0.15 reported for textual QA). A consistent self-evaluation versus perturbation dichotomy replicates across both benchmarks and all four fully-covered models: self-evaluation methods (verbalized, P(True)) achieve AUROC 0.42–0.76, while perturbation methods (semantic entropy, self-consistency, and our Multi-Format Agreement) achieve AUROC 0.78–0.86. Per-model paired bootstrap tests reject the null at p < 0.001 p<0.001 after Holm-Bonferroni correction, and a 3-seed check on GPT-4o-mini gives a per-seed standard deviation of only ± 0.006 \pm 0.006 . The paper proposes Multi-Format Agreement (MFA), which exploits the lossless and deterministic serialization variation unique to structured data (i.e., Markdown, HTML, JSON, CSV) to estimate confidence at 20% lower API cost than sampling baselines. MFA reduces ECE by 44–63%, generalizes across all four models on TableBench (mean AUROC 0.80), and combines complementarily with sampling: an MFA + self-consistency ensemble lifts AUROC from 0.74 to 0.82. A secondary contribution, structure-aware recalibration , improves AUROC by +10 percentage points over standard post-hoc methods. 1 Introduction LLMs produce different answers when the same table is serialized as Markdown versus HTML versus JSON ( Sui et al. 2024 ; Singha et al. 2023 ) . Prior work has treated this format sensitivity as a problem to be solved. This paper proposes a different perspective: format sensitivity is an uncertainty signal that can be exploited. If the answer changes depending on how a table is presented, the model is not reasoning about the structure of the table; instead, the model is pattern-matching on surface tokens, and the answer should not be trusted. This insight is the starting point for a broader investigation. Large language models are increasingly deployed for tabular question answering in financial analysis ( Chen et al. 2021 ) , business intelligence ( Wu et al. 2025 ) , and data exploration. Yet a fundamental reliability question remains largely unaddressed: when an LLM answers a question about a table, does the reported confidence reflect the actual likelihood of correctness? Two large research communities have developed in parallel with minimal overlap. The calibration community has produced hundreds of papers that study LLM confidence ( Xiong et al. 2024 ; Ye et al. 2024 ; Kadavath et al. 2022 ; Tian et al. 2023 ; Geng et al. 2024 ) , predominantly evaluated on textual QA, commonsense, or biomedical domains. The tabular QA community has produced hundreds of benchmarks and methods ( Wu et al. 2025 ; Wang et al. 2024b ; Pasupat and Liang 2015 ; Li et al. 2024a ) , which report accuracy without systematic calibration analysis. Recent work has studied confidence for text-to-SQL generation ( Liu et al. 2025 ; Entezari Maleki et al. 2025 ; Ramachandran and Sarawagi 2024 ; Somov and Tutubalina 2025 ; Lee et al. 2025 ) , and a recent system, TabLaP ( Wang et al. 2025 ) , proposes a single trustworthiness mechanism for tabular QA. Yet no published study systematically compares multiple general-purpose confidence estimation methods and recalibration techniques on standard tabular QA benchmarks (see § 2 ). This gap matters because tabular tasks introduce uncertainty sources with no analog in text QA: • Schema linking ambiguity : which column does “revenue” refer to? • Aggregation errors : SUM versus COUNT confusion • Numerical precision : rounding, unit conversion, financial formatting • Format sensitivity : the same table serialized differently yields different answers ( Sui et al. 2024 ) • Null handling : missing values cause systematic, silent errors The paper presents three contributions: 1. The first systematic calibration study for tabular QA. The study evaluates five frontier LLMs on two benchmarks (WikiTableQuestions, TableBench) with five confidence estimation methods, which include the most-cited baselines (P(True) ( Kadavath et al. 2022 ) and semantic entropy ( Kuhn et al. 2023 ) ). A consistent self-evaluation versus perturbation dichotomy emerges: self-evaluation methods achieve AUROC between 0.42 and 0.76, while input and output perturbation methods achieve AUROC between 0.78 and 0.86. This pattern is consistent with prior findings on math ( Wang et al. 2024a ) , text-to-SQL ( Ramachandran and Sarawagi 2024 ; Ma and Zhao 2025 ) , and reasoning ( Huang et al. 2024 ) , yet it is the first systematic documentation in the tabular domain. Self-evaluation is not unusable in isolation: combination with consistency sampling ( Taubenfeld et al. 2025 ) or with MFA (§ 6 ) recovers complementary signal. 2. Multi-Format Agreement (MFA) , which is a domain-specific instantiation of the input-perturbation uncertainty paradigm ( Gao et al. 2024 ; Portillo Wightman et al. 2023 ) that exploits the lossless and deterministic format variation unique to structured data. MFA matches semantic entropy and self-consistency at 20% lower API cost (4 versus 5 calls per question), generalizes to TableBench at 50% ECE reduction on a much harder benchmark, and combines complementarily with sampling methods: an MFA + self-consistency ensemble lifts AUROC from 0.74 to 0.82 on Llama (§ 3 , § 6 ). 3. Table-structure-aware recalibration, which extends Platt scaling ( Platt 1999 ) with structural covariates that uniquely improve discrimination by +10 percentage points AUROC, unlike standard recalibration. Feature-importance analysis identifies query complexity and table size as the most informative covariates (§ 6 ). 2 Related Work 2.1 LLM calibration: extensive, but text-only LLM calibration has been studied extensively on textual benchmarks. Kadavath et al. 2022 showed that LLMs can partially distinguish known from unknown answers. Tian et al. 2023 evaluated calibration elicitation strategies on TriviaQA, SciQ, and TruthfulQA. Xiong et al. 2024 demonstrated systematic overconfidence with an average AUROC of only 62.7% for GPT-4. Huang et al. 2024 show that LLMs cannot self-correct reasoning without external feedback. The most comprehensive survey ( Geng et al. 2024 ) covers verbalized, logit-based, and consistency-based methods with no section on tabular reasoning. Taubenfeld et al. 2025 show that combination of P(True) with consistency sampling outperforms either alone. 2.2 Input-perturbation uncertainty estimation The principle of measuring agreement across perturbed inputs has been established by several works. SPUQ ( Gao et al. 2024 ) perturbs inputs via paraphrasing and dummy tokens, with 50% ECE reduction. BayesPE ( Tonolini et al. 2024 ) uses Bayesian weighted ensembles over semantically equivalent prompts. More recently, Reing et al. 2025 models prompt sensitivity as generalization error; CCPS ( Khanmohammadi et al. 2025 ) applies adversarial perturbations to hidden states; DiverseAgentEntropy ( AWS AI Labs 2025 ) generates knowledge-preserving query reformulations. MFA is a domain-specific instantiation of this paradigm, distinguished by lossless perturbation (format switches preserve 100% of semantic content), deterministic operation, and natural availability for structured data. The paper evaluates MFA against self-consistency and semantic entropy as a direct comparison of input-variation versus output-variation uncertainty. 2.3 Tabular QA: accurate, but uncalibrated Tabular QA benchmarks — WikiTableQuestions ( Pasupat and Liang 2015 ) , TableBench ( Wu et al. 2025 ) , BIRD-SQL ( Li et al. 2024a ) , FinQA ( Chen et al. 2021 )