Beyond Logprobs: A Multi-Signal Confidence Engine

for LLM-Based Document Field Extraction · 2026

Read Beyond Logprobs: A Multi-Signal Confidence Engine by for LLM-Based Document Field Extraction (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-24420.txt.

Introduction

This paper builds a multi-signal confidence engine for LLM-based document field extraction, beyond logprobs. The building builds. Their engine is multi-signal confidence. The extraction is LLM-based document field. It is AI. The building is beyond logprobs.

Catalog record

CREATORfor LLM-Based Document Field Extraction
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.24420

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Beyond Logprobs: A Multi-Signal Confidence Engine for LLM-Based Document Field Extraction Abstract 1. Introduction Automated extraction of structured fields from business documents is a high-value production task where incorrect extractions carry direct financial consequences. Recent large multimodal LLMs have substantially advanced document information extraction [ 1 , 2 ] , achieving strong accuracy on structured benchmarks spanning invoices, receipts, and forms. Yet production deployments require more than accuracy: they require knowing when to trust an extraction . A calibrated confidence score enables workflows to route low-confidence predictions to human review, reducing costly downstream errors while maximising automation throughput. This selective prediction problem has been well studied in classification settings [ 4 , 5 ] , but remains largely unsolved for LLM-based document extraction. The standard approach is to use token-level log-probabilities as a proxy for correctness. We show this fails substantially: on DocILE, a 55-field invoice benchmark where frontier LLMs fail on 26% of fields, logprob-mean achieves only 0.705 ROC AUC and degrades to an all-positive classifier at any practical threshold. Verbalized self-assessed confidence fares similarly (0.692 AUC). Even self-consistency across five sampled calls achieves only 0.744 AUC, marginally better, but with AURC 0.138, barely below logprob-mean’s 0.145, and at 5 \times the single-call API cost. The failure is structural. These methods measure confidence in the generated token sequence , but extraction errors in document processing are frequently caused by failures the model cannot observe: unreadable source material, ambiguous layouts, OCR noise. A frontier LLM confidently transcribing OCR noise produces high log-probabilities for a wrong answer. ExtractConf takes a different approach: construct a reliability signal from multiple heterogeneous sources, namely what the model generated, what the document’s image quality reveals, where the extraction was spatially located, what the document OCR says and whether two structurally distinct extraction calls agree. The design question that motivates the architecture is: what if we read the same document twice, in two ways that have different failure modes? Contributions 1. We introduce a dual-call Hunter – Mapper design where two structurally asymmetric LLM calls approach the same document from opposite output-structure orientations.(§ 3 ). 2. We fuse this cross-call signal with LLM-internal uncertainty, document quality, and spatial layout into a gradient-boosted confidence classifier, achieving 0.928 ROC AUC and reducing selective prediction risk by 70% over logprob-mean (§ 5 ). 3. We show that all three baselines fail as routing signals; self-consistency costs 5 \times per-field yet yields negligible AURC improvement; and document quality alone outperforms LLM-internal uncertainty alone. (§ 5 ). 4. Post-hoc recalibration (M7/M8) reduces ECE by up to 83%, establishing three deployment configurations for discrimination, calibration, and routing quality (§ 5 ). 5. Zero-shot Lasso recalibration on CORD achieves ECE - 89% and Brier - 43%, confirming the reliability signals generalise across document domains (§ 5.1 ). 2. Related Work Multi-signal KIE confidence HYCEDIS [ 13 ] is the closest predecessor to ExtractConf : it fuses CRNN OCR logits, LSTM lingual features, Graph-KV structural logits, and a VAE image anomaly score through a learned binary classifier to predict whether each KIE extraction is correct, reporting AUC and ECE on SROIE and CORD. ExtractConf inherits the multi-signal fusion paradigm but extends it to the LLM era in three directions: LLM logprobs and entropy replace CRNN logits; the asymmetric Hunter – Mapper design adds a cross-call signal structurally absent from HYCEDIS; and evaluation uses DocILE, a 55-field benchmark with 26% natural failure rate vs. SROIE’s 4%, providing richer negative supervision. Selective prediction and abstention ExtractConf is fundamentally a selective prediction system: it decides whether to act on an extraction or defer to human review, and is evaluated with AURC, a selective prediction metric. Selective prediction with neural networks was formalised by Geifman and El-Yaniv [ 4 ] , who showed that a coverage-accuracy trade-off is achievable by abstaining on low-confidence predictions. Corbière et al. [ 5 ] proposed learning a separate confidence score for abstention rather than relying on softmax probability, motivating our CatBoost meta-classifier design. The risk-coverage curve and AURC used throughout this paper follow the selective prediction evaluation framework of Geifman and El-Yaniv [ 6 ] . Conformal prediction [ 14 ] provides coverage guarantees post-hoc but produces set-valued outputs rather than calibrated scalar scores; ExtractConf is complementary, providing the graded probability estimates that enable continuous routing decisions. Ensemble disagreement and epistemic uncertainty The Hunter – Mapper design belongs to a broader family of disagreement-based uncertainty estimators. Query-by-committee [ 7 ] uses disagreement among ensemble members as an active learning signal; co-training [ 8 ] exploits disagreement between two views of the same instance to propagate labels. Lakshminarayanan et al. [ 9 ] show that deep ensembles produce better-calibrated uncertainty than single models via prediction disagreement. ExtractConf ’s Hunter – Mapper pair differs from these in a critical respect: the two calls are not trained differently nor independently initialised; they are the same model receiving structurally asymmetric prompts with different failure modes. The disagreement is therefore epistemic in origin, measuring whether the document provides sufficient grounding for both readings to converge, rather than reflecting parametric variance across ensemble members. LLM uncertainty quantification Self-consistency [ 28 ] samples multiple generations and uses majority agreement as confidence; we adopt this as baseline B3. Its routing utility is limited by score granularity: with 5 calls. BSDetector [ 17 ] and CONSTRUCT [ 18 ] extend consistency-based UQ to structured extraction outputs but are document-unaware; they use no OCR, spatial, or image quality signals. Semantic entropy [ 19 ] clusters semantically equivalent generations and measures cluster entropy; P(True) [ 20 ] uses LLM self-evaluation; and verbalized confidence [ 21 ] elicits numeric scores. These methods are included as model-only baselines (B1, B2). Semantic entropy is not directly applicable to structured field extraction, where correctness is determined by exact or near-exact string match against ground truth, not semantic equivalence between free-form generations. Multimodal grounding uncertainty ExtractConf ’s core insight, that extraction uncertainty should arise from document-side grounding instability rather than just token probability, connects to the emerging literature on multimodal hallucination. Li et al. [ 10 ] show that vision-language models hallucinate objects not present in images with high confidence, exactly the failure mode Hunter exhibits on absent fields. The document AI-specific variant of this problem, where OCR confidence serves as a proxy for grounding quality, is validated by ConfBERT [ 15 ] , which injects OCR confidence into BERT for post-OCR error detection. Calibration for structured prediction Jagannatha and Yu [ 22 ] benchmark temperature scaling, MC-Dropout, and isotonic regression for NER and relation extraction calibration, establishing isotonic regression as a strong post-hoc calibrator for structured NLP tasks, motivating our M7 design. Nixon et al. [ 23 ] provide the multi-bin ECE framework; Kull et al. [ 24 ] provide the Brier score decomposition into calibration and refinement components used to interpret our mapper contribution. 3. The ExtractConf Architecture Figure 1 : ExtractConf architecture. A document feeds two asymmetric LLM calls in parallel: a field-guided Hunter (blue) and a document-guided Mapper (orange). Both report extracted values, token logprobs, and the neighbouring text they attended to. Value agreement, neighbourhood text overlap, and spatial centroid divergence (from OCR bounding boxes) form the cross-call reliability signals. OCR confidence and image

Read the complete text