Statistically Indistinguishable, Operationally Distinct: A Formal Barrier for Tabular Foundation Models

Tassilo Klein, Johannes Hoffart · 2026

Read Statistically Indistinguishable, Operationally Distinct: A Formal Barrier for Tabular Foundation Models by Tassilo Klein, Johannes Hoffart (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-29091.txt.

Introduction

Klein and Hoffart argue that foundation models for tables cannot judge whether data obey the rules of the system that produced them, and they make that claim testable. Their Operational Turing Test builds pairs of database states, one legal and one rule-violating, whose single-column and paired-column distributions match almost exactly. A classic lemma then bounds any classifier that sees only values: its error must sit near chance.

Three values-only baselines hit that bound exactly, scoring at chance level in pre-registered tests, and raw row access did not help. Exposing how values relate across rows closed most of the gap. A classifier fed by only seven executable rule-based audits was the sole one to reach perfect accuracy. Frontier language models fared worst: given schema, rules and state files, they classified almost no legal states as legal. The barrier, the authors conclude, is identifiability, not capacity.

Catalog record

CREATORTassilo Klein, Johannes Hoffart
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE28 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.29091

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Statistically Indistinguishable, Operationally Distinct: A Formal Barrier for Tabular Foundation Models Tassilo Klein, Johannes Hoffart Abstract Tabular foundation models cannot reason about data produced by running systems without access to the rules that govern them. We make this statement falsifiable. The Operational Turing Test (OTT) constructs pairs of legal and rule-violating database states whose 1 1 - and 2 2 -way column-value marginals match to a total variation of < 0.02 <0.02 ; Le Cam’s lemma then bounds any values-only classifier at ≥ 0.49 \geq 0.49 Bayes error. Three values-only baselines (XGBoost, TabICL, TabPFN) hit the bound exactly (accuracy 0.50 0.50 , pre-registered two one-sided tests (TOST) p < 0.002 p<0.002 ), raw row-level access does not help, exposing relational value consistency closes most of the gap, and only a classifier fed by seven executable rule-derived audits reaches 1.00 1.00 classification accuracy. In three matched 100 100 -state frontier large-language-model (LLM) runs, models given the schema, trigger source, rule tables, and state files classify at most 2 / 50 2/50 legal states as LEGAL; GPT-5.5 accepts 0 / 50 0/50 legal states even with higher reasoning effort and a Structured Query Language (SQL) executor. The access-ladder pattern also appears on a second schema with structurally distinct rule families (banking ledger: cross-row balance, cumulative aggregate). The barrier is identifiability, not capacity: scale, data, and richer features cannot cross it without operational grounding. 1 1 1 Code and data: github.com/SAP-samples/operational-turing-test . \icmlshowauthorstrue stored values joins groups rules audits full oracle Conceptual ladder ctrl RDB SQL oracle .50 .75 1.00 .50 .89 1.00 1.00 Accuracy Empirical evidence Figure 1 : Access ladder and empirical validation. Left: conceptual access levels. Right: representative evidence; the dashed line is chance. Leakage controls stay at chance; relational baselines (HistGB/RDB-PFN, 22 ) reach 0.89 but miss derivation; SQL audits derived from schema and code match the oracle. 1 Introduction Tabular foundation models have made rapid progress on benchmarks of single, self-contained tables. Production tables are different: application code writes rows, declarative constraints restrict values, procedural triggers enforce transitions, and business rules decide which states are legal. The training data available to models is therefore a projection of a running system: stored values remain, but much of the operational logic and procedure that generated them is left behind. This decoupling is severe in enterprise systems. Large schemas expose thousands of tables with technical identifiers whose semantics live outside the data definition language (DDL); column names such as BUKRS , MATNR , or VBAK-ERDAT are typical enough that automated column-name expansion is now studied as a pre-condition for working with such tables ( 24 ) . Recent cross-table ( 9 ) , relational ( 3 ; 19 ; 21 ) , and database-native ( 23 ) foundation models make the relational layer ingestible. The operational layer above it is less developed: value domains and referential integrity in schema text, plus procedural rules in triggers, validation scripts, and application code. This paper makes the resulting indistinguishability barrier falsifiable and tests whether source artifacts in a frontier LLM prompt close it. Reading the ladder. Figure 1 is not a leaderboard over model classes. The gray bar is a leakage-control sanity check: values-only models and raw-row variants should remain at chance when matched states preserve observable value statistics. Relational features add joins, key coverage, and group counts; rule-derived audits instead recompute legality predicates such as stored totals from line items, discounts, and taxes. The Operational Turing Test. We treat “operational context” concretely as the set of rules that decide whether a database state is legal: value-domain constraints, value-transformation logic, relational value consistency (referential integrity), and value-state-transition constraints. The Operational Turing Test (OTT) generalizes this idea to tests where statistical observations are insufficient without operational rules ( 11 ) ; here we instantiate it as a binary classification task on pairs of (legal, illegal) states where the rule-violating corruption preserves 1 - and 2 -way column-value marginals to within total variation distance (TV) \tau=0.02 . Le Cam’s lemma gives a Bayes error bound of \tfrac{1}{2}-\tfrac{\tau}{2}\geq 0.49 for any classifier whose input is restricted to such statistics ( Proposition 1 ). Values-only baselines from two broad architectural families, gradient-boosted trees and transformers, all reach accuracy statistically equivalent to chance under a pre-registered two one-sided tests (TOST) procedure for equivalence 2 2 2 The TOST equivalence margin \delta=0.02 and seed assignment were registered before evaluation. ( 12 , \alpha=0.05 ); seven executable rule-derived audits close the gap to 1.00 classification accuracy. Findings. 1. Operational grounding is information-theoretically necessary under marginal matching ( Proposition 1 ). 2. The barrier is architecture-, scale-, and representation-independent : three baselines and raw rows all hit the Le Cam bound ( Section 4 ). 3. Exposing relational value consistency closes most of the gap but cannot recover value-transformation logic ( Table 1 ). 4. Frontier reasoning models with full source artifacts in-prompt fail the test ( Section 5 ). 2 The Operational Turing Test Setting. Real-world tables do not stand alone; they are rows written by a running system, and the rules governing that system come in two kinds. Declarative rules (types, value-domain constraints, and relational value consistency such as referential integrity) are written into the schema text. Procedural rules (value-state-transition constraints, value-transformation logic, and per-aggregate cardinality bounds) live in trigger and application code. We write the union of both as a rule set \Pi . A dataset stripped of \Pi retains the values but loses much of the operational logic and procedure that produced them. Write \phi_{k}(S) for the vector of all k -way column-value marginals of database state S , and call a classifier values-only if its input is \phi_{k}(S) for some finite k (so it has no access to queries that read \Pi ). Let P_{\text{legal}} and P_{\text{illegal}} be the distributions over states from a legal data-generating process and from a corruption that violates a single rule of \Pi while preserving values elsewhere. Proposition 1 (Necessity of operational grounding) . If \|\phi_{k}(P_{\text{legal}})-\phi_{k}(P_{\text{illegal}})\|_{\mathrm{TV}}\leq\tau , then any values-only classifier has Bayes error \geq\tfrac{1}{2}-\tfrac{\tau}{2} . This is Le Cam’s lemma ( 13 ; 20 ) : two distributions within \tau in total variation cannot be distinguished better than \tfrac{1}{2}-\tfrac{\tau}{2} Bayes error. Our construction enforces \tau<0.02 ( Section 3 ), giving \geq 0.49 . The bound is an existence/impossibility result. The failure is identifiability-based, not capacity-based, so larger models, more data, and richer features cannot escape it; we verify the last empirically with row-level access ( Section 4 ). We do not claim every tabular foundation model fails on every task, only that where this condition holds, rule-guided queries are necessary. 3 Construction Schema. We instantiate the test on a three-table order-to-cash schema ( customers , orders , order_items ), the structural core of a standard enterprise sales process. The schema is deliberately minimal for transparency; the full schema definition together with the trigger and application-code rules is in Sections A.1 , A.2 and A.3 , and the construction generalises to any schema whose rules can be stated as executable code. Real-world linked business-table corpora at scale are already available ( 10 ) . Rules and corruptions. Four families of operational rules govern legal states; only some live in the schema text.

Read the complete text