A Universal Cliff and a Design Fingerprint: Cross-Section Defect Detection Under LLM Orchestration
Hiroki Fukui, M.D, Ph.D · 2026
Read A Universal Cliff and a Design Fingerprint: Cross-Section Defect Detection Under LLM Orchestration by Hiroki Fukui, M.D, Ph.D (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-26174.txt.
Introduction
Fukui asks what orchestration does to defects no lone worker can see. Production systems split requests across hidden worker agents that recombine one report. Fukui tests cross-section defects, contradictions between distant sections, all else fixed, models varied. Ten systems span five generations from one developer, plus five providers with distinct styles. Models finding such defects alone lose most of that ability under orchestration, detection falling by two-thirds or more. Scale and longer reasoning do not close the gap. Fallen models fail differently.
Catalog record
| CREATOR | Hiroki Fukui, M.D, Ph.D |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 65974 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.26174
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
A Universal Cliff and a Design Fingerprint: Cross-Section Defect Detection Under LLM Orchestration Hiroki Fukui, M.D, Ph.D Abstract Production language-model systems increasingly answer a single request by partitioning it across an invisible orchestration of worker agents and recomposing one integrated report. We ask what such orchestration does to a class of defect that no single worker is positioned to see: a contradiction that lives in the relation between two distant sections of a document. Holding the documents, the embedded defects, the orchestration mechanism, the scoring pipeline, and the random seed fixed, we vary only the model, across ten systems spanning five generations from one developer and five further providers from distinct alignment paradigms. Two layers separate. First, a detection cliff that is universal: every model able to find these cross-section defects under a single agent loses that ability under orchestration, with the detection rate falling by two-thirds or more from its single-agent value, across every provider and paradigm tested. The cliff is mechanism-derived — it follows from dividing a document among agents who each hold only a part — and is not closed by scale or by extended reasoning. Second, a difference in how models behave once they have fallen. Decomposing detection into a sensitivity and a reporting criterion (signal-detection theory) reveals that, among the six models whose discrimination exceeds chance, only one developer’s generations move along the criterion axis: as alignment is strengthened across five generations, the model misses fewer defects under orchestration and, at the same time, raises more false alarms on clean documents — two faces of a single criterion shift, not two independent changes. This shift is specific: it scales with generation within that developer (Cochran–Armitage trend, p < 0.001 at the error level) and is near-absent across the other providers, who hold at the silent end with false-positive rates at or near zero (a sevenfold pooled difference, Fisher p < 0.001). Reading the runs at the bottom of the cliff shows what the criterion shift produces there. In the most telling runs the missed defect is not out of view: the model’s private record reconstructs the document’s structural fault with full accuracy, and the integrated report nonetheless signs off on the soundness of the work, its concern spent — carefully — on the quality of the artifact and the fairness owed to an absent collaborator rather than on the defect. This behavior resists quantification: an automated judge does not score it stably (precision 17–50% across three prompt versions, all published) and a keyword rule cannot separate it from ordinary agreement, because what it asks them to find is not a property of any single text but a relation among the report, the miss, and what a competent reviewer would have flagged. We report that resistance as a finding. We release all runs, probes, defect keys, scorer prompts, and analysis scripts. For the production setting the implication is direct: an integrated report’s confidence is uninformative about defects that span the partition, the most carefully aligned systems are not the safest here in the way one would hope, and the cliff is structural — it will not be closed by better models, only by not arranging for the whole to be unseen. 1 Introduction A user who today asks a capable language-model system to review a contract, audit a codebase, or check a long compliance document is often no longer addressing a single model. The request is met by an orchestration: a controller decomposes the task, hands the pieces to worker agents, and recomposes their returns into one integrated report delivered in a single voice. The decomposition is usually invisible to the user, and increasingly it is the production default rather than a research configuration. The report arrives confident and whole, and reads as though one competent reader had held the entire document in view. Some defects survive this arrangement and some do not. A typo, a miscalculation, a clause that is wrong on its own terms — these are local: they sit inside a span that some worker was assigned, and a worker who reads that span can find them. But a different and consequential class of defect is not local. A contract whose liability cap in one section is silently voided by a carve-out in another; a specification whose interface promise is contradicted by an implementation constraint forty pages later; two clauses that cannot both hold — these defects are not in any one passage. They are constituted by the relation between two passages, and that relation is precisely what an invisible orchestration arranges for no single worker to hold. Each worker sees a part and reports faithfully on the part; the report composed from those returns inherits a confidence none of the parts could earn about the whole. This paper is about what happens to that class of defect under orchestration, and it asks two questions that the production setting tends to run together but that have to be kept apart. The first is structural: when a document is partitioned among agents who each see only a part, how much cross-section defect detection is lost, and is that loss a property of orchestration itself or an artifact of how a particular model was trained? The second is dispositional: once a model has lost the defect from view, how does it behave — does the loss show as a hesitation, a silence, a flagged uncertainty, or as a confident integrated report that the document is sound? The first question is about the terrain every model stands on; the second is about how a given model has been taught to stand once it is there. The two are independent — the same terrain admits different postures — and a design that holds the terrain fixed while varying only the model can separate them. We build exactly that design. A prior single-paradigm stage of this program established that cross-section detection collapses when one agent is replaced by an invisible orchestration of the same model. Here we hold the four documents, the sixteen embedded cross-section defects, the orchestration mechanism, the scoring pipeline, and the random seed fixed, and vary only the model — across ten systems: five successive generations from a single developer, which lets us read what strengthening one alignment program does as a dose, and five further systems from distinct alignment paradigms (a closed RLHF model, a less-restrictively aligned model, a large multimodal model, a reasoning-tuned model from a different national tradition, and an open-weights model), which lets us ask whether any effect we find is a property of one alignment regime or of orchestration as such. The generation axis turns alignment strength into something that can be read as a trend; the paradigm axis turns "is this specific to one developer?" into a question the data can answer rather than a matter of assertion. The structure of the paper follows the structure of the two questions, and we have tried to write each part in the vocabulary proper to it. The detection cliff (§3.1) is carried by measurement alone: detection rates under a single agent and under orchestration, across ten models, with no interpretation the numbers do not force. The difference in how models behave