Iterative Audit Convergence in LLM-Managed Multi-Agent Systems: A Case Study in Prompt Engineering Quality Assurance
Elias Calboreanu \orcidicon · 2026
Read Iterative Audit Convergence in LLM-Managed Multi-Agent Systems: A Case Study in Prompt Engineering Quality Assurance by Elias Calboreanu \orcidicon (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-12280.txt.
Introduction
Elias Calboreanu reports a case study of auditing prompt specifications for multi-agent language-model systems. He audits AEGIS, a pipeline with a 7152-line specification surface, across nine rounds, surfacing 51 defects with counts falling from 15 to 0. He offers a seven-category taxonomy with coding rules, non-monotonic convergence tied to cascading edits and scope expansion, and a protocol. Two replications follow on a mini-specification: a four-vendor panel detects all five seeded defects in union, and a check gives Cohen's kappa of 0.80 on category and 0.46 on severity.
Catalog record
| CREATOR | Elias Calboreanu \orcidicon |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36922 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.12280
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Iterative Audit Convergence in LLM-Managed Multi-Agent Systems: A Case Study in Prompt Engineering Quality Assurance Elias Calboreanu \orcidicon Abstract Prompt specifications for multi-agent large language model (LLM) systems carry data contracts and integration logic across many interdependent files but are rarely subjected to structured-inspection rigor. This paper reports a single-system empirical case study of iterative, agent-driven auditing applied to AEGIS (Autonomous Engineering Governance and Intelligence System), a production seven-lane orchestration pipeline whose prompt-specification surface comprises approximately 7150 lines: 6907 across seven lane PROMPT.md files and a 245-line shared Ticket Contract. Nine sequential audit rounds, executed by Claude sub-agents using a checklist-driven walkthrough adapted from Weinberg and Freedman, surfaced 51 prompt-specification consistency defects, distinct from the 51 STRIDE-categorized adversarial code findings reported in the companion preprint. Per-round counts were 15, 8, 12, 2, 8, 1, 4, 1, and 0. We report a seven-category post-hoc defect taxonomy with explicit coding rules, observed non-monotonic convergence consistent with cascading edits and audit-scope expansion, and an audit protocol distilled from the study, with the final locked checklist released as a reproducibility appendix. Single-file review missed defect classes that were surfaced only by later expanded-scope rounds in this system. The same LLM family authored and audited the specifications; replication with dissimilar models and human reviewers is required before generalization. keywords prompt engineering; multi-agent systems; iterative auditing; quality assurance; large language models; specification consistency; software inspection; cross-document validation; agentic automation; configuration management † † firstpage: 1 † † volume: 1 † † issue: 1 † † articlenumber: 0 † † year: 2026 † † copyright-year: 2026 † † hreflink: https://doi.org/ † † titlecitation: Iterative Audit Convergence in LLM-Managed Multi-Agent Systems † † authornames: Elias Calboreanu † † authorcitation: Calboreanu, E. † † address: 1 Swift (North) AI Lab, The Swift Group, LLC, Maryland, USA 2 Capitol Technology University, Laurel, MD 20708, USA † † corresponding: Correspondence: [email protected] 1 Introduction 1.1 Motivation Prompt engineering for multi-agent systems presents a quality assurance challenge distinct from traditional software testing. When an LLM-orchestrated pipeline spans multiple autonomous lanes, each with its own behavioral specification, data contracts, and integration points, the prompt layer becomes the system’s source of truth. Errors in prompt specifications can propagate as runtime failures: wrong field names may break data handoffs, stale references can trigger invalid API calls, and missing labels can corrupt downstream classification. The empirical evidence presented in this paper is consistent with such failure modes but does not establish their relative frequency. Yet prompt specifications are rarely subjected to the rigor applied to source code. They are authored in natural language, reviewed informally, and updated incrementally. When a design change touches one document, ripple effects across other documents are easy to miss. Software inspection research has long established that structured review catches defects that testing alone may miss Fagan 1976 ; Boehm and Basili 2001 ; Weinberg and Freedman 1984 . This paper extends that principle to prompt specifications and reports the iteration required in one production specification surface. 1.2 Research Questions The research questions are stated descriptively rather than causally: • RQ1. What defect classes remained after the initial audit pass and surfaced only through later iterative expanded-scope rounds in this case study? • RQ2. What convergence behavior was observed, and how many rounds were required for this 7150-line, eight-document specification surface? 1.3 Contributions This paper offers three contributions, each scoped to the case study: • C1 (taxonomy). A seven-category post-hoc taxonomy of the 51 surfaced defects, with category definitions, coding rules, and one representative example per category (Section 3 and Appendix B ). Percentages are descriptive; no inferential claims are made. • C2 (observed convergence). Per-round defect counts and a category-by-round cross-tabulation showing observed non-monotonic intermediate behavior consistent with cascading edits and audit-scope expansion. We do not claim that the curve is predictable in advance; we report what was observed and offer a phase-based interpretation. • C3 (audit protocol distilled from the study). An audit protocol distilled from the case study, with the final locked checklist, prompt template, finding schema, convergence criterion, and context-loading chronology released as a reproducibility appendix (Appendix A and Appendix C ). We distinguish protocol used during the study , which co-evolved with discovery, from protocol recommended after the study . We do not claim single-pass full-scope review is universally insufficient — only that single-file review cannot, by construction, detect cross-file defects, and that one-pass termination would have left later-discovered defects unresolved under this study’s protocol. A single-pass full-scope control was not run. 1.4 Background and Related Work 1.4.1 Software Inspection and Structured Review Fagan-style code inspection Fagan 1976 established that structured review detects defects that testing alone may miss. Boehm and Basili’s “Software Defect Reduction Top 10 List” Boehm and Basili 2001 reaffirmed this in 2001 with broader empirical evidence, identifying peer reviews and inspections as cost-effective defect-reduction mechanisms across multiple project contexts. Weinberg and Freedman Weinberg and Freedman 1984 articulated lighter-weight walkthroughs and reviews, observing that effectiveness depends on checklist quality and reviewer expertise without requiring the full Fagan moderator/reader/recorder/inspector role separation. Our audit protocol adapts the walkthrough tradition rather than full Fagan inspection. We did not have multiple human inspectors with separate role assignments; we had one LLM family acting as inspector against a checklist authored by the same family. The locked checklist in Appendix A serves as the inspection guide, the sub-agent serves as the inspector, and the convergence criterion replaces the single-pass exit gate. The key limitation, that the inspector belongs to the same LLM family that authored the specifications, is documented in Section 4.5 . 1.4.2 Specification Drift, Aging, and Traceability Parnas Parnas 1994 identified software aging as the phenomenon by which documentation accuracy degrades as the system it describes evolves. Gotel and Finkelstein Gotel and Finkelstein 1994 analyzed the requirements traceability problem, finding that cross-artifact consistency is the most labor-intensive and error-prone aspect of requirements management. Our findings are consistent with both observations in the prompt-specification domain: 12 of the 51 defects (23.5%) were stale Jira references and five (9.8%) were cross-lane schema mismatches. 1.4.3 LLM Self-Evaluation and Self-Refinement Self-consistency Wang et al. 2023 samples multiple reasoning paths and selects the most consistent answer within a single context window. Self-Refine Madaan et al. 2023 iterates a critique-and-revise loop within a single LLM. Our work applies an iterative principle to specification artifacts spanning many persistent files across multiple sessions, not to single completions within one context. The persistence of file-based state changes the convergence dynamics: drift can accumulate across rounds in ways that intra-context refinement is not designed to address. 1.4.4 Recent LLM-Based SE and Agentic Quality Assurance He, Treude, and Lo He et al. 2024 survey LLM-based multi-agent systems across the software development lifecycle and identify robustness and synergy gaps. Cemri et al. Cemri et al. 2025 propose a 14-failure-mode taxonomy for multi-agent LLM systems with human-annotated inter-rater agreement ( \kappa=0.88 ); their work directly motivates Appendix B ’s coding rules. Lubos et al. Lubos et al. 2024 evaluate LLMs for requirements quality assurance against ISO 29148-style requirement-quality characteristics. Chen et al. Chen et al. 2025 argue that prompts are software artifacts requiring lifecycle methods (the promptware position). Patil Patil 2025 and the agentic-AI assessment-framework literature Akshathala et al. 2025 are consistent