OrgForge: A Multi-Agent Simulation Framework for Verifiable Synthetic Corporate Corpora Separating Fact Control from Prose Generation in RAG Evaluation Benchmarks
Jeffrey Flynt · 2026
Read OrgForge: A Multi-Agent Simulation Framework for Verifiable Synthetic Corporate Corpora Separating Fact Control from Prose Generation in RAG Evaluation Benchmarks by Jeffrey Flynt (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-14997.txt.
Introduction
Flynt presents OrgForge, a multi-agent framework that builds synthetic corporate document sets with checkable ground truth. A deterministic Python engine tracks events, while language models write only the surface prose. Simulated staff changes, incidents, and customer events create fifteen linked artifact types tied to one event log. The system also models support escalation, deal risk, and invoices. In tests over ten incidents, fidelity to ground truth rose by 0.46 over chained language-model baselines.
Catalog record
| CREATOR | Jeffrey Flynt |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 53520 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.14997
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
OrgForge: A Multi-Agent Simulation Framework for Verifiable Synthetic Corporate Corpora Separating Fact Control from Prose Generation in RAG Evaluation Benchmarks Jeffrey Flynt Abstract Evaluating retrieval-augmented generation (RAG) pipelines requires corpora where ground truth is knowable, temporally structured, and cross-artifact—properties that real-world datasets rarely provide cleanly. Existing resources such as the Enron corpus carry legal ambiguity, demographic skew, and no structured ground truth. Purely LLM-generated synthetic data solves the legal problem but introduces a subtler one: the generating model cannot be prevented from hallucinating facts that contradict themselves across documents. We present OrgForge , an open-source multi-agent simulation framework that enforces a strict physics-cognition boundary : a deterministic Python engine maintains a SimEvent ground truth bus; large language models generate only surface prose, constrained by validated proposals. An actor-local clock enforces causal timestamp correctness across all artifact types, eliminating the class of timeline inconsistencies that arise when timestamps are sampled independently per document. We formalize three graph-dynamic subsystems—stress propagation via betweenness centrality, temporal edge-weight decay, and Dijkstra escalation routing—that govern organizational behavior independently of any LLM. Running a configurable N N -day simulation, OrgForge produces interleaved Slack threads, JIRA tickets, Confluence pages, Git pull requests, emails, and server logs, all traceable to a shared, immutable event log. We additionally describe a causal chain tracking subsystem that accumulates cross-artifact evidence graphs per incident, a hybrid reciprocal-rank-fusion recurrence detector for identifying repeated failure classes, and an inbound/outbound email engine that routes vendor alerts, customer complaints, and HR correspondence through gated causal chains with probabilistic drop simulation. OrgForge is available under the MIT license. Availability. Code and corpus generation tools: github.com/aeriesec/orgforge . Dataset: huggingface.co/datasets/aeriesec/orgforge . Archived release: zenodo.org/records/19036018 . 1. Introduction Retrieval-augmented generation has become a primary deployment pattern for enterprise LLM applications, yet the evaluation infrastructure lags the deployment reality. Benchmarks for RAG typically test either retrieval quality in isolation or end-to-end question answering on static Wikipedia-style passages. Neither captures the properties of real enterprise knowledge bases: documents that reference each other across systems, facts that evolve over time, incidents that leave traces in multiple artifact types simultaneously. The Enron corpus ( Klimt & Yang 2004 ) remains the most widely used real corporate dataset for NLP research, despite being two decades old, legally fraught, and representing a single pathological organizational context. More recent alternatives exist—FinanceBench, MultiHop-RAG ( Tang & Yang 2024 ) , LongBench ( Bai et al. 2023 ) —but none provide the cross-system, temporally grounded, ground-truth-annotated structure that enterprise RAG evaluation requires. The tempting solution is to generate synthetic data with large language models. LLMs write convincing Slack messages and JIRA tickets. The problem is that LLM-generated corpora have no external ground truth to validate against. If a language model generates a Slack thread in which an engineer says “the auth service has been down since 3am” and separately generates a JIRA ticket that records the incident as starting at 9am, there is no mechanism to detect or prevent the contradiction. The corpus looks realistic but is internally inconsistent in ways that will silently corrupt any RAG evaluation run against it. OrgForge addresses this through a strict architectural boundary: LLMs propose, the engine executes. A deterministic simulation engine controls all facts—who is on call, what time an incident started, which engineer owns a ticket, what the current system health is. Language models receive validated context and generate surface prose only. Every significant action emits a structured SimEvent to a persistent log. The corpus and the ground truth bus are produced by the same simulation run, making them structurally guaranteed to be consistent. This paper makes the following contributions: • A simulation architecture formalized as M=(S,P,V,E) that separates fact control from prose generation, preventing LLM hallucinations from contaminating synthetic corpora. • Three formally specified graph-dynamic mechanisms—stress propagation, edge decay, and Dijkstra escalation—that govern organizational behavior deterministically. • A proposal-validation loop that allows LLM planners to influence narrative direction without bypassing factual constraints. • A multi-artifact output schema grounded in a shared SimEvent bus, enabling cross-artifact reasoning benchmarks. • An actor-local clock ( sim_clock.py ) that guarantees causal timestamp correctness across all artifact types, eliminating the class of timeline inconsistencies produced by independent per-artifact timestamp sampling. • An ambient social interruption model that introduces ground-truth capacity noise into the corpus, enabling evaluation of retrieval systems against realistic benign communication rather than purely task-driven artifacts. • A causal chain tracking subsystem ( causal_chain_handler.py ) that accumulates ordered artifact graphs per incident, and a hybrid reciprocal-rank-fusion recurrence detector that identifies repeated failure classes via fused vector and text search over the SimEvent log. • An inbound/outbound email engine ( external_email_ingest.py ) that routes vendor alerts, customer complaints, and HR correspondence through gated causal chains, with probabilistic drop simulation creating verifiable communication gaps for retrieval evaluation. • A complete evaluation pipeline— eval_harness.py , scorer.py , and export_to_hf.py —that generates eight typed question categories with deterministic ground truth, scores agent answers with partial credit, and exports HuggingFace-ready Parquet datasets with BM25 and dense retrieval baselines. • An open-source implementation configurable by org chart, persona definitions, and incident triggers, supporting reproducible corpus generation. 2. Background and Motivation 2.1 Limitations of Existing Corpora The Enron email dataset ( Klimt & Yang 2004 ) has been used in hundreds of NLP studies but carries well-documented limitations: it represents a single company in crisis, skews heavily toward senior management communication, and is subject to ongoing legal sensitivity. More practically, it contains no structured ground truth—there is no authoritative record of what actually happened that researchers can use to verify whether a RAG system retrieved the correct facts. Synthetic alternatives generated by LLMs have proliferated with the availability of capable models, but they inherit the hallucination problem at the corpus level. When the generating model has no external constraint on factual consistency, the resulting corpus cannot be used as a reliable evaluation benchmark. A system that retrieves an internally inconsistent document and returns a “correct” answer by coincidence is indistinguishable from one that retrieves and reasons correctly. 2.2 RAG Evaluation Requirements Effective RAG evaluation requires corpora with at least four properties that existing resources lack simultaneously: 1. Traceable ground truth: each retrievable fact must have a canonical source that can be used to score retrieval. 2. Temporal structure: facts must change over time so that temporal reasoning—“what was the state of X on day N ?”—can be evaluated. 3. Cross-artifact coherence: the same fact must appear consistently across multiple document types, enabling multi-hop retrieval evaluation. 4. Configurable complexity: incident severity, organizational size, and communication patterns should be tunable to probe specific system behaviors. OrgForge is designed to satisfy all four. 2.3 Related Work MultiHop-RAG ( Tang & Yang 2024 ) focuses on multi-hop reasoning but uses news articles without organizational structure. FRAMES ( Krishna et al. 2024 ) provides factual, temporal, and multi-document questions but is not grounded in a controllable simulation. Enterprise RAG benchmarks accompanying proprietary systems are not publicly available for independent evaluation. In the agent-based simulation literature, social network simulations have been used to study information diffusion ( Watts & Strogatz 1998 ) and epidemic spread, but have not been connected to document corpus generation for NLP evaluation. OrgForge bridges this gap by using the organizational simulation as