Structural Enforcement of Statistical Rigor in AI-Driven Discovery: A Functional Architecture
Karen Sargsyan · 2025
Read Structural Enforcement of Statistical Rigor in AI-Driven Discovery: A Functional Architecture by Karen Sargsyan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-06701.txt.
Introduction
Karen Sargsyan designs guardrails that stop automated discovery systems from manufacturing fake findings through repeated testing. When software tests hypotheses endlessly, false discoveries pile up unless the error budget is policed. The architecture enforces that policing structurally, at two levels, rather than trusting the code to behave.
A Haskell-embedded language makes it impossible to run a test without updating the error budget, and a sandbox keeps validation data physically unreachable from generated code. The false discovery framework rests on a machine-checked Lean proof of an online control theorem, with threshold arithmetic verified separately. Simulations held the false rate near one percent where a naive setup reached forty-one.
Catalog record
| CREATOR | Karen Sargsyan |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 25 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-06701
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Structural Enforcement of Statistical Rigor in AI-Driven Discovery: A Functional Architecture Karen Sargsyan Abstract Sequential statistical protocols require meticulous state management and robust error handling—challenges naturally suited to functional programming. We present a functional architecture for structural enforcement of statistical rigor in automated research systems ("AI-Scientists"). These LLM-driven systems risk generating spurious discoveries through dynamic hypothesis testing. We introduce the Research monad, a Haskell eDSL that enforces sequential statistical protocols (e.g., Online FDR (false discovery rate) control) using a monad transformer stack. To address risks in hybrid architectures where LLMs generate imperative code, we employ Declarative Scaffolding— generating rigid harnesses that structurally constrain execution and prevent methodological errors like data leakage. We validate this approach through large-scale simulation (N=2000 hypotheses) and an end-to-end case study, demonstrating essential defense-in-depth for automated science integrity. 1 Introduction Managing the complex, evolving state required by sequential statistical protocols is a significant challenge in software engineering. Protocols designed to control the False Discovery Rate (FDR) during dynamic hypothesis testing require strict sequential accounting of an "error budget." Imperative implementations of these protocols are brittle; subtle errors in state management can invalidate statistical guarantees. Functional programming paradigms offer powerful mechanisms for managing state and effects rigorously. We propose a functional architecture that leverages these mechanisms for the structural enforcement of statistical rigor. The motivating application domain is the emerging field of "AI-Scientists" [ 1 , 2 , 3 , 4 , 5 , 6 , 7 , 8 , 9 ] , systems that use Large Language Models (LLMs) to automate the scientific process. Despite some initial critical reception [ 10 ] , the field is maturing: useful discoveries have been made within the AI-Scientist paradigm [ 11 ] , and a paper produced by an AI-Scientist has passed peer review at a workshop in a top machine learning conference [ 12 ] . In addition, tools to standardize and democratize AI-Scientists, such as ToolUniverse [ 13 ] , have been developed. Recently, the reliability and safety of results produced by such systems have been recognized as an important direction of research [ 14 , 15 , 16 ] . Among other concerns, these systems rapidly generate and test thousands of hypotheses, dramatically increasing the risk of spurious discoveries (p-hacking) [ 17 , 18 ] . We address this challenge at two levels. At the orchestration level, we introduce the Research monad, a generalized Haskell embedded Domain-Specific Language (eDSL) that uses a monad transformer stack to guarantee inevitable accounting and robust error handling for sequential protocols. However, AI-Scientists typically employ a hybrid architecture, interfacing with imperative environments like Python for execution, leveraging its mature machine learning ecosystem. This creates a trust boundary where functional guarantees end. To extend enforcement across this boundary, we employ Declarative Scaffolding . The functional orchestrator generates a rigid execution harness that structurally constrains the LLM-generated imperative code, preventing methodological errors such as data leakage. Our contributions are: 1. The Research monad: A type class abstraction for sequential statistical protocols, implemented using a robust monad transformer stack ( ExceptT over StateT ). 2. Declarative Scaffolding: A technique for structural enforcement of standardized methodology across the hybrid language boundary. 3. Validation: A large-scale simulation (N=2000) demonstrating the necessity of FDR control, and an end-to-end case study demonstrating the integrated architecture. 2 Architecture and Challenges Modern AI-Scientists often utilize a hybrid architecture (Figure 1 ). This design leverages the strengths of different paradigms but introduces significant challenges for maintaining scientific integrity. Haskell Orchestrator (Functional Core) Research Monad, Statistical State (FDR), Protocol Logic Python Execution (Imperative) LLM-Generated Code, Data Handling, ML Models IO Boundary (Trust Boundary) Generates Scaffold Returns P-Value Figure 1: The Hybrid AI-Scientist Architecture. The functional core manages the global statistical state, while the imperative environment executes experiments. The IO boundary represents the critical trust boundary. 2.1 The Hybrid Workflow The process begins with the Haskell Orchestrator, which manages the sequence of research ideas. For each idea, the orchestrator must ensure methodological correctness before execution. The execution itself occurs in the Python environment, where an LLM Coder Agent generates the specific implementation logic. Upon completion, the Python environment returns a P-value to the Haskell orchestrator, which then updates the global statistical state. 2.2 The Two Levels of Risk This architecture introduces risks at two distinct levels: Macro-level Risk (Orchestration): This concerns the global management of the False Discovery Rate (FDR) across the sequence of ideas. Online FDR protocols, such as LORD++ [ 19 ] , require strict sequential accounting of an " \alpha -wealth" (error budget). Errors in state management (e.g., timing errors, state corruption) at this level can invalidate the overall statistical guarantees. Micro-level Risk (Execution): This concerns the integrity of individual experiments within the Python environment. The LLM-generated code might introduce bias, such as data leakage between exploration and validation datasets, or use inappropriate statistical tests. If the input P-values are biased ("garbage in"), the macro-level FDR control is meaningless ("garbage out"). Furthermore, the dynamic nature of LLM execution means the total number of tests is uncertain. The LLM might fail to generate valid code, terminating a test prematurely. This uncertainty necessitates Online FDR control methods, as traditional methods (e.g., Bonferroni) require knowing the total number of tests in advance. 3 The Generalized Research Monad (Macro-Level Control) We address the macro-level risks using a functional framework based on a generalized eDSL and a robust monad transformer stack. 3.1 Abstracting the Protocol We abstract the sequential statistical test using a Haskell Type Class, StatisticalProtocol (Listing ). ⬇ 1 data ProtocolError = InvalidTransitionError String | ... 2 3 class StatisticalProtocol s where 4 type Config s 5 -- Initialization returns Either for robustness. 6 initializeState :: Config s -> Either ProtocolError s 7 8 advanceState :: Double -> s -> ( Bool , Double , s ) 9 10 -- Enforces constraints. No default implementation ensures safety. 11 isValidTransition :: s -> s -> Either ProtocolError () 12 13 -- Example: LORD++ instance enforcing sequential time advancement 14 instance StatisticalProtocol LordState where 15 -- ... (other methods omitted) 16 17 isValidTransition oldState newState = 18 if currentTime newState == currentTime oldState + 1 19 then Right () 20 -- Catches timing errors (e.g., off-by-one). 21 else Left $ InvalidTransitionError "Time must advance sequentially." Listing 1: The StatisticalProtocol Type Class definition and an example constraint. This abstraction enhances modularity. Crucially, isValidTransition has no default implementation, requiring each protocol instance (like LORD++) to explicitly enforce its specific constraints, such as strict sequential time advancement. 3.2 The Monadic Architecture: State and Exceptions To manage the sequential state and handle potential protocol violations robustly, we construct the Research monad using a monad transformer stack (Listing ). ⬇ 1 import Control . Monad . State 2 import Control . Monad . Except 3 4 data ResearchError = RProtocolError ProtocolError | ... 5 6 -- Combines Exceptions (ExceptT) over State (StateT) over effects (m, usually IO). 7 type Research s m a = ExceptT ResearchError ( StateT s m ) a Listing 2: The Robust Research Monad definition. We combine two essential capabilities: Inevitable Accounting (StateT): The foundation is the State Monad (‘StateT s m‘). This guarantees that the protocol state ‘s‘ is threaded purely and sequentially through every