HLER: Human-in-the-Loop Economic Research via Multi-Agent Pipelines for Empirical Discovery
Chen Zhu, Xiaolu Wang · 2026
Read HLER: Human-in-the-Loop Economic Research via Multi-Agent Pipelines for Empirical Discovery by Chen Zhu, Xiaolu Wang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-07444.txt.
Introduction
Zhu and Wang build a human-in-the-loop pipeline for empirical economics research with specialized agents. Fully autonomous discovery risks questions the data cannot answer, so their HLER system grounds question generation in dataset structure, variable availability and diagnostics. Agents audit and profile data, pose questions, run econometric analysis, draft manuscripts and review them, with human gates at key stages. Dataset-aware questions prove feasible far more often than unconstrained ones.
Catalog record
| CREATOR | Chen Zhu, Xiaolu Wang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36429 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.07444
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
HLER: Human-in-the-Loop Economic Research via Multi-Agent Pipelines for Empirical Discovery Chen Zhu, Xiaolu Wang Abstract Large language models (LLMs) have enabled agent-based systems that attempt to automate scientific research workflows. Existing approaches often pursue fully autonomous discovery, in which AI systems generate research ideas, conduct experiments, and produce manuscripts with minimal human involvement. However, empirical research in economics and the social sciences poses additional challenges: research questions must be grounded in available datasets, identification strategies require careful design, and human judgment remains essential for evaluating economic significance. We introduce HLER (Human-in-the-Loop Economic Research), a multi-agent architecture that supports empirical research automation while preserving critical human oversight. The system orchestrates specialized agents for data auditing, data profiling, hypothesis generation, econometric analysis, manuscript drafting, and automated review. A central design principle is dataset-aware hypothesis generation , in which candidate research questions are constrained by dataset structure, variable availability, and distributional diagnostics. This prevents infeasible or hallucinated hypotheses that frequently arise in unconstrained LLM ideation. HLER further introduces a two-loop research architecture: a question quality loop in which candidate hypotheses are generated, screened for feasibility, and selected by a human researcher; and a research revision loop in which automated review triggers re-analysis and manuscript revision. Human decision gates are embedded at key stages, including question selection and publication approval, allowing researchers to steer the automated pipeline. We evaluate the framework on three empirical datasets, including longitudinal survey data. Across 14 pipeline runs, dataset-aware hypothesis generation produces feasible research questions in 87% of cases, compared with 41% under unconstrained generation, and the system successfully completes end-to-end empirical manuscripts at an average API cost of $0.8-$1.5 per run. A detailed case study on the China Health and Nutrition Survey illustrates the full workflow from data inspection to a revised draft manuscript. These results suggest that Human-AI collaborative pipelines may provide a practical path toward scalable empirical research. The HLER working paper repository is publicly available at https://github.com/maxwell2732/hler-working-papers . 1 Introduction Recent advances in large language models (LLMs) have substantially improved automated scientific writing, including summarization, drafting, and the generation of coherent technical narratives [ 1 , 2 , 3 ] . Frontier model reports document broad improvements in academic and professional task performance [ 1 , 2 ] , and domain-targeted scientific LLMs illustrate the feasibility of using language models as interfaces for scientific knowledge [ 3 ] . At the same time, most existing LLM-based workflows remain primarily text-centric: they can draft plausible prose but struggle to reliably perform the end-to-end steps required for credible empirical research, particularly in data-driven social sciences. One widely documented limitation is hallucination, generation models can produce fluent but unsupported claims, which motivates verification, constraint, and critique mechanisms when models are used in research settings [ 4 ] . A second limitation is that empirical research requires interaction with external artifacts (datasets, code, econometric software, and intermediate results). Agentic paradigms such as ReAct demonstrate that reliable task completion often requires coupling LLM reasoning with tool-mediated actions that query external sources and reduce error propagation [ 5 ] . These limitations imply that simply generating a paper-like narrative is insufficient for empirical disciplines where credibility depends on data validation, transparent estimation, and robustness. A growing body of work explores autonomous research, vibe research, and “AI scientist” pipelines that extend LLMs beyond writing to include idea generation, code execution, and automated review, largely in machine learning research environments [ 6 ] . These systems provide an important precedent for end-to-end automation, but they also highlight a gap for empirical economics and related social sciences: credible empirical contribution depends on dataset feasibility checks, econometric identification choices, and reproducible workflows, not only on experimental iteration in simulated or ML-centric settings. Empirical economics is inherently procedural and iterative. A typical study requires (i) validating dataset structure and variable definitions, (ii) selecting feasible questions that match observed variation, (iii) specifying an identification strategy, (iv) estimating models, and (v) subjecting results to critical scrutiny prior to dissemination. Evidence from replication audits suggests that even when papers are required to provide data and code, reproducing published economics findings often fails or requires substantial author assistance, underscoring the fragility of empirical workflows [ 7 ] . More broadly, social-scientific conclusions can be highly sensitive to analytic choices: even holding data and the research question fixed, different teams can arrive at meaningfully different estimates due to modeling decisions [ 8 ] . This paper introduces the Human-in-the-Loop Economic Research (HLER) pipeline, a multi-agent architecture designed to automate the empirical research workflow while preserving key human oversight. The system orchestrates specialized agents that sequentially perform data auditing, data profiling, research question generation, econometric analysis, manuscript drafting, and automated review. Our orchestration approach is inspired by recent multi-agent frameworks that coordinate specialized LLM roles and allow flexible mixtures of tool use and human inputs [ 9 ] . To improve feasibility and reduce hallucinated hypotheses, HLER implements dataset-aware question generation , constraining candidate questions by dataset structure, variable availability, missingness, and distributional diagnostics rather than unconstrained ideation. A central design feature of HLER is a two-loop research architecture. The question quality loop generates candidate hypotheses, screens them for feasibility using dataset diagnostics, and presents them to a human researcher for selection. The research revision loop subjects the resulting manuscript to automated review; the reviewer agent may request additional robustness checks or improved exposition, triggering re-analysis and manuscript revision. This iterative design is motivated by evidence that structured self-feedback can substantially improve model outputs without additional training [ 10 ] . By integrating data constraints, econometric execution, and iterative critique within a single orchestrated pipeline, HLER aims to extend LLM utility from automated writing toward scalable, human-supervised empirical discovery in economics. Our main contributions are: 1. A dataset-aware hypothesis generation mechanism that conditions research question generation on dataset audits and statistical profiles, reducing infeasible hypotheses from 59% to 13% compared with unconstrained LLM ideation. 2. A two-loop research architecture consisting of a question quality loop (generation \rightarrow screening \rightarrow human selection) and a research revision loop (review \rightarrow re-analysis \rightarrow revision), with empirical evidence that reviewer scores improve across revision iterations. 3. An end-to-end evaluation across three empirical datasets and 14 pipeline runs, demonstrating that the system can reliably produce complete research manuscripts at an average cost of $0.8-$1.5 per run. 2 Related Work 2.1 LLM Agents for Scientific Discovery Recent advances in large language models have led to growing interest in using AI systems as autonomous or semi-autonomous researchers. Several studies propose multi-agent architectures capable of generating hypotheses, implementing experiments, and producing scientific manuscripts. The AI Scientist introduces an end-to-end framework in which LLM agents autonomously generate research ideas, write code, analyze results, and produce full scientific papers, followed by an automated review process [ 6 ] . More broadly, a growing literature explores agent-based research automation, where multiple specialized agents coordinate to perform different components of the scientific workflow [ 11 ] . Related work has also explored domain-specific autonomous research pipelines. BioResearcher proposes a modular multi-agent system for biomedical research workflows, including literature processing, experiment design, and protocol generation [ 12 ] . AutoLabs attempts to automate experimental workflows in chemistry using self-correcting agents and structured reasoning [ 13