When Should Humans Step In? Optimal Human Dispatching in AI-Assisted Decisions Conference: ; ;
Lezhi Tan, Naomi Sagan, Lihua Lei, José Blanchet · 2026
Read When Should Humans Step In? Optimal Human Dispatching in AI-Assisted Decisions Conference: ; ; by Lezhi Tan, Naomi Sagan, Lihua Lei, José Blanchet (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-13688.txt.
Introduction
Lezhi Tan, Naomi Sagan, Lihua Lei, and Jose Blanchet ask when humans should step in on AI-assisted decisions. AI systems give cheap factor-level assessments of complex inputs, but those can be biased or incomplete. They formulate selective human evaluation as a factor-level information-acquisition problem. Under squared loss and a linear model, the optimal dispatching reward splits into predictive relevance and residual uncertainty terms. On peer review, querying three of ten aspects under their rule matched the full ten-aspect human benchmark.
Catalog record
| CREATOR | Lezhi Tan, Naomi Sagan, Lihua Lei, José Blanchet |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 89384 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.13688
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
When Should Humans Step In? Optimal Human Dispatching in AI-Assisted Decisions Conference: ; ; Lezhi Tan, Naomi Sagan, Lihua Lei, José Blanchet Abstract . AI systems increasingly assist human decision making by producing preliminary assessments of complex inputs. However, such AI-generated assessments can often be noisy or systematically biased, raising a central question: how should costly human effort be allocated to correct AI outputs where it matters the most for the final decision? We propose a general decision-theoretic framework for human-AI collaboration in which AI assessments are treated as factor-level signals and human judgments as costly information that can be selectively acquired. We consider cases where the optimal selection problem reduces to maximizing a reward associated with each candidate subset of factors, and turn policy design into reward estimation. We develop estimation procedures under both nonparametric and linear models, covering contextual and non-contextual selection rules. In the linear setting, the optimal rule admits a closed-form expression with a clear interpretation in terms of factor importance and residual variance. We apply our framework to AI-assisted peer review. Our approach substantially outperforms LLM-only predictions and achieves performance comparable to full human review while using only 20–30% of the human information. Across different selection rules, we find that simpler rules derived under linear models can significantly reduce computational cost without harming final prediction performance. Our results highlight both the value of human intervention and the efficiency of principled dispatching. † † footnotetext: J. Blanchet gratefully acknowledges support from DoD through the grants Air Force Office of Scientific Research under award number FA9550-20-1-0397 and ONR 1398311. Support from NSF via grants 2229012, 2312204, 2403007 is gratefully acknowledged. L.Lei is grateful for the support of National Science Foundation grant DMS-2338464. N. Sagan is supported by Stanford Graduate Fellowship (SGF) for Sciences and Engineering, and National Science Foundation Graduate Research Fellowship Program under Grant No. DGE-2146755. 1. Introduction Many real-world decisions rely on interpreting complex, high-dimensional inputs. Examples include evaluating a legal document, reviewing a medical record, screening a financial report, and assessing a scientific manuscript. In each case, the decision maker examines a long and technical object and distills it into a small number of interpretable factors that support a final judgment. In scientific peer review, for instance, review guidelines explicitly specify such factors, including novelty, clarity, and soundness. Traditionally, this process is carried out entirely by human experts and proceeds in two stages. In the first stage, reviewers read the full document W and extract factor-level assessments X_{1}(W),\ldots,X_{J}(W) corresponding to the prescribed evaluation criteria. In the second stage, these factors are aggregated—often implicitly and without a formal decision rule—into a final outcome Y , such as an overall score or accept/reject decision, either by a single reviewer or by combining inputs from multiple experts. While reliable, this workflow is expensive, slow, and difficult to scale, as it requires substantial human labor to extract every factor for every instance. A common practice in statistical learning ( Hastie et al. 2009 ) is to learn a predictive model that maps the factor-level information (X_{1},\ldots,X_{J}) to the final outcome Y . Such models can then be applied to new instances by first collecting all factor values and computing a prediction. However, this practice conceals a major bottleneck: obtaining the full set of factor-level inputs X_{j}(W) is itself costly, often accounting for the majority of human effort in the decision-making process. Recent advances in AI systems have begun to reshape this pipeline. These systems are able to process complex, unstructured inputs and produce structured, factor-level evaluations at a fraction of the cost of human review. In scientific peer review, a prominent example is the use of large language models (LLMs) to generate detailed assessments directly from the manuscript text. This creates an opportunity to rethink the traditional workflow: instead of asking humans to generate all supporting reasons from scratch, AI-generated signals can serve as an initial scaffold that humans refine. At the same time, AI-generated evaluations are imperfect, and their widespread informal use has raised growing concerns. In peer review settings, for example, LLMs are often systematically over-optimistic, assigning inflated scores or failing to detect subtle technical flaws (see our literature review below). Nonetheless, human reviewers increasingly rely on LLMs to summarize submissions, draft reviews, and even shape final recommendations, often without clear delineation between human judgment and machine output. More broadly, AI signals may be noisy, biased, or unreliable on specific factors. Because human evaluations are costly, it is often impractical to verify every aspect manually. This raises a central question: how should costly human effort be dispatched to the factors where human judgment is most needed, so as to retain decision quality while reducing the overall burden of review? Figure 1. High-level diagram of our methodology in the AI-assisted peer review setting. First, an LLM generates a set of feature estimates A for a paper. Next, a selection policy determines which subset of aspects to query from a human reviewer. Finally, the selected human inputs, together with the LLM features, are used to fit a prediction model for the target outcome. In this paper, we study a general framework for human–AI collaborative decision making based on this dispatching perspective. We consider settings in which an AI system produces preliminary predictions over a collection of decision-relevant factors, and human effort is selectively allocated to correct a subset of these factors. Rather than treating human involvement as an all-or-nothing choice, we model it as a resource that must be strategically dispatched across factors. The goal is to design a selection rule that determines which factors to query—based on the observed AI signals—so as to minimize downstream decision error. We formalize this problem as an information acquisition task under squared error loss. Our analysis considers predictive models that are expressive enough to capture the relationship between the decision outcome and the available AI and human signals. Under this assumption, we characterize the optimal selection rule and establish statistical learning rates of convergence for its empirical estimation. This covers the nonparametric case where the selection criterion is learned from data without imposing a parametric form on the underlying relationships between factor values and final decision. We then move from this general setting to parametric cases to gain additional insight. Using linear models as an illustrative example, we express the selection rule explicitly in terms of linear coefficients and conditional variances. This representation clarifies how the value of querying a factor depends jointly on its relevance to the final decision and the residual uncertainty after observing AI-generated signals. We derive asymptotic normality guarantees in the parametric setting. We apply our framework to AI-assisted peer review, a domain where the interaction between human and AI evaluations is becoming increasingly consequential. The rapid growth in submission volumes at major conferences has substantially increased the burden on human reviewers, making it difficult to sustain high-quality evaluations at scale. At the same time, large language models are now widely used to summarize papers, draft reviews, and even inform final recommendations. While these tools offer clear efficiency gains, recent studies show that their evaluations are often systematically biased and overly optimistic ( Pangram Labs 2025 ; Fanous et al. 2025 ; Liang et al. 2024 ;