Anchor: Mitigating Artifact Drift in Agent Benchmark Generation Conference: RLEval: Methods and Reinforcement Learning Environments for Evaluating AI Agents, Workshop at the ACM Conference on AI and Agentic Systems (ACM CAIS 2026); May 26, 2026; San Jose, CA, USA CCS: Computing methodologies Artificial intelligence CCS: Computing methodologies Planning and scheduling CCS: Software and its engineering Software testing and debugging
Maksim Ivanov, Abhijay Rana · 2026
Read Anchor: Mitigating Artifact Drift in Agent Benchmark Generation Conference: RLEval: Methods and Reinforcement Learning Environments for Evaluating AI Agents, Workshop at the ACM Conference on AI and Agentic Systems (ACM CAIS 2026); May 26, 2026; San Jose, CA, USA CCS: Computing methodologies Artificial intelligence CCS: Computing methodologies Planning and scheduling CCS: Software and its engineering Software testing and debugging by Maksim Ivanov, Abhijay Rana (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-26321.txt.
Introduction
Maksim Ivanov and Abhijay Rana's 2026 pipeline, Anchor, fights artifact drift by turning expert workflow specs into constraint optimization programs. From one parametric specification, it jointly produces an instruction, environment setup, certified solution, and state verifier. Tuning parameters creates new tasks of controlled difficulty and known optima, with rewards hinging on end-state business correctness. ERP-Bench holds three hundred procurement and manufacturing tasks. Frontier models meet constraints in twenty-six point one percent of trials, but reach optima in only seventeen point four percent.
Catalog record
| CREATOR | Maksim Ivanov, Abhijay Rana |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 54613 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.26321
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Anchor: Mitigating Artifact Drift in Agent Benchmark Generation Conference: RLEval: Methods and Reinforcement Learning Environments for Evaluating AI Agents, Workshop at the ACM Conference on AI and Agentic Systems (ACM CAIS 2026); May 26, 2026; San Jose, CA, USA CCS: Computing methodologies Artificial intelligence CCS: Computing methodologies Planning and scheduling CCS: Software and its engineering Software testing and debugging Maksim Ivanov, Abhijay Rana Abstract . AI agents are beginning to complete valuable, long-horizon business operations tasks, but training and evaluation environments for enterprise work still struggle to balance realism, verifiability, and scale. Environment and task creation frequently suffers from a failure mode we call artifact drift : when instructions, environments, oracles, and verifiers are created by loosely coupled processes, they frequently disagree on what a task requires, producing environments that are unsolvable, reward-hackable, or inconsistent. We introduce Anchor, a task-generation pipeline that formalizes domain experts’ specifications of business workflows into constraint optimization programs. From a single parametric specification, the pipeline jointly produces a natural-language instruction, environment configuration, solver-certified ground-truth solution, and state-based verifier. With Anchor, altering parameters yields new tasks with controlled difficulty and known optimal solutions, producing harness-agnostic environments whose rewards depend solely on end-state business correctness. We apply Anchor to produce ERP-Bench: a benchmark of 300 long-horizon tasks spanning procurement and manufacturing workflows in a production-grade ERP system. We find that generation parameters predict realized difficulty, and that frontier models satisfy explicit task constraints in 26.1% of trials but reach a fully optimal solution in only 17.4% of trials. Overall, we show that Anchor and ERP-Bench offer a concrete recipe for building auditable evaluation environments for economically valuable agent work. We release the task generator and ERP-Bench dataset at erpbench.ai . 1. Introduction Figure 1. Artifact drift. Inconsistencies between a task’s four artifacts—instruction I , environment E , oracle solution x^{*} , and verifier V —each invalidate the intended task \tau in a different way. Diagram of the four task artifacts and the four failure modes that arise when they disagree. Figure 2. Anchor single-source task creation pipeline. A solved constraint satisfiability problem specification generates the instruction, environment setup, oracle solution, and terminal-state verifier for each task instance. Anchor single-source task creation pipeline. Recent surveys document substantial gaps between benchmark scores and the production performance of language-model agents on enterprise tasks ( 10 ; 7 ; 18 ) . Audits trace much of this gap to construction errors in the benchmarks themselves. Do-nothing agents pass 38% of \tau -bench airline tasks ( 17 ) because the verifier accepts empty responses ( 21 ) , and strengthening unit tests in SWE-bench ( 6 ) reranks 40.9% of SWE-bench Lite leaderboard positions ( 1 ; 19 ) . Benchmark authors face an inherent tension between realism, verifiability, and scale. Expert-authored benchmarks improve realism but require costly curation ( 16 ) ; synthetic generators scale task creation but ship noisy or single-path graders ( 15 ; 12 ) ; and \tau^{2} -bench authors observe that earlier benchmarks pushed instructions toward “carefully crafted …to help ensure a single, solvable path” ( 3 ) . Most benchmarks still author each task’s four artifacts of instruction, environment, oracle solution, and verifier in parallel and validate consistency post hoc through audit. This creates a four-way consistency failure we call artifact drift ( fig. 1 ): loosely coupled processes end up describing subtly different tasks, such as when the environment lacks data the instruction assumes, the oracle depends on state the environment omits, or the verifier accepts outcomes the instruction did not request. We address this with Anchor , a task-generation pipeline that compiles all four artifacts from a single solved specification. A domain expert, with an engineer, formalizes a business workflow as a parametric constraint program in OR-Tools CP-SAT ( 11 ) with decision variables, business-rule constraints, and an objective metric. For a proposed parameter setting, the CP-SAT solver either rejects the parameters as infeasible or certifies an optimal solution, which then compiles deterministically into the task artifacts. Because all four artifacts are deterministic projections of the same solved specification, the dataset mitigates artifact drift by construction. The same parameters that define a task also tune its difficulty, so the pipeline could support a verifiable training data curriculum for agent training. We apply Anchor to produce ERP-Bench , 300 procurement and manufacturing tasks across 29 workflow patterns in Odoo 19, a production-grade open-source ERP system. We evaluate five frontier models across coding, browser, and computer-use harnesses across 18,000 trials. Pass@5 falls monotonically from easy to hard tiers in every harness, dropping from 70.5% to 22.3% in coding, 46.5% to 7.7% in browser, and 56.0% to 9.5% in computer-use, whith zero instances of reward hacking during evaluation. Our key contributions are: • Anchor : a task creation pipeline that compiles instruction, environment, oracle, and verifier from one solved constraint program specification. • ERP-Bench : a 300-task verifiable benchmark of long-horizon procurement and manufacturing workflows in a production-grade ERP, with controlled difficulty and certified optima. • Evaluation : a controlled comparison of frontier proprietary and open-weight models across coding, browser, and computer-use harnesses on ERP-Bench tasks. 2. Anchor Our task creation pipeline follows from prior work in large language models and mathematical reasoning. AlphaProof and related systems treat an informal mathematical statement as the start of a pipeline that translates it into a formal Lean program, where a deterministic checker grades any candidate proof and where synthetic variants of the formalized statement become the curriculum for reinforcement learning ( 5 ; 2 ) . We similarly aim to translate informal business workflows into checkable programs and generate synthetic variants to address the data scarcity and fidelity problem. Unlike AlphaProof and other autoformalization work, we undertake the formalization step manually, and the generated variants are translated back into informal scenarios for agent training and evaluation. Many enterprise workflows operate on structured data in systems of record, follow explicit business rules, and optimize measurable outcomes, which makes them naturally expressible as constraint satisfiability and optimization problems. The Anchor pipeline ( fig. 2 ) starts when a domain expert and an engineer formalize a workflow such as invoice prioritization, deal qualification, or production scheduling as a parametric constraint program ( 11 ) with decision variables, business-rule constraints, and an objective function. This constraint program becomes the core of the task generation engine. Given a parameter setting \theta , the solver either rejects the sample as infeasible or certifies an optimal solution x^{*}_{\theta} . We call the resulting parameters, constraints, objective, and certified solution the solved specification S_{\theta} . Four translation layers then compile S_{\theta} into the task artifacts shown in fig. 2 : an instruction generator renders parameters, constraints, and objective as natural language; a setup generator writes the sampled initial records into the environment container; an oracle generator writes the solver’s solution as the reference terminal state; and a verifier generator grades terminal states against the program’s constraints and objective. Because the four artifacts are deterministic projections of S_{\theta} , the inconsistencies of fig. 1 are mitigated by construction. Because the solver certifies an optimal objective value, verification remains tractable without contriving the task to a single action path. Anchor does not eliminate every construction error: the constraint program