PROTEA : Offline Evaluation and Iterative Refinement for Multi-Agent LLM Workflows
Kazuki Kawamura, Satoshi Waki, Kei Tateno · 2026
Read PROTEA : Offline Evaluation and Iterative Refinement for Multi-Agent LLM Workflows by Kazuki Kawamura, Satoshi Waki, Kei Tateno (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-18032.txt.
Introduction
Kazuki Kawamura, Satoshi Waki, and Kei Tateno present PROTEA, an interface for offline improvement of workflows. Multi-agent workflows beat single prompts but are hard to debug, since errors propagate downstream. PROTEA runs a workflow, scores outputs with rubrics, and overlays states and rationales on the graph. For systems where only final answers supervise, it generates node expectations backward from final references and compares them with outputs. It offers prompt revisions and reruns the workflow, improving inspection accuracy from 64.3 percent to 83.9 percent and Hit@5 from 0.30 to 0.38.
Catalog record
| CREATOR | Kazuki Kawamura, Satoshi Waki, Kei Tateno |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 28118 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.18032
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
PROTEA : Offline Evaluation and Iterative Refinement for Multi-Agent LLM Workflows Kazuki Kawamura, Satoshi Waki, Kei Tateno Abstract Multi-agent LLM workflows—systems composed of multiple role-specific LLM calls—often outperform single-prompt baselines, but they remain difficult to debug and refine. Failures can originate from subtle errors in intermediate outputs that propagate to downstream nodes, requiring developers to inspect long traces and infer which agent to modify. We present PROTEA , a unified interface for offline, test-driven improvement of multi-agent workflows. PROTEA executes a workflow, scores intermediate node outputs with configurable rubrics, and overlays per-node states and rationales on the workflow graph to localize likely bottlenecks. To support complex systems where final-answer references are the primary supervision, PROTEA performs backward node evaluation: it generates candidate node-level expectations from final-answer references and graph context, then compares them with observed node outputs. For selected nodes, PROTEA presents targeted prompt revisions as editable before/after comparisons, then automatically reruns and re-evaluates the workflow to show output changes and score trajectories within the same interface. In two production-adjacent workflows, PROTEA improved document-inspection accuracy from 64.3% to 83.9% and recommendation Hit@5 from 0.30 to 0.38. In a formative study with six experienced LLM developers, participants valued graph-level localization, per-node rationales, and editable before/after prompt revisions. † † footnotetext: To appear in the Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (ACL 2026) System Demonstrations. 1 Introduction LLM applications are increasingly built as agentic and multi-agent systems that compose multiple role-specific LLM calls into a workflow graph. By splitting responsibilities across nodes (e.g., intent analysis, retrieval, planning, ranking, and response generation), such graph-based workflows can be more controllable and produce higher-quality outputs than single-prompt approaches. This design pattern is now widely supported by orchestration frameworks (e.g., AutoGen, LangGraph) and decomposition-based prompting paradigms 21 ; 7 ; 23 . However, this decomposition makes refinement a systems-level debugging task. When the final answer needs improvement, a small omission, misunderstanding, or formatting drift in an upstream artifact can propagate to downstream nodes. Two questions dominate day-to-day iteration: which node is a useful starting point for inspection, and what minimal prompt revision would improve the workflow while preserving desired behavior. Figure 1: Overview of PROTEA : an interactive, evaluation-driven framework for developer-steered, AI-assisted refinement of multi-agent LLM workflows. Developers identify bottlenecks, inspect evidence and prompt revisions, edit or approve changes, and compare behavior within one loop. A growing body of work provides evaluation resources for LLMs and agents, but these typically emphasize end-to-end success rather than graph-aware diagnosis. Agent benchmarks for tool use and multi-step behavior (e.g., AgentBench and ToolLLM/ToolBench) report task-level outcomes and aggregated statistics 11 ; 16 . Similarly, practical evaluation frameworks and libraries—including OpenAI Evals, lm-evaluation-harness, OpenEvals, DeepEval, and promptfoo—support offline testing and regression suites for LLM applications 14 ; 3 ; 9 ; 5 ; 19 . For specific architectures such as RAG, frameworks like RAGAs and ARES evaluate components such as retrieval relevance and answer faithfulness 2 ; 17 . These tools are useful for measurement and comparison, yet they often evaluate the application as an end-to-end system: even when traces are available, the developer must manually determine where in a workflow the error was introduced and how to revise a particular node prompt. Another line of tools focuses on the development and operation of LLM applications by providing tracing, datasets, evaluations, and prompt management in the interface. For example, LangSmith aims to support tracing and evaluation alongside prompt testing and deployment management, while platforms such as Arize Phoenix, Weave, Langfuse, and PromptLayer provide complementary combinations of observability, evaluation, and prompt/version management 8 ; 1 ; 20 ; 10 ; 15 . These platforms substantially improve visibility, but the iteration-and-comparison loop for multi-agent graphs still requires substantial manual effort: developers translate evaluation evidence into a candidate fix, edit prompts (often in a separate view or in code), rerun the workflow, and then reconstruct what changed. Moreover, in realistic product settings, teams often have labels only for the terminal output; defining reference outputs for every intermediate node is costly, and such references are rarely maintained. Automatic prompt optimization and self-improvement methods address a different part of the problem. DSPy compiles modular LM programs and can optimize prompts/modules against an end metric; OPRO and APE use LLMs to search for improved prompts; Self-Refine and Reflexion iteratively improve generations via self-feedback and reflection 6 ; 22 ; 25 ; 13 ; 18 . These approaches are complementary to PROTEA : they optimize program behavior against an objective, whereas PROTEA focuses on the developer workflow for a concrete multi-agent graph by surfacing candidate bottleneck nodes and proposing localized, inspectable prompt revisions that a developer can adopt, edit, or revert in context. We present PROTEA , a unified interface for offline evaluation and iterative refinement of multi-agent LLM workflows that is explicitly designed for workflow designers, prompt engineers, and planners. PROTEA is built around three ideas. First, it surfaces node-level evidence rather than only terminal scores: intermediate node outputs are evaluated with configurable rubrics using LLM-as-a-judge techniques 12 ; 24 ; 4 , and pass/warn/fail states plus rationales are overlaid directly on the workflow graph. Second, PROTEA reduces labeling effort via backward node evaluation: from final-answer references, it generates node-level expectations using the desired final output and downstream requirements, evaluates nodes in reverse topological order, and highlights upstream outputs as candidate bottlenecks for developer inspection. Third, PROTEA provides localized, adoptable fixes: for a developer-selected node, it proposes targeted prompt revisions in editable before/after comparisons grounded in the evaluator rationale and node rubric. After the developer adopts (or edits) the prompt revision, PROTEA automatically reruns and re-evaluates the workflow and shows before/after comparisons and score histories in the same interface, enabling rapid iteration without tool switching. We evaluate PROTEA through two production-adjacent workflow studies, an automatic iteration stress test, and a formative user study with six experienced LLM developers. The workflow studies improved document-inspection accuracy from 64.3% to 83.9% and recommendation Hit@5 from 0.30 to 0.38; the user study suggests that developers value graph-level localization, node-level rationales, and editable before/after prompt revisions for offline debugging and refinement. Figure 2: Representative PROTEA interface for inspecting and debugging a multi-agent workflow. The main view shows the workflow graph and node-level evaluation states. Red/yellow nodes indicate fail / warn states and guide inspection. The inspection panel shows details for the selected node, including its prompt, evaluation result, suggested revision, and expanded prompt. Developers can inspect connections, review node behavior, and revise prompts while monitoring workflow-level effects. 2 System Overview PROTEA supports offline refinement of multi-agent LLM workflows represented as directed acyclic graphs (DAGs). Each node corresponds to a role-specific LLM call (or a tool-augmented agent) with its own prompt and evaluation criteria, and edges represent intermediate outputs passed between agents. The system is organized around a fixed-suite iteration loop: run the current workflow, evaluate intermediate node outputs, inspect highlighted nodes, revise the selected node’s prompt, and rerun the workflow to check regressions. 2.1 Workflow setup and saved runs A PROTEA project combines (i) a workflow specification (loaded from a saved project or imported from LangGraph), (ii) an offline test suite of queries with optional final-answer references, and (iii) per-node evaluator settings (rubrics, judge prompts,