Agentified Assessment of Logical Reasoning Agents
Zhiyu Ni, Yifeng Xiao, Zheng Liang · 2026
Read Agentified Assessment of Logical Reasoning Agents by Zhiyu Ni, Yifeng Xiao, Zheng Liang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-02788.txt.
Introduction
Zhiyu Ni, Yifeng Xiao, and Zheng Liang build an agent whose job is to grade other agents fairly. Their 2026 framework tackles a practical problem: tests of reasoning agents must be repeatable, auditable, and able to survive crashes. The tested agent only needs a standard agent-to-agent interface.
An assessor agent hands out tasks, enforces time and step budgets, parses answers, and logs structured failure types. As a case study, the team graded an agent that turns plain-language logic problems into solver programs. On a cleaned, solver-checked set of first-order logic questions, that agent scored well above a chain-of-thought baseline. The setup shows how automated grading can stay rigorous without trusting the agent's own report.
Catalog record
| CREATOR | Zhiyu Ni, Yifeng Xiao, Zheng Liang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 12718 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.02788
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Agentified Assessment of Logical Reasoning Agents Zhiyu Ni, Yifeng Xiao, Zheng Liang Abstract We present a framework for evaluating and benchmarking logical reasoning agents when assessment itself must be reproducible, auditable, and robust to execution failures. Building on agentified assessment, we use an assessor agent to issue tasks, enforce execution budgets, parse outputs, and record structured failure types, while the agent under test only needs to expose a standardized agent-to-agent interface. As a case study, we benchmark an auto-formalization agent for first-order logic (FOL) reasoning on a solver-verified and repaired split of FOLIO. The agent translates natural language premises and conclusions into executable Z3Py programs and employs satisfiability modulo theories (SMT) solving to determine logical entailment. On the cleaned FOLIO validation set, the auto-formalization agent achieves 86.70 % 86.70\% accuracy under the assessor protocol, outperforming a chain-of-thought baseline ( 73.89 % 73.89\% ). 1 Introduction Evaluating and benchmarking reasoning agents is challenging because failures occur at multiple layers, including model reasoning and tool execution. As a result, static evaluation harnesses often conflate operational failures (e.g., timeouts, runtime errors, output parsing failures) with reasoning errors, and may hide these failure modes behind a single accuracy number. Moreover, traditional setups tightly couple benchmark logic to agent implementations, so integration effort grows with the number of benchmarks. In this work, we study first-order logic (FOL) inference on FOLIO ( 6 ) and adopt agentified agent assessment (AAA) as described in AgentBeats ( 2 ) , where assessment logic is implemented as an assessor agent that interacts with an agent under test through a standardized agent-to-agent (A2A) interface ( 1 ) . We conduct verification and repair on the FOLIO dataset ( 6 ) using a data cleaning pipeline, establishing a more reliable benchmark. On the cleaned benchmark, we implement a white-box auto-formalization agent that achieves 86.70% accuracy on the FOLIO validation set, outperforming a chain-of-thought baseline (73.89%) ( 10 ) . 2 Benchmark and Data Cleaning We evaluate on FOLIO ( 6 ) , a first-order logic (FOL) reasoning benchmark consisting of natural-language premises and conclusions paired with formal FOL annotations. FOLIO is derived from Wikipedia-based real-world scenarios and requires both natural language understanding and logical reasoning. For each example, FOLIO provides one of three labels: True (the conclusion is logically entailed by the premises), False (the conclusion contradicts the premises), or Uncertain (the conclusion cannot be determined from the premises). We evaluate classification accuracy against human-annotated ground-truth labels. 2.1 Verification and Repair Pipeline The original FOLIO dataset exhibits potential label errors and misalignments between natural language and formal annotations due to the complexity of semantic parsing. To establish a more reliable benchmark, we implement a systematic data cleaning pipeline (Figure 1 ) that leverages symbolic verification using the Vampire theorem prover ( 7 ) to execute formal reasoning on FOL representations. When verification results conflict with expected labels, we identify and iteratively repair translation errors. Figure 1: Overview of the data cleaning pipeline. Given a premise set \{\phi_{1},\phi_{2},\dots\} and a conclusion \varphi , we detail the pipeline as follows: Formal Premise-Conclusion Checking . We verify the implication relation between premises and conclusion in two steps. First, we check whether the premises are consistent (i.e., whether \bigwedge_{i}\phi_{i} is satisfiable). Then, we perform implication checking using the following satisfiability conditions: True \displaystyle\iff\bigwedge_{i}\phi_{i}\rightarrow\varphi\iff\bigwedge_{i}\phi_{i}\wedge\neg\varphi\text{ is unsatisfiable}, (1) False \displaystyle\iff\bigwedge_{i}\phi_{i}\rightarrow\neg\varphi\iff\bigwedge_{i}\phi_{i}\wedge\varphi\text{ is unsatisfiable}. (2) When both implication checks fail, we label the instance as Uncertain . We then compare verification results against expected labels to identify potential annotation errors. NL-FOL Misalignment Identification and Repair. When premises are inconsistent or verification results conflict with expected labels, we employ a check-and-fix procedure using two LLM-based agents. A critique agent diagnoses systematic translation errors (e.g., unbalanced parentheses, lexical typos, and naming convention inconsistencies) by analyzing the original natural language and FOL annotations. A refiner agent then executes targeted corrections. We iteratively re-verify the repaired FOL until the expected labels are achieved. If the number of iterations exceeds a predefined threshold without resolution, we flag the instance for manual review. We release the cleaned and repaired FOLIO split at https://huggingface.co/datasets/yfxiao/folio-refined . 2.2 Benchmark Statistics We classify instances into three categories: (i) verified directly with the original FOL, (ii) verified after automatic repair, and (iii) problematic instances flagged for manual review. For the training set (1,001 examples): 674 ( 67.3\% ) verified directly, 23 ( 2.3\% ) after automatic repair, and 304 ( 30.4\% ) remain problematic. For the validation set (203 examples): 154 ( 75.9\% ) verified directly, 10 ( 4.9\% ) after automatic repair, and 39 ( 19.2\% ) remain problematic. In total, we identify 3.8\% potential label errors in the training set and 1.5\% in the validation set, improving both the NL-FOL translation quality and the reliability of labels for evaluation. 3 Agentified Evaluation Framework 3.1 Agentified Assessment We formalize the evaluation of reasoning agents by treating assessment itself as an agent . Instead of relying on a static evaluation script, we separate the system into two interacting components: an agent under test , which performs reasoning, and an assessor agent , which controls task execution, interprets outcomes, and assigns scores. We adopt the agentified agent assessment (AAA) abstraction described in AgentBeats ( 2 ) , where the assessor and the agent under test communicate through an agent-to-agent (A2A) interface ( 1 ) . The primary value of AAA is not to address individual operational issues (e.g., timeouts), which can also be handled in traditional benchmarks. Instead, it changes the integration cost model for benchmarking agents: in traditional setups, integration cost grows with the number of benchmarks ( O(n) ), whereas under AAA, an agent implements A2A once and can participate in many assessors ( O(1) ). This decoupling enables plug-and-play evaluation and architecture freedom for agents with diverse internal designs. Figure 2: Traditional evaluation harnesses couple task execution, environments, and judging logic to a preset harness. In agentified assessment, an assessor agent evaluates an agent under test via an A2A interface ( 1 ) , reducing integration overhead. As illustrated in Figure 2 , traditional benchmarks couple task orchestration and judging to a benchmark-specific harness. AAA decouples these components by packaging evaluation logic as an assessor agent that interacts with an agent under test through a standardized interface, allowing the assessor to evolve without breaking participants. 3.2 Assessor Agent (Evaluation Protocol) The assessor agent implements the evaluation protocol. For each premise and conclusion pair, it issues the task to the agent under test, enforces execution budgets (e.g., timeouts and retries), parses and validates the response, and assigns a final label ( True , False , or Uncertain ) according to the benchmark criteria. If the agent under test outputs additional text, the assessor parses the final label deterministically; non-parseable outputs are recorded as ParseError . Rather than discarding failures, the assessor records structured failure types such as Timeout , RuntimeError , and ParseError . Finally, the assessor emits a machine-consumable evaluation artifact containing per-instance records (gold label, predicted label, correctness, error type if any, latency) and aggregate metrics. 3.3 Reasoning Agents Under Test We benchmark two agents under the same assessor. Chain-of-thought baseline. We use chain-of-thought prompting ( 10 ) : the agent is instructed to reason step by