CodeARC: Benchmarking Reasoning Capabilities of LLM Agents for Inductive Program Synthesis

Anjiang Wei, Tarun Suresh, Jiannan Cao, Naveen Kannan, Yuheng Wu, Kai Yan, Thiago S. F. X. Teixeira, Ke Wang, Alex Aiken · 2025

Read CodeARC: Benchmarking Reasoning Capabilities of LLM Agents for Inductive Program Synthesis by Anjiang Wei, Tarun Suresh, Jiannan Cao, Naveen Kannan, Yuheng Wu, Kai Yan, Thiago S. F. X. Teixeira, Ke Wang, Alex Aiken (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-23145.txt.

Introduction

Wei and eight co-authors built a benchmark for testing whether AI agents can infer a hidden program's behavior in twenty twenty-five. Inductive program synthesis means writing a function that generalizes from input-output examples. Language model agents do well when guided by prose instructions, but their example-driven reasoning was untested by static benchmarks.

Their framework, CodeARC, lets an agent query a hidden target function with fresh inputs, propose candidate functions, and refine them using a differential testing oracle that reports mismatches. The benchmark covers one thousand one hundred fourteen functions. Among eighteen models, the best reached only about fifty-three percent success, showing the task's difficulty. Fine-tuning an open model on curated synthesis traces gained up to thirty-one percent relative improvement. Code and data were released publicly.

Catalog record

CREATORAnjiang Wei, Tarun Suresh, Jiannan Cao, Naveen Kannan, Yuheng Wu, Kai Yan, Thiago S. F. X. Teixeira, Ke Wang, Alex Aiken
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE41 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-23145

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

CodeARC: Benchmarking Reasoning Capabilities of LLM Agents for Inductive Program Synthesis Anjiang Wei, Tarun Suresh, Jiannan Cao, Naveen Kannan, Yuheng Wu, Kai Yan, Thiago S. F. X. Teixeira, Ke Wang, Alex Aiken Abstract Inductive program synthesis, or programming by example, requires synthesizing functions from input-output examples that generalize to unseen inputs. While large language model agents have shown promise in programming tasks guided by natural language, their ability to perform inductive program synthesis is underexplored. Existing evaluation protocols rely on static sets of examples and held-out tests, offering no feedback when synthesized functions are incorrect and failing to reflect real-world scenarios such as reverse engineering. We propose CodeARC, the Code Abstraction and Reasoning Challenge , a new evaluation framework where agents interact with a hidden target function by querying it with new inputs, synthesizing candidate functions, and iteratively refining their solutions using a differential testing oracle. This interactive setting encourages agents to perform function calls and self-correction based on feedback. We construct the first large-scale benchmark for general-purpose inductive program synthesis, featuring 1114 functions. Among 18 models evaluated, o3-mini performs best with a success rate of 52.7%, highlighting the difficulty of this task. Fine-tuning LLaMA-3.1-8B-Instruct on curated synthesis traces yields up to a 31% relative performance gain. CodeARC provides a more realistic and challenging testbed for evaluating LLM-based program synthesis and inductive reasoning. Our code, data, and models are publicly available at https://github.com/Anjiang-Wei/CodeARC † † footnotetext: Correspondence to: [email protected] 1 Introduction Inductive reasoning, i.e., the ability to identify patterns and form abstractions from limited examples, is widely recognized as a fundamental aspect of human intelligence ( Hayes et al., 2010 ; Yan et al., 2025 ) . In the context of programming, inductive reasoning underpins the task of synthesizing functions that satisfy given input-output examples and generalize to unseen inputs. This task, commonly referred to as inductive program synthesis or programming by example ( Manna & Waldinger, 1971 ; Myers, 1986 ; Feser et al., 2023 ; Li & Ellis, 2025 ) , has broad application domains ( Wang et al., 2017 ; Deng et al., 2024 ) . Recent advances in Large Language Models (LLMs) have led to the emergence of autonomous agents capable of decision-making, multi-step planning, tool use, and iterative self-improvement through interaction and feedback ( Chen et al., 2023 ; Liu et al., 2023e ; Liu et al., 2024b ; Guo et al., 2024 ; Xi et al., 2025 ) . While much of the existing work focuses on programming tasks guided by natural language ( Chen et al., 2021 ; Austin et al., 2021 ; Jimenez et al., 2023 ; Jain et al., 2024b ) , we study a fundamentally different problem: inductive program synthesis, where the objective is to infer the target program solely from input-output examples. This setting provides a more rigorous test of inductive reasoning capabilities, as it eliminates natural language descriptions that can trigger retrieval-based completions memorized during model training. Designing an effective evaluation protocol for inductive program synthesis with LLMs is inherently challenging, as multiple valid functions may satisfy a given set of input-output examples (as demonstrated in Section 5.2 ). The state-of-the-art protocol ( Li & Ellis, 2025 ) , which evaluates synthesized functions on held-out test cases after presenting 10 fixed input-output examples, has several notable limitations. First, a small, static set of input-output examples may underspecify the target functions, especially for those with complex logic. Moreover, held-out tests may fail to reveal subtle semantic discrepancies between the generated and intended implementations. In addition, when the model produces an incorrect solution, it receives no feedback and has no opportunity to revise or explore alternatives. Finally, existing benchmarks for inductive program synthesis ( Gulwani, 2011 ; Wang et al., 2017 ; Wong et al., 2021 ; Deng et al., 2024 ; Li & Ellis, 2025 ) are focused on domain-specific tasks and do not assess the ability of LLMs to synthesize functions written in general-purpose programming languages. Figure 1 : Overview of CodeARC. Our framework evaluates LLMs’ reasoning capabilities in inductive program synthesis. The agent begins with input-output examples, interacts with a hidden target function via function calls, and uses a differential testing oracle to check the correctness of the synthesized function for self-reflection and refinement. To address the limitations of existing evaluation protocols for inductive program synthesis, we introduce CodeARC, the Code Abstraction and Reasoning Challenge , inspired by real-world scenarios such as decompilation and reverse engineering ( Lin et al., 2010 ; Mantovani et al., 2022 ; Al-Kaswan et al., 2023 ) . In such settings, an agent is given a binary executable (without source code) and must synthesize equivalent source code by observing input-output behavior. Instead of relying on a fixed dataset, the agent can query the binary with new inputs, invoke a differential testing oracle, and use counterexamples for iterative refinement. This setup parallels the classic learnability framework of queries and counterexamples ( Angluin, 1987 ; De la Higuera, 2010 ) , here applied to program synthesis. Figure 1 illustrates how CodeARC instantiates this process: LLM-based agents begin with an initial set of input-output examples, query the ground-truth function for more examples, and debug synthesized code using a differential testing oracle. We impose fixed budgets on both the number of observable input-output examples and the number of testing oracle invocations for self-debugging. The task requires agents to proactively generate inputs (function calls) and revise solutions based on feedback (self-reflection). This interactive setup offers a more realistic alternative to prior static evaluation protocols. We construct the first comprehensive dataset for general-purpose inductive program synthesis, featuring 1114 functions with initial input-output examples. Our benchmark targets general programming tasks and employs two state-of-the-art differential testing tools ( Lukasczyk & Fraser, 2022 ; Etemadi et al., 2024 ) for correctness evaluation. Our experiments demonstrate that CodeARC poses a significant challenge for LLM-based inductive program synthesis. Among the 18 models evaluated, OpenAI o3-mini performs the best overall, yet only achieves 52.7% success rate. We further conduct ablation studies to analyze how budgets on the number of input-output examples and oracle calls affect model performance. To enhance model capabilities, we generate synthetic fine-tuning data with curated synthesis traces that capture the reasoning steps. We show that supervised fine-tuning on LLaMA-3.1-8B-Instruct yields up to a 31% relative performance improvement. In summary, our contributions are as follows: • Interactive evaluation protocol for inductive program synthesis. We introduce a setup where agents start with fixed input-output examples but can generate new inputs to query ground-truth functions and invoke a differential testing oracle to self-correct their solutions. This setup brings the task closer to a real-world setting, e.g. reverse-engineering. • General-purpose benchmark with extensive evaluation. We construct the first large-scale, general-purpose benchmark for this task, including 1114 diverse functions. Among the 18 models evaluated, o3-mini achieves the best overall performance but still only reaches a success rate of 52.7%. • Synthetic data and fine-tuning. To boost model performance, we generate synthetic fine-tuning data containing curated synthesis traces that capture both function invocations and reasoning steps. We show that fine-tunining on LLaMA-3.1-8B-Instruct yields up to a 31% relative performance improvement. 2 Related Work Inductive Program Synthesis Traditional inductive program synthesis methods rely solely on input-output examples,

Read the complete text