QualityFlow: An Agentic Workflow for Program Synthesis Controlled by LLM Quality Checks

Yaojie Hu, Qiang Zhou, Qihong Chen, Xiaopeng Li, Linbo Liu, Dejiao Zhang, Amit Kachroo, Talha Oz, Omer Tripp · 2025

Read QualityFlow: An Agentic Workflow for Program Synthesis Controlled by LLM Quality Checks by Yaojie Hu, Qiang Zhou, Qihong Chen, Xiaopeng Li, Linbo Liu, Dejiao Zhang, Amit Kachroo, Talha Oz, Omer Tripp (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-17167.txt.

Introduction

Yaojie Hu and eight co-authors present, in a 2025 paper, a team of AI agents that writes computer programs under quality control. Given a problem stated in English plus a set of unit tests, the system must produce code that passes. Its agents play the roles of a software team: writing code, testing it, and debugging their own work.

The new piece is a quality checker that imagines running the code against the tests before anything is run for real. Its verdicts steer the flow: send the answer, restate the problem, or step back and redo earlier work. The checker accepts correct programs reliably and shields the team from faulty self-made tests. The system set top scores on four standard coding benchmarks.

Catalog record

CREATORYaojie Hu, Qiang Zhou, Qihong Chen, Xiaopeng Li, Linbo Liu, Dejiao Zhang, Amit Kachroo, Talha Oz, Omer Tripp
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE58 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-17167

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

QualityFlow: An Agentic Workflow for Program Synthesis Controlled by LLM Quality Checks Yaojie Hu, Qiang Zhou, Qihong Chen, Xiaopeng Li, Linbo Liu, Dejiao Zhang, Amit Kachroo, Talha Oz, Omer Tripp Abstract We introduce QualityFlow, a dynamic agentic workflow for program synthesis. Given the English description of a programming problem and a set of unit tests, the model’s goal is to synthesize the correct program that solves the problem and passes the tests. QualityFlow includes large language model (LLM) agents resembling a software development team, including code generation, testing, and self-debugging. We propose the LLM Quality Checker, which explicitly “imagines” whether the synthesized programs’ execution would conform to the unit tests. The Quality Checks dynamically control the workflow, including actions to submit the final answer, clarify the problem statement, and revert previous workflow steps. Our experiments show that the Quality Checker can precisely accept any correct program, mitigate faulty synthesized tests, and prevent potential workflow deviation. QualityFlow establishes the state-of-the-art results on four program synthesis benchmarks: MBPP, HumanEval, and stricter evaluations from MBPP-EvalPlus and HumanEval-EvalPlus. 1 Introduction Problem statement: Write a function to remove the first and last occurrence of a given character from the string. Visible unit tests: ⬇ 1 assert remove_Occ ( "hello" , "l" ) == "heo" 2 assert remove_Occ ( "abcda" , "a" ) == "bcd" 3 assert remove_Occ ( "PHP" , "P" ) == "H" Figure 1: An example problem from the MBPP code generation benchmark (No. 11). How does the model know that the generated code will pass the unit tests? Existing code generation systems assume this implicitly. Program synthesis is a long-standing goal of software engineering research Manna and Waldinger (1971) ; Summers (1977) , with a history dating back to the 1940s and 50s Backus et al. (1957) . Recently, large language models (LLMs) for program synthesis have created real-world applications that streamline software development and enhance programmer productivity Wang et al. (2021) ; DeepSeek (2023) ; Lu et al. (2021) . Outperforming zero-shot synthesis with no prohibitive costs to train the LLMs Kaplan et al. (2020) ; Wei et al. (2022) , Agentic Workflow has recently been proposed as a way to orchestrate multiple LLM agents that form a team to self-reflect Shinn et al. (2024) , debate Khan et al. (2024) , and solve the problem together in a collaborative style Hong et al. (2024) . Agentic Workflows provide fertile opportunities to incorporate ongoing software engineering research, such as self-debugging Chen et al. (2024b) ; Huang et al. (2024) and automatic test generation Chen et al. (2024c) ; Hanford (1970) , for program synthesis systems with all software engineering roles. We focus on a program synthesis setting that captures the typical software engineering process: given the natural language description (documentation) together with some visible unit tests, the model’s goal is to generate a program that solves the problem and passes the evaluation tests (possibly the same as the visible tests) (Figure 1 ). Program synthesis benchmarks today (e.g. MBPP, HumanEval Austin et al. (2021a) ; Chen et al. (2021a) ; Hendrycks et al. (2021) ) follow this setting and are incorporated into the standard evaluation suites of large language models (Table 3 ). Existing program synthesis methods have the following limitations: • Assumption of visible unit test conformity. Existing methods typically assume that the generated code will follow the visible unit tests required by the problem statement (Figure 1 ). • Bottleneck of synthesized test quality. Execution of synthesized tests can provide feedback for self-debugging Huang et al. (2024) , but incorrect tests may raise unwarranted errors and mislead the self-debugger to turn a correctly synthesized programs into an incorrect one. • Deviation of self-debugging trajectory. Repeated self-debugging steps may fail to improve intermediate synthesis: it could become stuck in a loop, reach a fixed point, or simply degrade with more bugs and mistakes. Existing multi-agent methods are static workflows and fail to explore flexible control flows during code generation Huang et al. (2024) . In this paper, we aim to address these limitations and introduce QualityFlow, an Agentic Workflow for program synthesis that resembles a software engineering team with quality assurance at every level. QualityFlow includes a Code Generator agent that drafts the program, a Test Designer agent that synthesizes unit tests, and a Self-Debugger agent that iteratively debugs the program based on error messages from failed unit tests. The agent that controls QualityFlow is the Quality Checker, who decides to invoke other agents: if an intermediate program passes the quality check, it is submitted as the final solution; otherwise, a selected agent continues to synthesize the next intermediate program. Quality checks are predicted by Imagined Execution , an LLM chain-of-thought Wei et al. (2023) self-reflection method that emulate the execution output and compares it with output asserted by the unit test. If the outputs are equal, then the quality check predicts to accept, thereby explicitly examining unit test conformity. Quality checks can mitigate the adverse impact of incorrectly synthesized tests during self-debugging by detecting incorrect programs. After self-debugging, if the Quality Checker still does not accept the program, it indicates that the self-debugging trajectory likely failed to reach a good result, and a Clarifier agent will explain the problem statement to remove misunderstandings and restart the Code Generator for one more attempt. Finally, if the Quality Checker still rejects, it is likely that the workflow trajectory has fatally deviated, and all self-debugging and clarification steps will be reverted. QualityFlow effectively addresses limitations above and achieves the state-of-the-art (SOTA) program synthesis performance on all benchmarks that we evaluate on, including MBPP Austin et al. (2021a) , HumanEval Chen et al. (2021a) , and more extensive evaluations from EvalPlus for both Liu et al. (2023) . The contributions of our paper are as follows: • We introduce QualityFlow, an Agentic Workflow with the SOTA program synthesis performances on four benchmarks: MBPP, MBPP-EvalPlus, HumanEval, and HumanEval-EvalPlus. • We introduce Imagined Execution, a self-reflection method tailored for the program synthesis domain where LLMs predict the correctness of synthesized program through emulated execution with Chain-of-Thought reasoning and explicitly checks for unit test conformity. • We introduce Quality Checker, a controller agent that selects correctly produced programs and navigates the workflow, making control flow decisions—including continuing, restarting, or reverting—based on Imagined Execution. • We study the bottleneck of synthesized tests and introduce Test Quality Checker, extending the idea of quality checks to filter out incorrect tests and improve workflow performance. • We introduce Diversified Prompting that uses a diverse set of prompts in parallel to maximize the possibility that a correct solution is produced and accepted by the quality checker. 2 Related Work Code LLM Large language models (LLMs) have emerged as a powerful tool for various code-related tasks, including program synthesis Nijkamp et al. (2022) ; Zheng et al. (2023) ; Daniel et al. (2023) ; Rozière et al. (2023) ; Jiang et al. (2023) ; Lozhkov et al. (2024) , bug fixing Hossain et al. (2024) , program testing Kang et al. (2023) , and fuzzing Deng et al. (2023) . Through extensive pre-training, they recognize patterns, comprehend context, and generate coherent and contextually relevant code snippets. Agentic workflow Generation AI has moved

Read the complete text