LLM Assisted Coding with Metamorphic Specification Mutation Agent DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/18/06
Mostafijur Rahman Akhond, Gias Uddin · 2025
Read LLM Assisted Coding with Metamorphic Specification Mutation Agent DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/18/06 by Mostafijur Rahman Akhond, Gias Uddin (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-18249.txt.
Introduction
Mostafijur Rahman Akhond and Gias Uddin use a test trick to make code writing by large language models more sound. Vague user specs lead models to give mixed or wrong code. Their agent, CodeMetaAgent, uses paired rules that say when two inputs should count as the same task in new words.
Those rules guide the agent to sharpen the spec and build test cases that stay true to the task sense. Trials run on three code test sets with four large models. Scores rise by up to about a sixth, and test cover nears full in the best runs. The work shows a classic test idea can curb model drift when it steers the prompt, not just the check at the end.
Catalog record
| CREATOR | Mostafijur Rahman Akhond, Gias Uddin |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 64 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-18249
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
LLM Assisted Coding with Metamorphic Specification Mutation Agent DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/18/06 Mostafijur Rahman Akhond, Gias Uddin Abstract . Metamorphic Relations (MRs) serve as a foundational mechanism for generating semantically equivalent mutations. Software engineering has advanced significantly in recent years with the advent of Large Language Models (LLMs). However, the reliability of LLMs in software engineering is often compromised by ambiguities and inconsistencies due to improper user specification. To address this challenge, we present CodeMetaAgent (CMA) , a metamorphic relation–driven LLM-agent that systematically refines task specifications and generates semantically constrained test cases. Our proposed framework uses MRs with LLMs to improve generation consistency and reduce variability caused by specifications, unlike the traditional use of MRs as post validations. Our framework has been experimented on the HumanEval-Pro, MBPP-Pro, and SWE-Bench_Lite datasets with the GPT-4o, Mistral Large, GPT-OSS, and Qwen3-Coder models. It improved code generation accuracy by up to 17% and gained code coverage by up to 99.81%. These results show that metamorphic relations can be a simple but effective guide in assisting LLM-based software development. 1. Introduction Metamorphic Relations (MRs) provide a structured framework for the transformation of inputs or specifications ( 27 ; 19 ) . MRs capture semantic invariants that should hold across multiple representations of a task. It enables the validation or refinement of software even in the absence of ground-truth oracles. This property makes MRs especially useful for Large Language Models (LLMs), where code generation ( 16 ; 7 ; 9 ) , bug fixing ( 15 ; 34 ) , and test synthesis ( 12 ; 4 ; 26 ) are very sensitive to changes from the user descriptions. LLMs can produce more accurate, consistent, and robust outputs across a wide range of software engineering tasks by using problem specifications to logically equivalent mutations. Large Language Models (LLMs) like GPT ( 1 ) , Qwen ( 37 ) , and Mistral ( 3 ) have demonstrated that they can translate natural language specifications into executable code, produce test cases corresponding to the source codes, and identify and resolve bugs from the program ( 9 ; 39 ; 16 ) . Even though these models work well, the outputs they give can be very different depending on how the input is presented or what assumptions are made. Minor linguistic variations can result in divergent reasoning pathways and lead to inconsistencies ( 8 ; 28 ) . Recent prompt-based refinement or retrieval-augmented methods try to fix this problem, but they struggle to make the model robust against ambiguous specifications. This gap shows that we need a systematic way for LLMs to think about multiple aligned ways to solve a problem, instead of just one static prompt. We present CodeMetaAgent (CMA) , a metamorphic relation–driven agent that enhances LLM-based software development through systematic specification refinement and validation. Instead of using a single static prompt, CMA uses metamorphic relations as a semantic operator that creates and evaluates mutations. This enables LLMs to explore different ways of reasoning, which makes different Software Engineering (SE) related tasks, for example, code generation, or test case synthesis, more robust. By integrating transformation, validation, generation, and execution within a unified pipeline. In the CMA framework, metamorphic relations are facilitated to enhance both code generation and test case synthesis. For the code generation tasks, MRs are applied to reformulate the original problem description into multiple semantically equivalent variants. These variants expose several reasoning perspectives, enabling LLMs to generalize beyond a single prompt and produce more accurate and reliable implementations. For the test cases generation, the relations encode expected behavioral properties over input–output transformations. This allows the automatic derivation of test cases covering divergent conditions and edge cases. CMA sets up a way to improve robustness and coverage in LLM-assisted software development by making it easier to generate and validate these two types of mutations. In this work, we make three key contributions. First, we introduce CMA , a unified agentic framework that leverages metamorphic relations as proactive semantic operators, transforming MRs from post-hoc validation mechanisms to core drivers of code and test generation. Second, we formalize the role of metamorphic relations across two fundamental dimensions of software engineering—code generation and test case generation, defining task-specific transformations that enable systematic refinement with semantic alignment. Third, we show that our framework significantly improves code generation accuracy (up to 17% improvement), and achieves test coverage as high as 99.81% through extensive experiments on three widely used benchmarks (HumanEval Pro, MBPP Pro, and SWE-Bench-Lite) using state-of-the-art LLMs. These results establish metamorphic relation–guided agentic reasoning as a principled approach for improving the robustness, reliability, and consistency of LLM-based software development. Ensuring semantic robustness in LLM-assisted software engineering is increasingly critical as these models are integrated into practical development workflows. This research investigates the systematic enhancement of reasoning reliability, accuracy, and comprehensiveness of LLMs through metamorphic relations in essential software engineering tasks. We focus on two top-level research questions: (1) can metamorphic relations make LLM better at generating code and fixing bugs, and (2) can they make LLM-generated test cases more useful and cover more edge cases? The rest of the paper discusses the Background, CMA framework, the experimental setup, and the empirical evaluation to demonstrate its effectiveness. 2. Background & Related Work 2.1. Metamorphic Relations in Software Engineering A metamorphic relation (MR) is defined as how the output of a function ought to change when its input is systematically altered ( 41 ; 21 ) . Instead of verifying whether a single output is correct in isolation, an MR defines a relationship between the outputs corresponding to inputs. Consider the function f(x)=x^{2} as an example. If we double the input from x to 2x , the output should increase by a factor of four, since f(2x)=(2x)^{2}=4x^{2}=4f(x) . This consistent behavior under a specific input transformation is a metamorphic relation. Metamorphic relations (MRs) have played a central role in improving software engineering activities. The application of MRs in software engineering ranges from reliability, testing, and verification ( 10 ; 19 ; 41 ) . Traditionally, MRs were employed as part of metamorphic testing (MT), a methodology that validates program correctness without requiring test oracles. By expressing expected relationships between multiple executions of a program with transformed inputs, MRs enable testers to detect inconsistencies that indicate latent errors. Beyond their role in testing, recent work has explored MRs as semantic specifications ( 35 ; 32 ) . This enables automated reasoning about the software behavior in scenarios where explicit expected outputs are not available. The adoption of metamorphic relations in software testing workflows has gained momentum in recent years. Xu et al. ( 35 ) proposed a framework for automatically discovering MRs to support black-box testing. In another work, Hoard et al. ( 14 ) investigated developer acceptance of MR-based testing in industrial settings. They emphasized the need for agents in automation. These studies show that MRs improve fault detection and developer productivity. However, they still treat MRs as secondary validation mechanisms, rather than primary operators that actively drive the software development process. In contrast, in this work, we presented MRs as semantic operators