Understanding Specification-Driven Code Generation with LLMs: An Empirical Study Design

Giovanni Rosa † , David Moreno-Lumbreras † , Gregorio Robles † , Jesús M. González-Barahona † · 2026

Read Understanding Specification-Driven Code Generation with LLMs: An Empirical Study Design by Giovanni Rosa † , David Moreno-Lumbreras † , Gregorio Robles † , Jesús M. González-Barahona † (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-03878.txt.

Introduction

Giovanni Rosa and three co-authors lay out an empirical study of how developers generate code with language models when a structured specification process guides them. Models are entering real development workflows, but their behavior inside disciplined, specification-driven work is poorly understood. Their tool, CURRANTE, is a Visual Studio Code extension that keeps a human in the loop through three stages: specification, tests, and function.

Participants will solve medium-difficulty problems from the LiveCodeBench dataset while the tool logs interactions in fine detail. The team will record effectiveness measures like pass rates, efficiency measures like time to pass, and how often people iterate on specifications and tests. The goal is to learn how human refinement of specs and tests changes the quality and dynamics of generated code, informing both tool design and team practice.

Catalog record

CREATORGiovanni Rosa † , David Moreno-Lumbreras † , Gregorio Robles † , Jesús M. González-Barahona †
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE31 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-03878

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Understanding Specification-Driven Code Generation with LLMs: An Empirical Study Design * Thanks: * This paper is a Stage 1 Registered Report. The study protocol and analysis plan were peer reviewed and accepted at SANER 2026 with a Continuity Acceptance (CA) score for Stage 2. Giovanni Rosa † , David Moreno-Lumbreras † , Gregorio Robles † , Jesús M. González-Barahona † Abstract Large Language Models (LLMs) are increasingly integrated into software development workflows, yet their behavior in structured, specification-driven processes remains poorly understood. This paper presents an empirical study design using CURRANTE , a Visual Studio Code extension that enables a human-in-the-loop workflow for LLM-assisted code generation. The tool guides developers through three sequential stages—Specification, Tests, and Function—allowing them to define requirements, generate and refine test suites, and produce functions that satisfy those tests. Participants will solve medium-difficulty problems from the LiveCodeBench dataset, while the tool records fine-grained interaction logs, effectiveness metrics (e.g., pass rate, all-pass completion), efficiency indicators (e.g., time-to-pass), and iteration behaviors. The study aims to analyze how human intervention in specification and test refinement influences the quality and dynamics of LLM-generated code. The results will provide empirical insights into the design of next-generation development environments that align human reasoning with model-driven code generation. I Introduction Test-Driven Development (TDD) has long been a fundamental paradigm in software development, but the introduction of Large Language Models (LLMs) is paving the way for new approaches to coding assistance, providing a potential transformation in how developers write and verify code. The current most-used assistance for code generation is provided via code completion plugins, such as GitHub Copilot [ 17 ] , or fully-featured AI-powered IDEs, such as Cursor. 1 1 1 https://cursor.com/ . These tools offer powerful automation of the coding process leaveraging natural language instructions, shifting from the traditional code-centric development approach to a more abstract specification-driven process, namely Spec-Driven Development (SDD) [ 2 ] . Stil, they lack a structured approach to requirements specification and testing that are crucial factors for ensuring code quality as highlighted in several studies on the integration of LLMs with TDD workflows [ 9 , 6 , 11 , 14 ] . This prior studies validated these TDD workflows leaving the human factor underexplored. In fact, this presents a gap in understanding the user’s role in guiding the LLM through structured requirements specification and test case definition, and, more importantly, how effectively they can express their intent through the overalll TDD-based workflow. In this paper, we propose a an empirical experiment to investigate how developers can effectively interact with such a workflow, focusing more on how they can express their intent through the specification and test case definition phases. To this end, we rely on Currante , a plugin that implements a typical TDD-code generation workflow based on the existing literature [ 9 , 6 , 14 ] , within a popular IDE, Visual Studio Code (VSCode). Currante mainly consists of a three-phase TDD workflow: the problem description phase, (1) where the user inputs the specification, (2) the test cases phase, where the user guides the LLM in generating and refining a test suite that formally describes the requirements. (3) the final phase, i.e., code generation , is delegated entirely to the LLM, which generates the code and verifies its execution results against the test suite. We plan a controlled experiment with human participants, asked to solve a set of programming problems from the LiveCodeBench dataset. Working exclusively within Currante and VS Code, participants will be asked to produce a valid solution for the proposed problem. We will collect detailed data on effectiveness ( e.g., PassAll , PassRate , TestCoverage , TestDiversity ), efficiency ( e.g., TimeToPass ), and interactions ( e.g., TestEdits , SuiteRegenerations ) in order to analyze the real impact of user involvement in the test curation phase on the overall code generation success. The expected outcome of our study is twofold: first, to provide empirical evidence on the effectiveness of a TDD-based workflow for LLM-assisted code generation focusing on the user perspective and second, to inform the design of future IDEs, clarifying the trade-off between investing in requirements engineering and code refinement, and how the user can best contribute to the process. The rest of the paper is structured as follows. Section II provides background information and related works. Sections IV and VI presents Currante and the proposed experiment, while Section VII discusses threats to validity and Section VIII summarizes the conclusions. II Background and Related Work A significant contribution in this area was made by Chen et al. [ 3 ] , who introduced the Codex model and an evaluation protocol defining the pass@k metric, the current standard, and the HumanEval benchmark, consisting of coding problems with associated unit tests. Other research has examined how LLMs can be combined with established software engineering methodologies to improve code quality and reliability, such as TDD. Mathews et al. [ 9 ] proposed the TGen approach, which implements a TDD-guided code generation flow using LLMs. Starting from a set of initial test cases, TGen generates code and iteratively updates it until all tests pass. Their experiments showed that integrating TDD workflows enhances both the accuracy and dependability of code produced by LLMs. Along the same lines, Fakhoury et al. [ 6 ] introduced TICODER , incorporating human feedback into the TDD-based code generation loop. Additionally, Piya et al. [ 12 , 11 ] explored how combining TDD practices with assistants like ChatGPT improves problem-solving on LeetCode -style tasks. In parallel, recent work has investigated the integration of LLMs within development environments and the dynamics of human–AI interaction. Amershi et al. [ 1 ] established fundamental principles for effective human–AI collaboration, emphasizing timely feedback, transparency, and user control—principles embodied in our structured Specification–Tests–Function workflow. Sergeyuk et al. [ 15 ] and Nghiem et al. [ 10 ] highlight the growing importance of IDE-embedded assistants and the need for transparent, user-centric design in coding tools. Complementarily, Crupi et al. [ 4 ] and Evtikhiev et al. [ 5 ] discuss the limitations of existing automatic evaluation metrics, such as BLEU or pass@k , arguing for interactive and context-aware evaluation frameworks. Finally, Gao et al. [ 7 ] point to broader challenges in the use of LLMs for software engineering, including trust, reproducibility, and transparency—issues that our work directly addresses through controlled, in-IDE experimentation. We build upon these prior works by exploring how a structured TDD workflow, starting from user-defined specifications, can be effectively leveraged for generating correct code with LLMs. Our study focuses on the interaction between the user and the TDD approach itself, an aspect that has been limitedly investigated, with the aim of examining the role of user involvement in test case definition and refinement within TDD. III The Currante plugin Figure 1 : Screenshot of the prototype of Currante plugin integrated into Visual Studio Code (VS Code). We designed and developed a Visual Studio Code (VS Code) plugin named Currante , which serves as the experimental platform for this study. Currante implements a TDD workflow that leverages LLMs to generate code based on user-provided specifications formally defined through test cases. The workflow is structured into three main phases, represented in the graphical

Read the complete text