Citation Discipline in Spec-Driven Development: A Cross-Model Empirical Study of Output Determinism and Automated Hallucination Detection in LLM-Generated Code

Subham Panda · 2026

Read Citation Discipline in Spec-Driven Development: A Cross-Model Empirical Study of Output Determinism and Automated Hallucination Detection in LLM-Generated Code by Subham Panda (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-30689.txt.

Introduction

Subham Panda studies a trade-off in spec-driven development, where language models write code from formal specifications. Frameworks differ in how they tie code back to requirements. One, traceSDD, forces a citation to a numbered requirement on every line. Others trace at the level of whole artefacts, or build trace maps afterwards. Panda ran two controlled studies across two frontier models, with hundreds of implementations per condition.

The pre-registered analysis found the same pattern in both models. Uncited code was more deterministic, meaning independent sessions produced more similar text, than cited code. But only the cited condition allowed automated hallucination detection: software could flag code drifting from its specification. Detection rates reached about eighty-six to eighty-eight percent for cited code, against zero for every alternative, with no false positives. TraceSDD also beat one rival framework on determinism. Panda concludes that citations trade sameness for verifiability.

Catalog record

CREATORSubham Panda
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.30689

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Citation Discipline in Spec-Driven Development: A Cross-Model Empirical Study of Output Determinism and Automated Hallucination Detection in LLM-Generated Code Subham Panda Abstract Spec-Driven Development (SDD) frameworks guide Large Language Model (LLM)-powered code generation through formal specifications, yet they differ fundamentally in how they enforce traceability between requirements and generated code. This paper presents two controlled empirical studies comparing three SDD frameworks: traceSDD , which enforces mandatory per-line requirement citations using hierarchical REQ-XXX.Y.Z identifiers; Spec Kit , which uses artifact-level traceability through user stories and acceptance criteria; and OpenSpec , which relies on post-hoc external trace maps. We measure two primary outcomes across two frontier LLMs—Claude Sonnet 4.6 (N=20, 4 conditions, 240 implementations) and GLM-5-turbo (N=50, 4 conditions, 600 implementations): output determinism (lexical similarity across independent LLM sessions) and automated hallucination detection rate (TDR). Our pre-registered analysis reveals a consistent, cross-model replicated trade-off: the uncited condition produces significantly higher determinism than the cited condition (Claude: d = − 0.76 d=-0.76 , p = 0.003 p=0.003 ; GLM: d = − 0.72 d=-0.72 , p < 0.001 p<0.001 ), while only the cited condition enables automated hallucination detection (TDR: Claude 86.4%, GLM 88.0%, vs 0% for all alternatives, FPR=0% across both studies). traceSDD (cited) significantly outperforms Spec Kit on determinism (Claude: d = 0.47 d=0.47 , p = 0.049 p=0.049 ; GLM: d = 0.42 d=0.42 , p = 0.003 p=0.003 ) but not OpenSpec (Claude: d = 0.18 d=0.18 , p = 0.44 p=0.44 ; GLM: d = 0.14 d=0.14 , p = 0.32 p=0.32 ). These findings establish that citation annotations trade determinism for verifiability, and that this trade-off generalizes across model architectures. 1 Introduction The emergence of agentic coding systems—AI agents that autonomously write, test, and revise software—introduces a fundamental accountability problem: how do we verify that the code an agent writes actually implements what was specified? Unlike human developers who can explain their decisions, LLM-generated code appears plausible even when it introduces subtle deviations, including out-of-scope features, prior-art bleed-in from training data, and over-engineering beyond what was specified [ 14 , 11 ] . These hallucinations are particularly dangerous because they typically pass all automated tests while introducing unauthorized functionality with security, compliance, or maintenance implications [ 22 ] . The challenge of verifying LLM-generated code against specifications draws on a rich history of requirements traceability research in software engineering [ 8 , 5 , 13 ] . The IEEE 830 standard defines requirements traceability as the association between requirements and the work products that implement them [ 12 ] . Tools such as DOORS and Jama have provided enterprise-grade traceability matrices for decades. However, these traditional tools require substantial manual effort and are not designed for the rapid iteration cycles inherent to AI-assisted coding workflows [ 18 ] . Three specification-driven development (SDD) frameworks have emerged with distinct philosophies on traceability for LLM-generated code. traceSDD enforces mandatory inline citations on every line of code, binding each line to a specific Tier-3 requirement point (REQ-XXX.Y.Z). These citations are treated as verifiable claims: any REQ ID cited in code that does not appear in the specification is an automatically detectable orphan, signaling a hallucinated requirement. GitHub Spec Kit enforces artifact-level consistency through a spec-plan-tasks chain, but once implementation begins, the citation chain ends. OpenSpec uses post-hoc external trace maps (YAML sidecar files) that link line ranges to spec IDs after the fact, without annotating source code directly. The core question this work investigates is whether forcing an agentic coding system to cite a requirement for every line of code it writes materially improves implementation correctness, output determinism, and hallucination detectability—and whether any observed effects replicate across different LLM architectures. To address this question, we conduct two independent controlled studies: Study 1 uses Claude Sonnet 4.6 with 20 benchmark tasks, and Study 2 uses GLM-5-turbo with 50 benchmark tasks spanning 8 software engineering domains, 3 difficulty levels, and 2 size classes. This cross-model design provides the strongest possible evidence for generalizability of our findings. Our contributions are fourfold. First, we provide the first cross-model controlled comparison of SDD framework citation disciplines (70 total tasks, 4 conditions, 840 implementations). Second, we replicate the citation isolation finding across two fundamentally different LLM architectures, confirming that inline citations significantly reduce output determinism (Claude d=-0.76 , GLM d=-0.72 ). Third, we provide the first comprehensive subgroup analysis across domains, difficulty levels, and size classes, revealing that the determinism penalty of citations is most pronounced for easy and small tasks. Fourth, we demonstrate that hallucination detection via the orphan-REQ check achieves 86–88% TDR with 0% FPR across both models, a unique capability unmatched by any alternative framework. 2 Background and Related Work 2.1 Requirements Traceability in Software Engineering Requirements traceability—the ability to describe and follow the life of a requirement in both a forwards and backwards direction—has long been recognized as critical for safety-critical and regulated systems [ 8 ] . Cleland-Huang et al. [ 5 ] identified the gap between theoretical traceability frameworks and practical tooling, arguing that traceability must be embedded in the development workflow rather than treated as a post-hoc documentation exercise. Jalali and Wohlin [ 13 ] conducted a systematic literature review finding moderate evidence that traceability improves software quality, though the effect size depends heavily on tool support and process maturity. Pinto et al. [ 18 ] surveyed traceability tools and found that most operate at the artifact level (requirement-to-test, requirement-to-module) rather than at the line-of-code level that traceSDD targets. The work presented here extends this line of inquiry into the novel context of LLM-generated code, where the “developer” is an AI agent that cannot be asked to explain its decisions post-hoc. 2.2 LLM Code Generation and Hallucination Large language models have demonstrated remarkable code generation capabilities [ 4 , 17 , 3 ] . Codex [ 4 ] demonstrated that LLMs can solve competitive programming problems, while GPT-4 [ 17 ] showed further improvements in complex reasoning tasks. Recent models including StarCoder [ 16 ] and DeepSeek-Coder [ 9 ] have pushed code generation quality further. Claude [ 1 ] and GLM [ 20 ] represent two distinct architectures: Claude uses a transformer with constitutional AI training [ 2 ] , while GLM uses an autoregressive model with bidirectional attention for pre-training. However, LLMs are known to hallucinate: generating plausible-looking but factually incorrect or ungrounded content [ 14 , 11 ] . In the code generation domain, hallucinations manifest as out-of-scope functions, unauthorized imports from prior training data, and over-engineering through unnecessary decorators or optimizations. Ji et al. [ 14 ] provided a comprehensive survey of hallucination types in natural language generation, while Huang et al. [ 11 ] extended this taxonomy to cover multimodal and code generation contexts. Approaches to detecting code hallucinations include test-based validation [ 10 ] , static analysis heuristics, and specification comparison. The orphan-REQ check employed by traceSDD represents a novel approach: by requiring the code to cite specific requirement IDs, the system creates a set-difference operation that is both fully automated and language-agnostic. 2.3 Output Determinism and Consistency in LLMs Output determinism in LLMs—the degree to which identical prompts produce identical or highly similar outputs—has received increasing attention. Wang et al. [ 21

Read the complete text