When “Better” Prompts Hurt: Evaluation-Driven Iteration for LLM Applications A Framework with Reproducible Local Experiments
Daniel Commey · 2026
Read When “Better” Prompts Hurt: Evaluation-Driven Iteration for LLM Applications A Framework with Reproducible Local Experiments by Daniel Commey (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-22025.txt.
Introduction
Daniel Commey warns that improving a prompt can quietly make a language model application worse. Testing such applications differs from traditional software testing. Outputs are stochastic, high-dimensional, and sensitive to small changes in the prompt or the model behind it.
Commey presents an evaluation-driven workflow with four stages: Define, Test, Diagnose, Fix. Each round of prompt edits is checked against defined tests before it ships, turning prompt tweaking into a repeatable engineering loop. His 2026 preprint stresses reproducibility, since a fix that cannot be retested is just a hope. Teams shipping chatbots and assistants need exactly this discipline as prompts become production code.
Catalog record
| CREATOR | Daniel Commey |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 90 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-22025
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
When “Better” Prompts Hurt: Evaluation-Driven Iteration for LLM Applications A Framework with Reproducible Local Experiments Daniel Commey Abstract Evaluating Large Language Model (LLM) applications differs from traditional software testing because outputs are stochastic, high-dimensional, and sensitive to prompt and model changes. We present an evaluation-driven workflow—Define, Test, Diagnose, Fix—that turns these challenges into a repeatable engineering loop. We introduce the Minimum Viable Evaluation Suite (MVES), a tiered set of recommended evaluation components for (i) general LLM applications, (ii) retrieval-augmented generation (RAG), and (iii) agentic tool-use workflows. We also synthesize common evaluation methods (automated checks, human rubrics, and LLM-as-judge) and discuss known judge failure modes. In reproducible local experiments (Ollama; Llama 3 8B Instruct and Qwen 2.5 7B Instruct), we observe that a generic “improved” prompt template can trade off behaviors: on our small structured suites, extraction pass rate decreased from 100% to 90% and RAG compliance from 93.3% to 80% for Llama 3 when replacing task-specific prompts with generic rules, while instruction-following improved. These findings motivate evaluation-driven prompt iteration and careful claim calibration rather than universal prompt recipes. All test suites, harnesses, and results are included for reproducibility. Keywords. Large language models, evaluation, benchmarks, metrics, RAG, retrieval-augmented generation, LLM-as-judge, prompt engineering, regression testing, MVES. Contributions. This paper makes six contributions. First, we introduce the MVES framework : a tiered standard defining minimum evaluation requirements for general LLM applications (MVES-Core), retrieval-augmented systems (MVES-RAG), and agentic workflows (MVES-Agentic). Second, we provide a synthesis of evaluation methods from literature, discussing correlation with human judgment, cost per 1,000 examples, and execution time. Third, we present a taxonomy of quality dimensions distinguishing correctness, helpfulness, harmlessness, groundedness, and format adherence. Fourth, we give a detailed analysis of LLM-as-judge failure modes , including position bias, verbosity bias, self-preference, style bias, and instruction leakage. Fifth, we provide actionable checklists for test set design, metric selection, human evaluation rubrics, and production monitoring. Sixth, we present original experiments demonstrating that task-specific prompts can outperform generic improvements on structured tasks, using local inference for full reproducibility. 1 Introduction The deployment of Large Language Models (LLMs) in production applications has accelerated dramatically. Organizations now use LLMs for customer support, document summarization, code generation, knowledge retrieval, and countless other tasks. Yet evaluating these systems remains surprisingly difficult. Traditional software testing, which assumes deterministic outputs for given inputs, does not translate directly to LLM-powered applications. 1.1 Why LLM Evaluation Differs from Traditional Testing Consider a conventional API: given a well-formed request, the response is deterministic and can be validated against an expected output. LLM applications violate nearly every assumption underlying this paradigm. The first challenge is non-determinism . In many deployments, even with identical prompts and temperature set to zero, LLMs can produce different outputs across inference calls due to hardware numerics, decoding implementations, concurrency, or upstream model changes [ 18 ] . This variability means that exact-match testing, the foundation of traditional software verification, can be unreliable unless the inference stack is fully controlled. The second challenge is output space complexity . Natural language responses can be semantically equivalent while being lexically distinct. The statements “The capital of France is Paris” and “Paris serves as France’s capital” convey identical information but differ textually. Evaluating semantic equivalence requires more sophisticated methods than string comparison. The third challenge involves implicit specifications . User expectations for “good” responses are often context-dependent and difficult to formalize. A concise answer may be preferred in one context and insufficient in another. Unlike APIs with explicit schemas, LLM quality is often in the eye of the beholder. Finally, model churn complicates evaluation over time. LLM providers frequently update their models, sometimes without explicit versioning. A system that worked yesterday may behave differently today [ 22 ] . Continuous evaluation becomes necessary to detect regressions introduced by upstream model changes. 1.2 Key Risks in LLM Applications Insufficient evaluation exposes applications to several categories of risk that must be addressed before deployment. Hallucination refers to the generation of plausible-sounding but factually incorrect statements. This phenomenon has been extensively documented in the literature [ 8 , 14 ] . In high-stakes domains such as healthcare or legal advice, hallucinations can cause material harm. Evaluation must specifically test for factual accuracy against verified sources. Safety violations occur when LLMs produce harmful, biased, or inappropriate content. Without appropriate guardrails, models may respond to adversarial prompts in dangerous ways. Red-teaming and adversarial testing are essential to surface these failure modes before deployment [ 20 , 3 ] . Safety evaluation should cover toxic content, privacy violations, and refusal of genuinely harmful requests. Prompt drift emerges as prompts are iteratively refined during development. Subtle changes to system prompts or few-shot examples may have unintended effects on unrelated behaviors. Without regression testing, these regressions go undetected until reported by users. Comprehensive test suites help ensure that improvements in one area do not cause degradation in others. Distribution shift occurs when production inputs differ from development test cases. Users may phrase requests in unexpected ways, submit adversarial inputs, or use the system for unanticipated purposes. Evaluation should include realistic samples from production, not just curated examples that developers find convenient. 1.3 What This Paper Provides This paper provides a complete evaluation harness with 50 curated test cases spanning extraction (20 cases), RAG question-answering (15 cases), and instruction-following (15 cases). We demonstrate evaluation-driven iteration using local inference via Ollama, enabling full reproducibility without API costs. Our experiments reveal a counterintuitive finding: generic prompt improvements are not monotonic. Adding a “helpful assistant” system wrapper with explicit rules degraded extraction accuracy by 10% and RAG compliance by 13% on Llama 3 8B, while improving instruction-following by 13%. A four-condition ablation isolates the mechanism: the system wrapper itself has no effect; the degradation comes from generic rules conflicting with task-specific constraints. The practical implication is that prompt changes should be validated against task-specific test suites rather than assumed beneficial based on conventional wisdom. The evaluation loop described in Section 2 operationalizes this approach. Artifacts. All code, datasets, and experiment logs are available at github.com/dcommey/llm-eval-benchmarking . 2 The Evaluation Loop Effective LLM evaluation follows a structured iteration cycle. This section introduces a four-phase workflow that serves as the organizing principle for the remainder of this paper. 2.1 The Core Workflow Traditional software testing verifies outputs against known correct answers. LLM applications complicate this model because outputs are often unstructured, subjective, or context-dependent. Nevertheless, a disciplined evaluation process remains essential. The evaluation loop consists of four phases applied repeatedly throughout development. In the Define phase, teams articulate quality requirements in testable terms. What constitutes acceptable output for this application? What failures are most costly? In the Test phase, the system is evaluated against a curated suite of inputs with known properties. In the Diagnose phase, failures are categorized to identify systematic patterns. In the Fix phase, prompts, retrieval logic, or model selection are adjusted based on the diagnosis. The cycle then repeats. This workflow differs from one-time benchmarking in two important ways. First, it treats evaluation as continuous rather than gated. Each prompt change or model update triggers re-evaluation. Second, it emphasizes failure analysis over aggregate metrics. Understanding why a case failed matters more than computing a single accuracy number. 2.2 Translating Requirements into Tests