PRISM: Prompt Reliability via Iterative Simulation and Monitoring for Enterprise Conversational AI

Keshava Chaitanya, Jahnavi Gundakaram · 2026

Read PRISM: Prompt Reliability via Iterative Simulation and Monitoring for Enterprise Conversational AI by Keshava Chaitanya, Jahnavi Gundakaram (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-15665.txt.

Introduction

Keshava Chaitanya and Jahnavi Gundakaram treat enterprise prompt work as a reliability job, not a one-off writing task. Deployed models drift in silence over time. Their PRISM loop writes test cases from plain-language needs, plays full multi-turn chats, grades pass or fail, finds fault roots, and mends prompts by surgery. It repeats till all tests pass and reruns each day. Across 35 enterprise agents over three weeks it cut median writing time from two days to under thirty minutes. It hit 99 percent production reliability.

Catalog record

CREATORKeshava Chaitanya, Jahnavi Gundakaram
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE30524
LICENSEhttp://creativecommons.org/publicdomain/zero/1.0/
RIGHTSCC0
SOURCEarXiv

Source: https://arxiv.org/abs/2605.15665

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

PRISM: Prompt Reliability via Iterative Simulation and Monitoring for Enterprise Conversational AI Keshava Chaitanya, Jahnavi Gundakaram Abstract Deploying large language model (LLM)-driven conversational agents in enterprise settings requires prompts that are simultaneously correct at launch and resilient to the non-deterministic behavioral drift that characterizes production LLM deployments. Existing prompt optimization frameworks address prompt quality as a one-time compile-time problem, leaving open the equally critical question of how to detect and repair prompt regressions caused by silent LLM behavior changes over time. We present PRISM (Prompt Reliability via Iterative Simulation and Monitoring), a closed-loop framework that treats prompt engineering as a continuous reliability engineering problem rather than a one-time authorship task. PRISM takes as input plain-language agent requirements, a set of configured tools and memory variables, and an initial draft prompt. It automatically generates test cases from requirements, simulates full multi-turn conversations against a platform-faithful LLM environment, evaluates pass/fail using an LLM-as-judge, diagnoses root causes of failures, and surgically repairs the prompt—iterating until all tests pass. Critically, PRISM is designed to run on a scheduled basis (daily), treating LLM behavioral drift as a first-class reliability concern. We evaluate PRISM across 35 enterprise conversational agents over a three-week deployment period on the Yellow.ai V3 platform. PRISM reduces median prompt authoring time from 2 days to under 30 minutes, achieves 99% production reliability across all evaluated agents, and successfully identifies and repairs production regressions caused by LLM behavioral drift within a 24-hour detection window. Our results suggest that continuous, simulation-driven prompt optimization is both tractable and necessary for reliable enterprise conversational AI at scale. 1 Introduction Enterprise conversational AI systems—customer support bots, cancellation agents, onboarding assistants—are increasingly powered by large language model (LLM) backends executing complex, multi-step business logic. Unlike general-purpose chat interfaces, these agents must follow precise operational sequences: authenticate the user before proceeding, call a specific workflow with specific parameters, route to a human agent under specific conditions, never reveal internal state, and use exact prescribed phrases in regulated contexts. The instructions encoding this logic, known as frontend agent prompts , are the principal artifact of enterprise AI deployment—they are the source of truth for what the agent is supposed to do. Writing a correct frontend prompt is a fundamentally different challenge from general prompt engineering. The failure modes are subtle and operationally consequential: a missing instruction causes a mandatory step to be silently skipped; a wrong tool name causes a silent call failure with no error visible to the user; an ambiguous conditional causes wrong-branch routing; an overly compressed instruction set causes the model to collapse multi-step procedures into a single response, skipping intermediate logic. In our study of 35 enterprise agents, we identify four dominant failure classes: tool call skipping (the agent omits a required workflow call), rule violation (the agent ignores an explicit constraint in the prompt), step reordering (the agent performs steps out of sequence), and step collapsing (the agent skips to a terminal step without completing intermediate steps). Each of these failures produces a qualitatively poor user experience and carries direct brand risk for enterprise operators. The problem is compounded by a second challenge that the literature has largely ignored: LLM behavioral drift . Large language models—even pinned versions accessed via API—exhibit non-deterministic behavior that changes subtly over time. A prompt that produces 100% correct behavior today may silently degrade over days or weeks, producing the same failure modes described above but now in a production system serving real users. This drift is not caused by explicit model updates; it arises from temperature, sampling variation, and undisclosed inference-layer changes at the provider level. Enterprise operators currently have no systematic way to detect, localize, or repair this drift before it affects users. Existing prompt optimization frameworks do not address these challenges adequately. DSPy ( 3 ) frames prompt optimization as program compilation, optimizing few-shot demonstrations and instructions for pipeline-level metrics. OPRO ( 6 ) uses LLMs to propose and score candidate prompts against a fixed dataset. APE ( 9 ) generates candidate prompts and ranks them by execution accuracy. These are valuable contributions, but all share a critical assumption: the optimization target is fixed at compile time, and the resulting prompt is treated as a static artifact. None of these frameworks model the multi-turn, tool-integrated conversational environment that enterprise agents operate in, and none address the ongoing maintenance problem of production drift. We present PRISM , a framework designed specifically for the enterprise conversational AI context. PRISM makes the following contributions: 1. A two-phase formulation of the prompt reliability problem , distinguishing creation-time correctness (can we produce a prompt that passes all tests?) from runtime reliability (does that prompt continue to pass tests in production over time?). 2. A requirement-driven test generation methodology that derives structured, executable test cases from plain-language business requirements, producing realistic multi-turn conversation scripts, precise pass criteria, and mock tool responses calibrated to the tool’s declared return schema. 3. A platform-faithful simulation environment that replicates the two-layer prompt architecture of production Yellow.ai V3 deployments, intercepting LLM tool calls and substituting per-test mock responses—enabling deterministic, repeatable test execution without touching the live production bot. 4. A surgical repair mechanism informed by Yellow.ai platform guides and grounded in the actual failure transcript, which modifies only the sections of the prompt responsible for observed failures—preserving passing test behavior while fixing failing cases. 5. A continuous monitoring protocol in which PRISM runs daily against the production prompt, detecting and repairing any behavioral regressions within a 24-hour window. We evaluate PRISM on 35 enterprise agents over three weeks and demonstrate that it achieves 99% production reliability, reduces authoring time from a median of 2 days to under 30 minutes, and successfully detects and repairs production drift events that would otherwise have been discovered only through user complaints. 2 Background and Related Work 2.1 LLM-Based Conversational Agents The use of large language models as the reasoning core of goal-directed conversational agents has grown rapidly since the introduction of GPT-3 ( 1 ) . ReAct ( 7 ) demonstrated that interleaving reasoning traces with action invocations enables LLMs to complete multi-step tasks in interactive environments. Toolformer ( 5 ) showed that models can learn when and how to call external APIs. More recent work on function-calling models ( 4 ) has formalized this interaction pattern, making LLM-driven tool use a first-class feature of production APIs. Enterprise conversational agents extend this paradigm with additional constraints: strict step ordering, explicit routing logic, compliance-driven language requirements, and integration with business-specific workflows. These constraints are encoded in natural-language system prompts, making prompt correctness the primary reliability lever available to operators. 2.2 Prompt Optimization Automatic prompt optimization has emerged as an active research area. APE ( 9 ) frames prompt generation as a black-box optimization problem, using an LLM to generate candidate instructions and selecting the best-performing candidate based on held-out task accuracy. OPRO ( 6 ) extends this idea by including previous candidates and their scores in context, allowing the optimizer to refine proposals iteratively. PromptBreeder ( 2 ) evolves populations of prompts using mutation and selection operators. DSPy ( 3 ) represents the most architecturally sophisticated approach, compiling declarative pipeline specifications into optimized prompts by jointly optimizing instructions and few-shot demonstrations. DSPy’s key

Read the complete text