When the Database Fails: Prompting LLM Dialogue Agents for Safe Recovery in Task-Oriented Dialogue

Mohammad Alijanpour Shalmani, Alale Rezvani Boroujeni, Jiann Shiun Yuan · 2026

Read When the Database Fails: Prompting LLM Dialogue Agents for Safe Recovery in Task-Oriented Dialogue by Mohammad Alijanpour Shalmani, Alale Rezvani Boroujeni, Jiann Shiun Yuan (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-31307.txt.

Introduction

Shalmani and two colleagues study how dialogue agents should react when their database fails. Task-oriented agents book hotels and tables, yet failed, empty, or wrong database results push language models to invent venues and confirmations. The team test a light prompting fix that needs no retraining and no extra model calls.

Across six open model families and four database conditions, naive agents hallucinate on about three in ten failure turns on MultiWOZ. Their Guided-Retry prompt, fed with structured database status, cuts that rate roughly in half on MultiWOZ and by 42 percent on SGD. Wrong-domain results remain the hardest case, and leftover hallucination still ranges widely by model, so prompting helps without solving the problem.

Catalog record

CREATORMohammad Alijanpour Shalmani, Alale Rezvani Boroujeni, Jiann Shiun Yuan
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE15 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.31307

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

When the Database Fails: Prompting LLM Dialogue Agents for Safe Recovery in Task-Oriented Dialogue Mohammad Alijanpour Shalmani, Alale Rezvani Boroujeni, Jiann Shiun Yuan Abstract Large language models used in task-oriented dialogue often produce fluent but unsafe responses when backend database calls fail, return empty results, or surface mismatched information, inventing venues, confirmations, or booking details not grounded in the database. We study a lightweight prompting-based recovery approach that improves robustness without retraining or additional model calls. We compare three response strategies, including a guided recovery prompt conditioned on structured database status, across six open-weight model families (DeepSeek-R1, Gemma-2, Llama-3, Mistral, Phi-3, and Qwen-2.5) and four database conditions: empty result, wrong-domain retrieval, API error, and clean retrieval. Using fault-injected benchmarks built on two structurally different datasets, MultiWOZ 2.2 (5 domains) and SGD (20 domains), we find that naive agents hallucinate on 30.5% of failure turns on MultiWOZ and 20.9% on SGD. Our Guided-Retry strategy reduces hallucination by 50% on MultiWOZ (30.5 → \to 15.3%) and by 42% on SGD (20.9 → \to 12.2%) without retraining. However, residual hallucination remains substantial (6–37% across models), with wrong-domain failures the hardest case. Results are consistent across both datasets and all six model families, and human annotation shows substantial agreement while supporting the validity of the automatic commitment-safety metric. 1 Introduction Task-oriented dialogue (TOD) systems help users book hotels, find restaurants, and arrange transport. Modern LLM-based agents ( 8 ) typically follow a standard pipeline: extract user intent, query a backend database, and generate a response grounded in the result. This pipeline is fragile at the database boundary. In practice, backend failures can still lead models to produce fluent but unsupported responses, including invented venues, confirmations, or booking details. Real deployments routinely encounter three failure modes: (1) empty result–no record matches the user’s constraints; (2) wrong-domain returns–the database routes to an incorrect domain ( 3 ) ; (3) API errors–timeouts and server failures. 2 Related Work Prior work has studied tool failure in LLM agents in broader settings. TRACE/SCOPE ( 7 ) identifies “Hallucination Fallback” when tools return empty results or time out. ReliabilityBench ( 6 ) injects faults into general agent tasks but not TOD. PALADIN ( 14 ) trains recovery policies over injected tool malfunctions. Non-collaborative simulators ( 13 ) stress-test agents in MultiWOZ under out-of-scope user requests , rather than backend execution faults. To our knowledge, prior work does not provide a MultiWOZ- and SGD-grounded evaluation that: (i) injects runtime DB execution faults rather than user-behavior stressors; (ii) compares prompting-only recovery without retraining; and (iii) quantifies residual hallucination under structured guidance. We make four contributions: (1) a controlled fault-injection framework over MultiWOZ 2.2 and SGD; (2) an evaluation of three prompting strategies across six models; (3) a set of automatic and human-validated metrics for failure recovery, including Commitment Safety Rate (CSR); and (4) the finding that structured prompting reduces, but does not eliminate, hallucination, leaving a residual rate of 6–37% across models. 3 Method 3.1 Datasets and Fault Injection We use the test splits of MultiWOZ 2.2 ( 15 ) (5 domains: hotel, restaurant, taxi, train, attraction) and SGD ( 12 ) (20 domains spanning restaurants, flights, hotels, events, media, and more). For each test dialogue, we extract the first user turn and synthetically inject one of four DB execution conditions , sampling 100 dialogues per condition (400 per dataset). We used the first user turn as a controlled diagnostic setting that isolates immediate recovery behavior under backend failure without additional confounds from longer dialogue context or multi-turn state tracking. • Clean : DB returns a valid result. Sanity baseline. • Empty Result : DB returns zero matches. Correct action: acknowledge failure and offer constraint relaxation. • Wrong Domain : DB returns results from an incorrect domain. Correct action: detect mismatch and confirm with user. • API Error : DB returns timeout or 503 error. Correct action: apologise and ask user to retry. 3.2 Recovery Strategies For each model, all strategies use the same decoding setup (temperature 0.2) and differ only in their system prompt. • Naive : raw DB response passed with no special instruction. • Inform : model explicitly instructed not to hallucinate and to acknowledge failures honestly. • Guided-Retry : model receives a structured decision procedure keyed on the DB status field, with explicit per-failure instructions and an explicit prohibition on fabricating venue names, booking references, or confirmation details. We evaluate six open-weight model families from six organizations: DeepSeek-R1 ( 5 ) , Gemma-2 ( 4 ) , Llama-3 ( 10 ) , Mistral ( 9 ) , Phi-3 ( 1 ) , and Qwen-2.5 ( 11 ) . 3.3 Metrics We evaluate recovery quality using four complementary metrics that capture hallucination, action appropriateness, commitment safety, and user friction. HR (Hallucination Rate): fraction of non-clean turns where the agent fabricates a result absent from the DB response, detected via regex patterns. AAR (Appropriate Action Rate): fraction matching ground-truth correct action per failure type, assessed via keyword heuristics. CSR (Commitment Safety Rate): fraction of non-clean turns free of explicit false commitments (e.g., invented booking refs, confirmations, or other booking-style commitment language). In the current automatic implementation, CSR is narrower than HR: it detects only explicit booking-style false commitments, so a response may increase HR without reducing CSR. CSR is validated by nine human annotators rating 60 sampled responses; inter-annotator agreement \kappa=0.7672 . UFS (User Friction Score): composite 0–3 (lower = better): +2 for hallucination, +1 for silent failure. Statistical significance is assessed using paired tests over matched fault-injected dialogue instances ( \alpha=0.05 ). Statistical testing. For each model independently, we compare Guided-Retry and Naive on matched fault-injected dialogue instances, where the same test turn under the same failure condition is evaluated under both strategies. For the binary metrics HR and AAR, we use McNemar’s exact test. For UFS, we use a two-tailed paired t -test. We observe the same significance pattern across all six models; all reported comparisons remain significant at p<0.001 . 4 Results 4.1 Overall Comparison Table 1 shows results on MultiWOZ. Guided-Retry consistently outperforms both baselines across all six models. Averaged across models, Naive achieves HR = 30.5% and AAR = 66.5%; Guided-Retry reduces HR by 50% to 15.3% and improves AAR to 83.0%. These improvements are statistically significant under our paired test setup ( p<0.001 ). CSR remains near ceiling because it is intentionally narrower than HR in the current automatic implementation. HR captures broad unsupported fabrication, while CSR captures only explicit booking-style false commitments; therefore, many hallucinations counted by HR do not affect CSR. DeepSeek-R1 achieves the lowest HR under Guided-Retry (5.8%). Phi-3 is the exception: Guided-Retry performs worse than Inform (HR 35.2 vs. 31.0), indicating that this model does not reliably follow structured recovery instructions. Table 2 shows a similar pattern to MultiWOZ: Guided-Retry achieves the best AAR on all six models, improving the 6-model average from 58.7% to 83.9%, and reduces average HR from 20.9% to 12.2%. Phi-3 remains the main exception, with higher HR under Guided-Retry than Inform. Table 1: MultiWOZ overall results. HR = Hallucination Rate (%), AAR = Appropriate Action Rate (%), CSR = automatic Commitment Safety Rate (%), UFS = User Friction Score. Best values per model and metric

Read the complete text