LLM-Based Robustness Testing of Microservice Applications: An Empirical Study Practical Experience Report

Hrushitha Goud Tigulla, Marco Vieira · 2026

Read LLM-Based Robustness Testing of Microservice Applications: An Empirical Study Practical Experience Report by Hrushitha Goud Tigulla, Marco Vieira (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-14202.txt.

Introduction

Tigulla and Vieira run a controlled experiment on language-model robustness testing for microservice APIs. Malformed, missing or boundary-value inputs to microservice APIs can cascade across dependent services and threaten uptime. Robustness testing probes such inputs to reveal server-side failures, but crafting diverse, effective tests is hard. The experiment applies seven prompt strategies to three open-source language models, targeting two architecturally distinct systems. One is a Java monolingual system with six services, the other a polyglot system with twenty-seven services, yielding 663 generated tests.

Catalog record

CREATORHrushitha Goud Tigulla, Marco Vieira
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE55663
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.14202

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

LLM-Based Robustness Testing of Microservice Applications: An Empirical Study Practical Experience Report Hrushitha Goud Tigulla, Marco Vieira Abstract Malformed, missing, or boundary-value inputs in microservice APIs can cascade across dependent services, threatening reliability. Robustness testing systematically exercises such inputs to expose server-side failures, but generating diverse, effective tests remains challenging. Large Language Models can generate such tests from API specifications; however, it is unknown whether different models and prompt strategies produce diverse failure sets or converge on the same failures. We report a controlled experiment applying 7 prompt strategies to 3 open-source LLMs (14B–70B parameters) targeting 2 architecturally distinct microservice systems: one Java monolingual (6 services, 9 failure modes) and one polyglot (27 services, 14 failure modes), yielding 38 valid runs and 663 generated tests. We find that prompt strategy explains more variation in diversity than model size: a Structured prompt collapses diversity entirely, while a single model varied across three prompt strategies achieves complete failure-mode coverage on one system, outperforming any multi-model ensemble under a fixed prompt. We introduce two strategies, Guided and GuidedFewShot, that embed a mutation taxonomy from prior robustness testing research as domain context. GuidedFewShot achieves the highest single-run coverage on both systems (5 of 9 and 8 of 14 failure modes) while maintaining low cross-model similarity. A key lesson is that taxonomy rules alone are insufficient: LLMs cannot distinguish key-absent from value-empty mutations without concrete examples. Findings replicate across both systems. This work has been submitted to the IEEE for possible publication. Copyright may be transferred without notice, after which this version may no longer be accessible. I Introduction Microservice architectures decompose applications into independently deployable services that communicate over HTTP and messaging protocols, each exposing endpoints that must handle not only well-formed requests but also malformed, missing, and boundary-value inputs without producing server-side failures. Such failures compromise service reliability: a single unhandled input in one microservice can cascade through dependent services, degrading or disrupting the entire application. Robustness testing, the systematic exercise of such abnormal inputs, is well established for distributed systems. Mutation-based approaches provide structured taxonomies of input faults [ 14 ] , and tools like Ballista [ 6 ] demonstrated that automated fault injection can expose failures that conventional testing misses. More recently, Large Language Models (LLMs) have been applied to generate test cases from API specifications [ 5 , 13 ] , raising a practical question: when different models and prompt strategies generate robustness tests for the same system, do they expose different failure modes, or do they all find the same bugs? The answer matters for testing practice. If every model-prompt configuration converges on the same failures, running multiple configurations wastes effort. If they diverge, practitioners can combine them to cover more failure modes at modest cost. Recent work has begun to systematically evaluate LLM-generated tests. For example, TestForge [ 13 ] introduced a benchmarking framework that measures the success of compilation, functional correctness, and code coverage of LLM-generated JUnit tests across prompting strategies and models. That work confirms LLMs can produce functional tests for well-defined problems but finds they struggle with behavioral diversity. Existing studies of LLM-based test generation focus on code correctness, compilation rate, and line coverage [ 13 , 4 ] . None measure the diversity of failure modes discovered across models and prompt strategies in robustness testing, and the relationship between prompt design and failure-mode diversity has not been studied. Our study addresses a complementary question: rather than measuring whether generated tests are correct, we measure whether different configurations discover different failures in running microservice systems. We report a controlled empirical study based on practical experience applying LLM-based test generation to two open-source microservice systems. We apply 7 prompt strategies to 3 open-source LLMs (14B, 32B, and 70B parameters, including one code-specialized model) targeting 2 architecturally distinct microservice systems: a Java monolingual application with 6 services and a polyglot application with 27 services written in Go, C++, Rust, JavaScript, .NET, and other languages. The experiment yields 38 valid runs and 663 generated tests. We define failure modes (FMs) as unique (endpoint, parameter, mutation type) triples verified against the application source code, identifying 9 FMs in the first system and 14 in the second. Coverage is measured as the fraction of FMs triggered per run. Diversity is measured as pairwise Jaccard similarity of FM sets across models using the same prompt strategy. We also investigate whether injecting domain knowledge from an established mutation taxonomy [ 14 ] into LLM prompts changes coverage and diversity. The study addresses three research questions: RQ1 (Coverage): How does failure mode coverage vary across models and prompt strategies? RQ2 (Diversity): How diverse are the failure mode sets produced by different models under the same prompt strategy? RQ3 (Domain Knowledge): Does incorporating mutation taxonomy rules from robustness testing research into LLM prompts improve coverage without collapsing diversity? The main contributions are: • Empirical evidence that the prompt strategy explains more diversity variation than model size. Changing the prompt for a fixed model produces greater FM set variation than changing the model under a fixed prompt. • Identification and cross-system replication of Structured collapse , where a prescriptive prompt causes all three models to produce identical FM sets (Jaccard =1.00 on both systems). • Design of two new prompt strategies, Guided and GuidedFewShot, that embed the mutation taxonomy from Vieira et al. [ 14 ] directly into the LLM prompt as domain context, providing mutation rules without prescribing specific test inputs. • Evidence that domain knowledge transforms a zero-coverage model-strategy pair into the highest-performing configuration. A code-specialized model that triggered zero robustness failures under Self-Refine achieved 5 of 9 FMs (56%) and 8 of 14 FMs (57%) under GuidedFewShot, the highest single-run coverage on both systems. • Evidence that prompt variation for a single model outperforms model variation under a fixed prompt: one model varied across three strategies achieves complete FM coverage on one system, while the best single-strategy multi-model ensemble reaches only 64%. • Cross-system replication of all findings across two architecturally different microservice applications, covering monolingual versus polyglot implementations and form-encoded versus JSON communication protocols. This paper is organized as follows. Section II reviews related work. Section III describes the study design. Section IV defines failure modes. Section V presents results by research question. Section VI discusses implications. Section VII addresses threats to validity, and Section VIII concludes. II Related Work Automated robustness testing originated with Ballista [ 6 ] , which tested POSIX interfaces by substituting invalid values drawn from type-specific fault models. The same work introduced the CRASH failure classification (Catastrophic, Restart, Abort, Silent, Hindering) as a severity scale for observed failures. Vieira et al. [ 14 ] extended this approach to web services, organizing mutations by data type: string (null, empty, nonprintable, overflow), numeric (null, zero, negative, boundary), list (null, empty, element removal, duplication), and structural (field-absent, field-null, empty body). These ideas were later operationalized for SOAP services by wsrbench [ 8 ] and for REST services by black-box testing tools [ 7 ] . Our Guided prompt strategies embed this mutation taxonomy directly as domain context for LLM-based test generation, extending its applicability to a new test generation paradigm. Stateful REST API testing has been addressed by RESTler

Read the complete text