Prompt-Induced Over-Generation as Denial-of-Service: A Black-Box Attack-Side Benchmark

Manu, Yi Guo, Jo Plested, Tim Lynar, Kanchana Thilakarathna, Nirhoshan Sivaroopan, Jack Yang, Wangli Yang · 2025

Read Prompt-Induced Over-Generation as Denial-of-Service: A Black-Box Attack-Side Benchmark by Manu, Yi Guo, Jo Plested, Tim Lynar, Kanchana Thilakarathna, Nirhoshan Sivaroopan, Jack Yang, Wangli Yang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-23779.txt.

Introduction

Manu and seven co-authors benchmark a quiet way to attack language models: tricking them into never stopping. A model driven into over-generation emits thousands of tokens before its end-of-sequence signal, wrecking answer quality while inflating latency and cost; aimed deliberately, that is a denial-of-service attack. Earlier studies usually examined one attack method or assumed insider access to the model.

Their benchmark instead compares prompt-only attackers in a realistic black-box, query-only setting. One attacker evolves token prefixes that suppress the stop signal; the other trains a network to generate prefixes aimed at a target output length. A new metric, the over-generation factor, measures output against the context window. The learned attacker was far more severe, roughly doubling the evolutionary one's scores and driving nearly half its trials into budget-exhausting non-termination.

Catalog record

CREATORManu, Yi Guo, Jo Plested, Tim Lynar, Kanchana Thilakarathna, Nirhoshan Sivaroopan, Jack Yang, Wangli Yang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE44 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-23779

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Prompt-Induced Over-Generation as Denial-of-Service: A Black-Box Attack-Side Benchmark Manu, Yi Guo, Jo Plested, Tim Lynar, Kanchana Thilakarathna, Nirhoshan Sivaroopan, Jack Yang, Wangli Yang Abstract Large language models (LLMs) can be driven into over-generation, emitting thousands of tokens before producing an end-of-sequence (EOS) token. This degrades answer quality, inflates latency and cost, and can be weaponized as a denial-of-service (DoS) attack. Recent work has begun to study DoS-style prompt attacks, but typically focuses on a single attack algorithm or assumes white-box access, without an attack-side benchmark that compares prompt-based attackers in a black-box, query-only regime with a known tokenizer. We introduce such a benchmark and study two prompt-only attackers. The first is an Evolutionary Over-Generation Prompt Search (EOGen) that searches the token space for prefixes that suppress EOS and induce long continuations. The second is a goal-conditioned reinforcement learning attacker (RL-GOAL) that trains a network to generate prefixes conditioned on a target length. To characterize behavior, we introduce Over-Generation Factor (OGF): the ratio of produced tokens to a model’s context window, along with stall and latency summaries. Our evolutionary attacker achieves mean OGF = 1.38 ± 1.15 \mathrm{OGF}=1.38\pm 1.15 and Success@ OGF ≥ 2 \mathrm{OGF}\geq 2 of 24.5% on Phi-3. RL-GOAL is stronger: across victims it achieves higher mean OGF (up to 2.81 ± 1.38 2.81\pm 1.38 ). 1 Introduction Large language models (LLMs) can be forced into severe over-generation by delaying or preventing emission of the end-of-sequence (EOS) token ( Gao et al. 2024 ; Dong et al. 2024 ) . This is known as a denial of service (DoS) or “sponge” attack, where the aim is to overwhelm or exhaust LLM service resources ( Gao et al. 2024 ; Geiping et al. 2024 ; Zhang et al. 2025 ) . In DoS attacks, LLM output is degraded and computational costs increase ( Zhang et al. 2025 ) . This increases inference latency and can make LLMs inaccessible to other users or tasks, and in extreme cases can lead to service disruption ( Dong et al. 2024 ) . We focus on prompt-induced attacks in which a model generates excessive output in response to a crafted prefix that interferes with termination behavior. For example, Geiping et al. 2024 suppress an EOS token and drastically increase system response length via Greedy Coordinate Gradient (GCG) or string repetition. Further, Dong et al. 2024 demonstrate that adversarial Engorgio prompts can cause LLMs to generate abnormally long outputs and increase computation costs. Note also that DoS attacks can deplete system resources without affecting output accuracy ( Hasan et al. 2025 ) . As such, detection and prevention of such attacks are key. To address the lack of standardized attack-side evaluation for prompt-induced over-generation, we introduce a black-box, query-only (tokenizer-known) benchmark that compares prompt-based attackers under a consistent interface and decoding regime. While prior work has studied other forms of programmatic misuse that increase inference cost (e.g., over-reasoning slowdowns or interruption-style behaviors) ( Kumar et al. 2025 ; Cui and Zuo 2025 ) , our benchmark focuses specifically on stopping-time vulnerability: how easily an attacker can delay termination and inflate output length. We report consistent stopping-time metrics and attacker setups so results are reproducible and comparable across models. Within this benchmark, we study two automated, prompt-only attackers for short- to medium-length prefixes. EOGen performs gradient-free evolutionary search over a restricted prompt space, serving as a strong black-box, query-only baseline. RL-GOAL is a goal-conditioned reinforcement learning attacker that learns to generate prefixes targeting specific continuation lengths. We evaluate attacks using Over-Generation Factor (OGF) and complementary stopping-time summaries including stall rate and tail-persistence , enabling model-agnostic comparison across model families, sizes, and decoding settings while remaining strictly attack-side. 2 Related Work Adversarial prompting and limits of alignment. A large body of work shows that LLM behavior can be steered into undesirable regimes via adversarial prompts, including universal or transferable suffixes found by automated search ( Zou et al. 2023 ; Liao and Sun 2024 ) . From a theoretical perspective, Wolf et al. 2023 argue that for any behavior that occurs with non-zero probability, there exist prompts that can increase its likelihood, highlighting fundamental limitations of alignment under adversarial querying. Complementary empirical evidence suggests that safety alignment can be shallow and brittle, motivating evaluation settings where attackers can probe termination and stopping behavior rather than relying on single-case demonstrations ( Qi et al. 2024 ) . Automated prompt optimization for adversarial prompt construction. Our attackers are automated and prompt-only, which connects to prior work on discrete prompt optimization and trigger search. Early work on universal adversarial triggers and gradient-guided token search demonstrated that short token sequences can reliably elicit targeted behaviors and expose model vulnerabilities ( Wallace et al. 2019 ; Shin et al. 2020 ) . More recent work scales these ideas to aligned LLMs using automated suffix construction and search procedures that can transfer across prompts and even to black-box model interfaces ( Zou et al. 2023 ) . Closest in spirit to our EOGen baseline, Guo et al. 2024 introduce an evolutionary framework for discrete prompt optimization that is gradient-free and operates through iterative population-based exploration. Similarly, Deng et al. 2022 propose reinforcement learning for discrete prompt optimization, learning a policy that generates prompts by maximizing a task-defined reward, which conceptually motivates our RL-GOAL attacker as a goal-conditioned prefix generator. Over-generation and LLM denial-of-service. Recent work has begun to study DoS-style attacks that exhaust inference-time resources by inducing excessive generation or delaying termination. Token-level prompt optimization can suppress EOS and drive extreme length inflation (e.g., GCG-style methods) ( Tang et al. 2025 ; Liao and Sun 2024 ) , while crafted prompts such as Engorgio demonstrate prompt-only over-generation and increased compute cost ( Dong et al. 2024 ) . Beyond prompt-only settings, training-time manipulations such as P-DoS can also alter termination tendencies and prolong generation ( Gao et al. 2024 ) . In the broader security literature, “sponge” examples show that carefully constructed inputs can maximize latency/energy and induce worst-case inference behavior, including for language models ( Shumailov et al. 2021 ) . Most closely related to our setting, Zhang et al. 2025 introduce a black-box automated DoS prompt construction approach (AutoDoS) and report large latency amplification across models. Together, these findings motivate standardized, attack-side benchmarks that compare prompt-based attackers under consistent black-box (query-only) interfaces and stopping-time metrics. 3 Method: Evolutionary Over-Generation Prompt Search (EOGen) 3.1 Black-Box, Query-Only Over-Generation Setup We study standard next-token prediction with decoder-only LLMs in a text-completion setting. A prompt p is mapped to a token sequence \mathbf{x}=(x_{1},\dots,x_{m}) , and the model then autoregressively samples a continuation \mathbf{y}=(y_{1},\dots,y_{L}) conditioned on \mathbf{x} and the previously generated tokens. The decoding process is truncated either when a special end-of-sequence symbol \langle\mathrm{eos}\rangle appears or when a pre-defined upper bound on the number of generated tokens is reached. We refer to L=|\mathbf{y}| as the continuation length. In our experiments (Section A.1 ), we fix this upper bound and the decoding hyperparameters for each victim model so that differences in behavior arise only from the attacker’s choice of prompt, up to stochasticity introduced by sampling. Our threat model is a black-box, query-only, inference-time attacker. The adversary may (i) supply short textual prefixes p to the victim

Read the complete text