SLO-Guard: Crash-Aware, Budget-Consistent Autotuning for SLO-Constrained LLM Serving

Christian Lysenstøen, github.com/Chrislysen · 2026

Read SLO-Guard: Crash-Aware, Budget-Consistent Autotuning for SLO-Constrained LLM Serving by Christian Lysenstøen, github.com/Chrislysen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-17627.txt.

Introduction

Lysenstøen presents SLO-Guard, a crash-aware autotuner for serving language models with vLLM under latency objectives. Many plausible configurations crash or miss latency targets, and standard optimizers waste those trials. SLO-Guard treats crashes as first-class observations, combining feasible-first Thermal Budget Annealing exploration with warm-started Tree-structured Parzen Estimator exploitation. Tested on Qwen2-1.5B with vLLM on an NVIDIA A100 40GB, it ties random search on best latency but spends a fixed budget more predictably, with tighter cross-seed consistency under concurrent load.

Catalog record

CREATORChristian Lysenstøen, github.com/Chrislysen
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE42030
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.17627

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SLO-Guard: Crash-Aware, Budget-Consistent Autotuning for SLO-Constrained LLM Serving Christian Lysenstøen, github.com/Chrislysen Abstract Serving large language models (LLMs) under latency service-level objectives (SLOs) is a configuration-heavy systems problem with an unusually failure-prone search space: many plausible configurations crash outright or miss user-visible latency targets, and standard black-box optimizers treat these failures as wasted trials. We present SLO-Guard, a crash-aware autotuner for vLLM serving that treats crashes as first-class observations. SLO-Guard combines a feasible-first Thermal Budget Annealing (TBA) exploration phase with a warm-started Tree-structured Parzen Estimator (TPE) exploitation phase; the handoff replays all exploration history, including crashes encoded as extreme constraint violations. We additionally contribute a configuration-repair pass, a GPU-aware KV-cache memory guard, and a four-category crash taxonomy. On Qwen2-1.5B served with vLLM 0.19 on an NVIDIA A100 40 GB, and across a pre-specified five-seed study, both SLO-Guard and uniform random search attain 75 / 75 75/75 feasibility with zero crashes, and are statistically indistinguishable on best-achieved latency (Mann–Whitney two-sided p = 0.84 p\!=\!0.84 ). SLO-Guard’s advantage lies in budget consistency : it allocates significantly more trials to the fast-serving regime ( 10.20 ± 1.10 10.20{\pm}1.10 vs. 7.40 ± 2.51 7.40{\pm}2.51 out of 15; one-sided p = 0.014 p\!=\!0.014 ) and maintains higher post-handoff consistency ( 0.876 ± 0.123 0.876{\pm}0.123 vs. 0.539 ± 0.224 0.539{\pm}0.224 ; p = 0.010 p\!=\!0.010 ). Under concurrent load, SLO-Guard’s cross-seed standard deviation on best latency is 4.4 × 4.4{\times} tighter than random search’s ( 2.26 2.26 ms vs. 10.00 10.00 ms). A harness-replication analysis shows that the consistency findings survive an independent sequential-dispatch measurement condition. We present SLO-Guard as a method paper with narrow but replicated empirical validation rather than a general solution to LLM serving autotuning: the central claim is not that SLO-Guard finds a better final configuration, but that it spends a fixed tuning budget more predictably once the fast regime has been found. Keywords: LLM serving, vLLM, autotuning, constrained Bayesian optimization, crash-aware search, service-level objectives, goodput. Introduction Serving LLMs under latency service-level objectives is now a systems problem as much as a modeling problem. Once a model is chosen, engineers must still select runtime parameters—batch size, token budget, memory utilization, execution mode, prefill strategy, and prefix caching—that interact in non-obvious ways with user-visible latency, memory pressure, and failure risk. A configuration that looks attractive on aggregate throughput may still be unusable if it violates time-to-first-token (TTFT) or inter-token latency (ITL) targets, or if it crashes the serving engine under realistic concurrent request mixes. Three properties of LLM-serving tuning differentiate it from classical hyperparameter optimization: • Failure-prone. Candidate configurations may crash at engine start, during single-request preflight, or only under realistic concurrent load. These are not observation noise; they are structural holes in the feasible set. • Conditional. Several knobs are meaningful only when others take specific values (e.g. enable_chunked_prefill is relevant only when enforce_eager = false in vLLM 0.19). A naive Cartesian search space misstates the problem. • Small-budget. Each trial starts a serving engine, warms it up, measures latency under load, and tears it down. In practice, tens of trials is a realistic budget; thousands is not. Research question. This paper asks a narrow but practically important question: when many vLLM configurations are crash-prone or unstable, can a crash-aware optimizer use a limited tuning budget more effectively than crash-agnostic baselines? We are not looking for asymptotic optimality, but for robust behavior under small budgets with mixed discrete/continuous, conditional search spaces and heavy-tailed failure events. Approach. We present SLO-Guard, a two-phase crash-aware autotuner. The method extends the author’s earlier Thermal Budget Annealing (TBA) work on constrained ML deployment [ 12 ] to the LLM inference setting: 1. TBA-explore : a feasible-first, crash-aware annealing phase that searches for viable regions and records unstable subspaces. 2. TPE-exploit : a warm-started Optuna TPE phase that exploits history gathered during exploration, including crashes represented as extreme constraint violations. A configuration-repair pass and a GPU-aware KV-cache guard bridge the gap between clean mathematical problem statements and the messy reality of a production-style serving API. Both pieces are deliberately boring engineering rather than learned components; their job is to remove implementation-level impossibilities so the optimizer can spend its limited budget on informative decisions. Key finding. Our primary empirical result is not a peak-latency win. Under the corrected concurrent harness, both SLO-Guard and random search are 75/75 feasible with zero crashes and are statistically tied on best-achieved latency. What differs is how the 15-trial budget is spent . SLO-Guard allocates more trials to the fast-serving regime, stays there more reliably after entering it, and yields substantially tighter cross-seed variance on the best-latency statistic. We therefore frame the contribution around budget consistency rather than peak-value superiority. Contributions. The paper makes four contributions: 1. A crash-aware formulation of SLO-constrained serving autotuning, including the repair map and GPU-aware feasibility guard that make the formulation implementable ( Section 3 ). 2. A simple two-phase optimizer (TBA-explore \to warm-started TPE-exploit) that encodes crashes as extreme constraint violations and replays them through the TPE bad-density model ( Section 3 ). 3. A pre-specified, multi-seed empirical study with non-parametric testing and explicit effect-size reporting, arguing that SLO-Guard’s value lies in consistent budget allocation rather than in final-latency gains ( Section 5 ). 4. A harness-replication analysis showing that the consistency result survives correction of the load-dispatch pipeline from sequential to truly concurrent request dispatch ( Section 5.4 ). We position the work as a case study in crash-aware serving autotuning. The method contribution is real, the implementation and data are public, the originally-flagged measurement defect is corrected and replicated, and the conclusions are deliberately kept at the level the evidence supports. Background and Related Work vLLM, serving knobs, and search-space structure vLLM is a widely used LLM serving engine built around PagedAttention-style KV-cache management [ 9 ] . The configuration space studied here contains eight knobs: quantization , max_num_seqs , max_num_batched_tokens , gpu_memory_utilization , max_model_len , enforce_eager , enable_chunked_prefill , and enable_prefix_caching . Together they span batching, memory pressure, context length, execution mode, and prompt reuse. Several of these knobs interact structurally in vLLM 0.19: enable_chunked_prefill is only meaningful when enforce_eager = false ; max_num_batched_tokens must be at least both max_num_seqs and max_model_len ; and the product max_num_seqs \times max_model_len must fit the available KV-cache budget on the specific GPU. This makes the tuning problem mixed-type, conditional, and hardware-aware. SLOs and goodput Raw throughput is insufficient as a standalone metric for user-facing LLM serving because users experience latency, not aggregate token counts [ 18 ] . Following this line of work we optimize goodput : throughput that remains within the latency envelope. Formalism follows in Section 3 . Our TTFT/ITL thresholds parallel the inference-benchmark conventions of MLPerf [ 14 ] and the metric set used by NVIDIA’s GenAI-Perf measurement tool [ 13 ] . Model-serving autotuning Auto-configuration for model serving predates the current LLM wave. Morphling [ 17 ] uses meta-learning to transfer cloud-native serving configurations across deployments. Managed tuning services such as AWS SageMaker Automatic Model Tuning [ 3 ] and Google Vizier [ 7 ] are mature examples of black-box optimization as a service. These systems are important context but do not specifically address the question of how to use crash signals when the serving

Read the complete text