Predict, Don’t React: Value-Based Safety Forecasting for LLM Streaming

Pride Kavumba, Koki Wataoka, Huy H. Nguyen, Jiaxuan Li, Masaya Ohagi · 2026

Read Predict, Don’t React: Value-Based Safety Forecasting for LLM Streaming by Pride Kavumba, Koki Wataoka, Huy H. Nguyen, Jiaxuan Li, Masaya Ohagi (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-03962.txt.

Introduction

Pride Kavumba and colleagues built StreamGuard, a streaming guardrail that forecasts harm instead of hunting for the first unsafe token. Prompt moderation sees full text, but streaming moderation must judge partial outputs, so StreamGuard predicts the expected harm of likely continuations from Monte Carlo rollouts. That allows early action without exact boundary labels. At the 8B scale it beat Qwen3Guard-Stream on input and streaming F1, reaching 97.5 F1 on a response-location benchmark.

Catalog record

CREATORPride Kavumba, Koki Wataoka, Huy H. Nguyen, Jiaxuan Li, Masaya Ohagi
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE67367
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.03962

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Predict, Don’t React: Value-Based Safety Forecasting for LLM Streaming Pride Kavumba, Koki Wataoka, Huy H. Nguyen, Jiaxuan Li, Masaya Ohagi Abstract In many practical LLM deployments, a single guardrail is used for both input and output moderation. Input moderation operates on fully observed text, whereas streaming output moderation requires safety decisions to be made over partial generations. Existing text-based streaming guardrails commonly frame this output-side problem as boundary detection, training models to identify the earliest prefix at which a response has already become unsafe. In this work, we introduce StreamGuard, a unified model-agnostic streaming guardrail that instead formulates moderation as a forecasting problem: given a partial prefix, the model predicts the expected harmfulness of likely future continuations. We supervise this prediction using Monte Carlo rollouts, which enables early intervention without requiring exact token-level boundary annotations. Across standard safety benchmarks, StreamGuard performs strongly both for input moderation and for streaming output moderation. At the 8B scale, StreamGuard improves aggregated input-moderation F1 from 86.7 to 88.2 and aggregated streaming output-moderation F1 from 80.4 to 81.9 relative to Qwen3Guard-Stream-8B-strict. On the QwenGuardTest response_loc streaming benchmark, StreamGuard reaches 97.5 F1, 95.1 recall, and 92.6% on-time intervention, compared to 95.9 F1, 92.1 recall, and 89.9% for Qwen3Guard-Stream-8B-strict, while reducing the miss rate from 7.9% to 4.9%. We further show that forecasting-based supervision transfers effectively across tokenizers and model families: with transferred targets, Gemma3-StreamGuard-1B reaches 81.3 output-moderation F1, 98.2 streaming F1, and a 3.5% miss rate. These results show that forecasting future risk enables effective low-latency streaming moderation without exact boundary labels. 1 1 1 Upon acceptance, we plan to release the research artifacts associated with this work. 1 Introduction Safety guardrails are now a standard component of Large Language Model (LLM) deployments, mitigating risks such as harmful instructions and toxic content ( 1 ; 21 ; 29 ) . In practice, a single guardrail is often deployed to handle both input and output moderation, but the two settings differ in observability. Input moderation operates on fully observed text, whereas output moderation in streaming systems requires decisions over partial prefixes before the final completion is available, creating a tension between safety and responsiveness. Text moderators such as LlamaGuard ( 8 ) are designed for full-sequence classification and are therefore most naturally applied after generation has completed. In a streaming product, that leaves an undesirable choice: either hold back the response until moderation finishes, increasing latency, or display tokens before the safety decision is finalized, creating an exposure window during which unsafe content may already have been shown. White-box defenses address this problem by attaching classifiers to the generator’s internal activations ( 11 ) , but these approaches require access to hidden states and are coupled to particular model architectures and checkpoints, which limits portability. Figure 1: Boundary detection versus forecasting in streaming moderation. Top: Boundary-detection methods evaluate whether the current prefix has already crossed an unsafe boundary, so intervention is tied to when unsafe content becomes explicit in the text. Bottom: StreamGuard instead scores each prefix by its expected future harmfulness , predicting whether likely continuations are likely to lead to an unsafe completion. This supports early intervention in streaming settings without requiring exact token-level boundary annotations. Recent text-based streaming guardrails address this gap by framing moderation as a boundary detection problem: the model is trained to identify the earliest prefix at which a response has already become unsafe. This is a natural formulation, and systems such as Qwen3Guard-Stream ( 27 ) provide a strong reference point for streaming moderation. At the same time, exact boundary supervision is expensive to construct, depends on the rollout-and-judge pipeline used to define the boundary, and is tied to tokenizer-specific coordinates. These practical costs motivate a different approach to streaming moderation. In this work, we introduce StreamGuard, a unified model-agnostic guardrail for streaming LLM systems that instead formulates moderation as a forecasting problem. As illustrated in Figure 1 , StreamGuard estimates the expected harmfulness of likely future continuations from the current prefix. We train this predictor using Monte Carlo rollouts: for each prefix, we sample possible futures, score them with a safety judge, and use the resulting future-risk estimate as supervision. A key advantage of this formulation is portability. Because supervision is defined over text prefixes with soft future-risk targets rather than tokenizer-specific boundary indices, the resulting training data transfers naturally across tokenizers and model families. Across our evaluations, StreamGuard delivers strong performance both as an input moderator and as a streaming response guardrail. At the 8B scale, StreamGuard improves aggregated input-moderation F1 from 86.7 to 88.2 and aggregated streaming output-moderation F1 from 80.4 to 81.9 relative to Qwen3Guard-Stream-8B-strict. On the QwenGuardTest response_loc streaming benchmark, the 8B StreamGuard model reaches 97.5 F1, 95.1 recall, and 92.6% on-time intervention, compared to 95.9 F1, 92.1 recall, and 89.9% for Qwen3Guard-Stream-8B-strict, while reducing the miss rate from 7.9% to 4.9%. These results show that strong end-to-end streaming moderation can be obtained without exact boundary labels, while retaining portability across model families. Our main contributions are as follows: • We introduce StreamGuard, a unified guardrail for prompt and streaming output moderation, with a forecasting-based formulation for streaming output moderation that provides a model-agnostic alternative to boundary supervision. • We show that forecasting yields strong end-to-end moderation performance, improving both standard input/output moderation scores and streaming unsafe-event detection against a strong published boundary-based reference. • We demonstrate that forecasting supervision transfers across model families and tokenizers, including to Gemma ( 19 ) and Qwen ( 24 ) backbones, and we provide ablations characterizing how rollout construction, aggregation, filtering, and supervision density affect performance and annotation cost. 2 Methodology StreamGuard’s core methodological idea is to cast streaming output moderation as a forecasting problem. Given a partially generated response, the model estimates a scalar future-risk score: the expected harmfulness of plausible continuations conditioned on the prompt and the observed prefix. 2.1 Prefix-Conditioned Safety Value Let x denote a user prompt and let y=\{y_{1},y_{2},\dots,y_{T}\} denote a model response. For each observed prefix y_{\leq t} , we define the target value as the expected harmfulness of plausible future completions: V^{*}(x,y_{\leq t})=\mathbb{E}_{\tilde{y}_{t+1:T}\sim q(\cdot\mid x,y_{\leq t})}\left[\mathcal{O}\!\left(x,y_{\leq t}\mathbin{\|}\tilde{y}_{t+1:T}\right)\right], (1) where q(\cdot\mid x,y_{\leq t}) is a continuation distribution over possible futures and \mathcal{O}(\cdot) is a safety judge that maps a completed response to a score in [0,1] . Unlike boundary-detection approaches such as Qwen3Guard-Stream ( 27 ) , which predict the first prefix that is already explicitly unsafe, we estimate the expected risk of continuing from the current prefix. For example, for “To build a bomb, first you must” , a boundary detector may still predict safe because no explicit violation has yet appeared, whereas our objective assigns a high score if likely continuations are harmful. This framing also avoids propagating the final response label to all earlier prefixes, which would incorrectly mark generic scaffolds such as “I” , “The” , or “This” as unsafe whenever they occur in an unsafe completion. 2.2 Monte Carlo Future-Risk Targets The expectation in Equation ( 1 ) is intractable to compute exactly, so we approximate it with Monte Carlo rollouts. For a supervised prefix y_{\leq t} , we sample M continuations \mathcal{R}_{t}=\left\{\tilde{y}^{(m)}_{t+1:T}\right\}_{m=1}^{M},\qquad\tilde{y}^{(m)}_{t+1:T}\sim q(\cdot\mid x,y_{\leq t}). (2) We then score each completed continuation with the offline guardrail

Read the complete text