From Judgment to Interference : Early Stopping LLM Harmful Outputs via Streaming Content Monitoring
Yang Li, Qiang Sheng, Yehan Yang, Xueyao Zhang, Juan Cao · 2025
Read From Judgment to Interference : Early Stopping LLM Harmful Outputs via Streaming Content Monitoring by Yang Li, Qiang Sheng, Yehan Yang, Xueyao Zhang, Juan Cao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-09996.txt.
Introduction
Li and co-authors build a monitor that stops harmful language model output mid-stream in this 2025 paper. Safety filters usually judge a finished reply, which adds delay. Judging partial text is faster but models trained on full text perform poorly on fragments.
The team built FineHarm, a dataset of 29,000 prompt-response pairs labeled at fine grain. Their streaming content monitor trains on both whole-response and token-level labels. It follows the output as it forms and cuts it off once harm is clear. On average it decided after seeing only 18 percent of the tokens, yet scored above 0.95 in macro F1, matching full-text judges. It can also label data to improve other safety training.
Catalog record
| CREATOR | Yang Li, Qiang Sheng, Yehan Yang, Xueyao Zhang, Juan Cao |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 56 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-09996
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
From Judgment to Interference : Early Stopping LLM Harmful Outputs via Streaming Content Monitoring Yang Li, Qiang Sheng, Yehan Yang, Xueyao Zhang, Juan Cao Abstract Warning: this paper may contain potentially generated harmful content. Though safety alignment has been applied to most large language models (LLMs), LLM service providers generally deploy a subsequent moderation as the external safety guardrail in real-world products. Existing moderators mainly practice a conventional full detection, which determines the harmfulness based on the complete LLM output, causing high service latency. Recent works pay more attention to partial detection where moderators oversee the generation midway and early stop the output if harmfulness is detected, but they directly apply moderators trained with the full detection paradigm to incomplete outputs, introducing a training-inference gap that lowers the performance. In this paper, we explore how to form a data-and-model solution that natively supports partial detection. For the data, we construct FineHarm , a dataset consisting of 29K prompt-response pairs with fine-grained annotations to provide reasonable supervision for token-level training. Then, we propose the streaming content monitor , which is trained with dual supervision of response- and token-level labels and can follow the output stream of LLM to make a timely judgment of harmfulness. Experiments show that SCM gains 0.95+ in macro F 1 F_{1} score that is comparable to full detection, by only seeing the first 18% of tokens in responses on average. Moreover, the SCM can serve as a pseudo-harmfulness annotator for improving safety alignment and lead to a higher harmlessness score than DPO. = Date : June 11, 2025 \emails { liyang23s , shengqiang18z , caojuan }@ict.ac.cn 1 Introduction Figure 1: Paradigm differences between the existing harmful content detectors and our proposed Streaming Content Monitor (SCM). (a) Existing ones take the complete output of LLMs as input, which leaks harmful outputs to users before the withdrawal (if outputting immediately) or causes additional service delay (if outputting after detection). (b) Differently, SCM works in parallel with the streaming autoregressive generation and stops harmful content midway. Output tokens are in blue shaded boxes . Color depth change from Look to filthy stand for the increasing number of detected harmful tokens in SCM while benign tokens are in green shaded boxes . Large language models (LLMs) that utilize the autoregressive paradigm have demonstrated remarkable capabilities across diverse tasks, playing increasingly important roles in multiple applications ( Zhao et al. 2023 ) . However, potential safety risks such as harmful content generation still remain key concerns for stakeholders ( Wen et al. 2023 , Li et al. 2024 ) . LLMs may fail to refuse to output harmful content, no matter whether they are induced by well-designed malicious prompts or just receive benign requests, which may cause undesirable societal consequences ( Chao et al. 2023 , Deng et al. 2024 , Liu et al. 2024 , Yuan et al. , Jin et al. 2024 ) . To mitigate harmful outputs, safety alignment techniques are applied to enhance LLMs’ intrinsic capability of refusing harmful outputs ( Bai et al. 2022 , Rafailov et al. 2023 , Yuan et al. 2025 ) , but current alignment still cannot maintain helpfulness without trading harmlessness, leaving harmful output risks unsolved ( Tran et al. 2025 , Greenblatt et al. 2024 , Qi et al. 2025 ) . This necessitates the equipment of a content moderator as the second safety guardrail for LLM services in real-world production environments, which oversees LLM-user interactions and checks whether LLM-generated outputs conform to the corresponding safety specifications before they are delivered ( Dong et al. 2024 , AWS 2024 , Jarvis 2023 ) . The existing moderation paradigm primarily fall into the full detection , which executes in a post-hoc manner, i.e . , determining the harmfulness after receiving the complete LLM output. This aligns with conventional content moderation applied to human-written contents, where moderators are trained using complete text pieces based on encoder-only or decoder-only language models ( Eskelinen et al. 2023 , Logacheva et al. 2022 , Lin et al. 2023 ) and some have been developed as online services ( e.g . , Google Perspective API ( Perspective 2024 ) , OpenAI Moderation API ( OpenAI 2024 ) , and LlamaGuard ( Inan et al. 2023 ) ). Due to their reliance on complete semantics expressed in complete outputs, full detection may cause high service latency. To reduce the impact of service latency on user experience, partial detection has received more attention recently, where moderators may interfere with the generation if harmfulness is above the detection threshold midway. For example, ProtectAI 2023 and GuardrailsAI 2023 directly apply moderators trained using the full detection paradigm, where the incomplete LLM response at a certain step is regarded as complete for harmfulness judgment during generation. It undoubtedly reduces the latency and performs a more timely moderation, but it requires repeated encoding of the output content at each step and will cause redundancy costs. More importantly, an inherent gap exists between the training and inference because the applied moderators are intended to make judgments based on complete semantics. This results in a relatively late stopping of harmful outputs as the moderator is better at judging with more complete responses. Ideally, a streaming content moderator for native partial detection is capable of predicting potential unsafe behaviors as early as possible with only incomplete responses available. Though the mentioned works have emerged, there are still challenges for constructing such a moderator: 1) How to obtain fine-grained token-level annotations for LLM-generated responses; And 2) how to improve the moderator’s capability to predict the potential harmful behaviors with incomplete responses only. In this paper, we propose a data-and-model solution for constructing a harmful content moderator that natively supports partial detection: 1) For data acquisition, we construct a harmful content detection dataset FineHarm consisting of 29K prompt-response pairs that contain token-level annotations. 2) For moderator construction, propose the plug-and-play Streaming Content Monitor (SCM) which is specially designed to work in parallel with the LLM streaming generation, and design hierarchical consistency-aware learning strategy to guide token-level learning on incomplete semantics with future response-level knowledge. Along with the LLM output streaming, SCM fetches the latest token at each step and provides a harmfulness judgment. When the number of tokens that are judged as harmful reaches a preset threshold, this output will be considered harmful and terminated midway. Experiments demonstrate that our proposed partial detection SCM achieves comparative performance (95%+) with full detection models by only seeing the first 18% of tokens in responses on average. 2 Related Work LLM Harmful Output Mitigation To mitigate harmfulness of LLMs’ output, an intuitive way is to eliminate the harmful instructions before they are fed into the language model, such as detection-based intercepting ( Alon and Kamfonas 2023 , Jain et al. 2023 ) , perturbation-based filtering ( Cao et al. 2023 , Robey et al. 2023 , Kumar et al. 2023 ) , and system-prompt-based guarding ( Sharma et al. 2024 , Zou et al. 2024 , Zheng et al. 2024 ) . To internalize the safety awareness into LLMs, safety alignment techniques have been widely applied, which fine-tunes the language model to enhance the intrinsic safety guardrails so that the models