JavelinGuard: Low-Cost Transformer Architectures for LLM Security
Yash Datta, Sharath Rajasekar · 2025
Read JavelinGuard: Low-Cost Transformer Architectures for LLM Security by Yash Datta, Sharath Rajasekar (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-07330.txt.
Introduction
Yash Datta and Sharath Rajasekar present JavelinGuard in 2025, a set of small guard models for language models. Guards screen chats for malicious intent before harm is done. Big guards cost too much for live use. The team aim for high accuracy on plain hardware.
Five transformer designs grow in sophistication across the suite. They range from a baseline classifier to a multi-task frame with custom losses. Models run at a few hundred million parameters and infer fast on CPUs. Nine adversarial datasets test them, plus a new bench of hard borderline cases. Against open guards and a top commercial model, the suite won on cost for its performance level.
Catalog record
| CREATOR | Yash Datta, Sharath Rajasekar |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 41 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-07330
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
JavelinGuard: Low-Cost Transformer Architectures for LLM Security Yash Datta, Sharath Rajasekar Abstract We present JavelinGuard , a suite of low-cost, high-performance model architectures designed for detecting malicious intent in Large Language Model (LLM) interactions, optimized specifically for production deployment. Recent advances in transformer architectures, including compact BERT [ Devlin et al. 2019 ] variants (e.g., ModernBERT [ Warner et al. 2024 ] ), allow us to build highly accurate classifiers with as few as approximately 400M parameters that achieve rapid inference speeds even on standard CPU hardware. We systematically explore five progressively sophisticated transformer-based architectures: Sharanga (baseline transformer classifier), Mahendra (enhanced attention-weighted pooling with deeper heads), Vaishnava and Ashwina (hybrid neural-ensemble architectures), and Raudra (an advanced multi-task framework with specialized loss functions). Our models are rigorously benchmarked across nine diverse adversarial datasets, including popular sets like the NotInject series, BIPIA, Garak, ImprovedLLM, ToxicChat, WildGuard, and our newly introduced JavelinBench , specifically crafted to test generalization on challenging borderline and hard-negative cases. Additionally, we compare our architectures against leading open-source guardrail models as well as large decoder-only LLMs such as gpt-4o , demonstrating superior cost-performance trade-offs in terms of accuracy, and latency. Our findings reveal that while Raudra’s multi-task design offers the most robust performance overall, each architecture presents unique trade-offs in speed, interpretability, and resource requirements, guiding practitioners in selecting the optimal balance of complexity and efficiency for real-world LLM security applications. Keywords: Jailbreak detection, Prompt injection, Large Language Models, Transformer-based models, Hybrid architecture, Multi-task classification 1 Introduction Large Language Models (LLMs) have become integral to modern software applications, powering functionalities from code assistants to customer service chatbots. As these models grow in complexity and capability, they also become susceptible to prompt injection and jailbreak attacks . In parallel, LLM outputs can pose risks of harm if not properly safeguarded, prompting research on specialized moderation or guardrail models designed to screen both user inputs and model-generated text. Despite progress in open-source and commercial guardrails [ Padhi et al. 2024 , Li and Liu 2024 , Zhou et al. 2025 , Kim et al. 2023 , lakera.ai 2024 ] , many detection solutions face challenges in balancing accuracy, cost, speed, and over-fitting to known attack patterns. Besides, none of the existing solutions differentiate between jailbreak and prompt-injection classes, considering both as one in most cases or catering to either one of these categories. Further, there is a lack of classifiers that differentiate between these two classes and toxic content that is often wrongly classified as a jailbreak attempt. Our work systematically explores five architectures that incrementally build upon one another, and give high-accuracy feedback on both of these classes simultaneously, without mislabeling toxic content as jailbreak. These are five different series of models (named after celestial weapons from the Indian epic Mahabharata) • Sharanga: Baseline Transformer Classification • Mahendra: Enhanced Pooling and Loss Functions • Vaishnava: Hybrid Neural-Forest Architecture • Ashwina: Hybrid Neural-XGBoost Architecture • Raudra: Advanced Multi-Task Framework We test these architectures on nine benchmarks encompassing a variety of adversarial scenarios, including short override prompts, multi-turn dialogues, domain-specific manipulations, code-oriented injections, and toxic content. Our results demonstrate that while each approach has its merits, ( Raudra ) consistently yields superior performance across metrics of accuracy, precision, recall, and robustness to out-of-distribution attacks. We also share performance of existing open-source models on the same benchmarks to give an idea of the superior performance of the suggested architectures. The remainder of this paper is organized as follows: Section 2 surveys prior work in LLM security and prompt injection detection. Section 3 describes our five architectures in detail, including the motivation behind each design choice. Section 5 presents experimental findings across the nine benchmarks. Finally, Section 7 concludes with future research directions. 2 Related Work Prompt Injection Attacks. Prompt injection attacks specifically target LLMs’ reliance on natural language instructions. Since LLMs often cannot discern malicious intentions encoded in apparently benign text, attackers can coerce them into revealing system prompts or performing unintended actions [ Perez and Ribeiro 2022 , Zou et al. 2023 , Qi et al. 2023 , Wei et al. 2023 ] . These attacks are either prompt-engineering based [ Perez and Ribeiro 2022 , Liu et al. 2024b ] , or gradient-based [ Huang et al. 2024 , Shi et al. 2024 ] . Several Prompt injection datasets such as PINT [ lakera.ai 2024 ] , InjecAgent [ Zhan et al. 2024 ] , TaskTracker [ Abdelnabi et al. 2024 ] , and BIPIA [ Yi et al. 2023 ] have been introduced to benchmark detection methods against these attacks. Several techniques are proposed for defense against these attacks [ Wang et al. 2023 ] like handcrafted approaches [ Toyer et al. 2023 ] , fully automated algorithms [ Liu et al. 2024a ] , fine-tuning, prompt-engineering, or other clever techniques [ Chen et al. 2025 ] . Jailbreak Attacks. While jailbreak attacks also manipulate LLM behavior through malicious instructions, they differ from prompt injection by explicitly overriding the model’s policy constraints (e.g., instructing the model to “ignore previous instructions”) rather than merely embedding deceptive commands in otherwise benign text. Recent studies highlight diverse techniques for inducing jailbreaks: Huang et al. 2023 examine decoding variations that elicit unintended model responses, Yu et al. 2024 employ fuzzing to mutate numerous prompts, and Doumbouya et al. 2024 present a multi-step iterative algorithm modeling jailbreaks as compositions of string-to-string transformations. Other works frame jailbreak prompting as a quality-diversity search [ Samvelyan et al. 2024 ] or utilize random token optimization [ Andriushchenko et al. 2024 ] . Another recent automated red-teaming method called BoN [ Hughes et al. 2024 ] explored jailbreaking by repeatedly sampling augmentations to prompts until target LLM produces a harmful response. Although sophisticated jailbreak methods also exist for multimodal models, our focus remains strictly on textual LLMs. Early text-based defenses included backtranslation [ Wang et al. 2024 ] and representation engineering [ Zou et al. 2024 ] , but alignment-based mitigations alone have proven insufficient [ Wolf et al. 2024 ] . Consequently, external guardrails [ Rebedea et al. 2023 ] and “LLM Firewalls”—sometimes even backed by vector databases—have gained traction for detecting and mitigating jailbreak attempts. Ensuring these methods remain both accurate and efficient in real-world, resource-constrained deployments continues to be an open challenge. Prompt Guard Models. A variety of prompt guard models specifically focus on classifying malicious or manipulative text inputs before they reach the main LLM. These smaller classifiers eschew multiple inference passes in favor of direct binary or multi-class decisions, thereby being much more efficient as guard-rails in production. Examples include Fmops [ fmops 2024 ] , Deepset [ Deepset 2024 ] , PromptGuard [ Meta 2024 ] , and ProtectAIv2 [ ProtectAI.com 2024b ] , most of which rely on mid-sized transformer backbones like DistilBERT [ Sanh et al. 2020 ] or DeBERTaV3-base [ He et al. 2023 ] . Li and Liu 2024 pointed out recently that most of these models suffer from over-defense issue, which is mitigated by careful data curation after analyzing common keyword shortcuts. LakeraGuard [ lakera.ai 2024 ] is a commercial offering with undisclosed training and architecture details. InjecGuard [ Li and Liu 2024 ] itself uses DeBERTaV3-base as