Adversarial Attacks on LLM-as-a-Judge Systems: Insights from Prompt Injections

Narek Maloyan, Dmitry Namiot · 2025

Read Adversarial Attacks on LLM-as-a-Judge Systems: Insights from Prompt Injections by Narek Maloyan, Dmitry Namiot (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-18333.txt.

Introduction

Narek Maloyan and Dmitry Namiot probe the safety of systems that use a language model to judge text quality. Their 2025 paper tests prompt injection attacks, where hidden instructions trick the judge. The framework splits attacks into two kinds: those planted by the content author and those aimed at the system prompt.

Five models, from small open ones to GPT-4 and Claude 3 Opus, faced attacks across four judging tasks with several defenses in place. Success rates reached about 74 percent, and smaller models proved easier to fool. Attacks also carried over between models about half the time. The authors suggest judging panels of several models and scoring that compares answers side by side, and they release their code and data.

Catalog record

CREATORNarek Maloyan, Dmitry Namiot
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE55 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-18333

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Adversarial Attacks on LLM-as-a-Judge Systems: Insights from Prompt Injections Narek Maloyan, Dmitry Namiot Abstract Large Language Models (LLMs) are increasingly used as automated judges for evaluating text quality, code correctness, and argument strength. However, these LLM-as-a-judge systems are vulnerable to adversarial attacks that can manipulate their assessments. This paper investigates the vulnerability of LLM-as-a-judge systems to prompt injection attacks, drawing insights from both academic literature and practical solutions from the ”LLMs: You Can’t Please Them All” Kaggle competition. We present a comprehensive framework for developing and evaluating adversarial attacks against LLM judges, distinguishing between content-author attacks and system-prompt attacks. Our experimental evaluation spans five models (including Gemma-3-27B-Instruct, Gemma-3-4B-Instruct, Llama-3.2-3B-Instruct, and frontier models like GPT-4 and Claude-3-Opus), four distinct evaluation tasks, and multiple defense mechanisms with precisely specified implementations. Through rigorous statistical analysis (n=50 prompts per condition, bootstrap confidence intervals), we demonstrate that sophisticated attacks can achieve success rates of up to 73.8% against popular LLM judges, with Contextual Misdirection being the most effective method against Gemma models at 67.7%. We find that smaller models like Gemma-3-4B-Instruct are more vulnerable (65.9% average success rate) than their larger counterparts, and that attacks show high transferability (50.5-62.6%) across different architectures. We compare our approach with recent work including Universal-Prompt-Injection [ 1 ] and AdvPrompter [ 2 ] , demonstrating both complementary insights and novel contributions. Our findings highlight critical vulnerabilities in current LLM-as-a-judge systems and provide recommendations for developing more robust evaluation frameworks, including using multi-model committees with diverse architectures and preferring comparative assessment over absolute scoring methods. To ensure reproducibility, we release our code, evaluation harness, and processed datasets. I Introduction Large Language Models (LLMs) have emerged as powerful tools for evaluating text quality, code correctness, and argument strength [ 3 , 4 ] . These LLM-as-a-judge systems offer scalable, cost-effective alternatives to human evaluation, with studies showing high correlation between LLM and human judgments in many domains [ 5 ] . As a result, LLM judges are increasingly deployed in educational settings, programming competitions, and research benchmarks. However, the reliability of these systems depends on their robustness against adversarial manipulation. Recent work has demonstrated that LLMs are vulnerable to various forms of adversarial attacks, particularly prompt injections that can override their intended behavior [ 6 , 7 ] . These vulnerabilities raise serious concerns about the trustworthiness of LLM-based evaluation systems, especially in high-stakes contexts. The ”LLMs: You Can’t Please Them All” Kaggle competition [ 8 ] specifically challenged participants to create inputs that would cause LLM judges to diverge in their assessments, highlighting the potential for manipulating these systems. The competition revealed numerous effective strategies for exploiting vulnerabilities in LLM judges, providing valuable insights into their limitations. In this paper, we investigate the vulnerability of LLM-as-a-judge systems to adversarial attacks, with a focus on prompt injection techniques. Our work builds upon and extends recent advances in this area, including Universal-Prompt-Injection [ 1 ] and AdvPrompter [ 2 ] , while offering several novel contributions: • A comprehensive analysis of existing approaches to adversarial attacks on LLMs in evaluation contexts, with clear distinction between content-author attacks (where malicious text is submitted for evaluation) and system-prompt attacks (where the evaluation template itself is compromised) • Detailed examination of top solutions from the ”LLMs: You Can’t Please Them All” Kaggle competition, revealing practical attack strategies not previously documented in academic literature • A novel framework for developing and evaluating adversarial attacks on LLM-as-a-judge systems, with precisely specified components that facilitate systematic comparison • Rigorous experimental evaluation across five models (including both open-source and frontier proprietary models), four diverse evaluation tasks, and multiple defense mechanisms with fully specified implementations • Quantitative evidence for the effectiveness of multi-model committees as a defense mechanism, with statistical analysis of committee voting patterns • Recommendations for enhancing the robustness of LLM-as-a-judge systems based on empirical findings Our findings demonstrate that current LLM-as-a-judge systems remain highly vulnerable to sophisticated adversarial attacks, with important implications for their deployment in real-world applications. We show that combining multiple defense approaches—particularly using diverse model committees and comparative assessment—significantly enhances robustness. II Related Work II-A LLM-as-a-Judge Systems Large language models have demonstrated impressive capabilities as zero-shot assessors for various tasks. Wang et al. [ 4 ] showed that LLMs can effectively evaluate text quality with minimal prompting. Zheng et al. [ 3 ] found that LLMs can serve as reliable judges for comparing the outputs of different AI systems. Liu et al. [ 5 ] demonstrated high correlation between GPT-4 evaluations and human judgments across multiple domains. However, these systems are not without limitations. Several studies have identified biases in LLM evaluations, including positional bias [ 9 ] , length bias [ 10 ] , and self-preferential behaviors [ 11 ] . These biases can significantly impact the reliability of LLM judgments, even without adversarial manipulation. II-B Adversarial Attacks on LLMs Adversarial attacks on LLMs have received increasing attention as these models are deployed in more critical applications. Perez and Ribeiro [ 6 ] demonstrated that prompt injection attacks can manipulate LLMs by inserting malicious instructions that override their intended behavior. Zou et al. [ 12 ] showed that universal adversarial phrases can be effective across different models and contexts. Liu et al. [ 7 ] introduced HOUYI, a black-box prompt injection attack framework with three components: a framework component that blends the attack with the application’s natural flow, a separator component that shifts the LLM’s focus, and a disruptor component that executes the malicious intent. This framework achieved an 83.4% success rate across 36 LLM-integrated applications. More recently, Yan et al. [ 13 ] proposed Virtual Prompt Injection (VPI), which targets instruction-tuned LLMs by poisoning their training data. This approach demonstrates that vulnerabilities can be introduced during the model training process, not just at inference time. Recent work by Liu et al. [ 1 ] introduced Universal-Prompt-Injection, an adaptive approach that iteratively refines attack strings based on model feedback. Similarly, Paulus et al. [ 2 ] developed AdvPrompter, which uses gradient-free optimization to generate adversarial prompts. These approaches represent the current state-of-the-art in automated attack generation and provide important baselines for our work. II-C Defenses Against Adversarial Attacks Several defense mechanisms have been proposed to protect LLMs from adversarial attacks. Chen et al. [ 14 ] introduced StruQ, a system that separates prompts and data into distinct channels to prevent injection attacks. Liu et al. [ 7 ] developed LLM-based detection methods that use a separate model to identify potential attacks. Liu et al. [ 15 ] categorized defenses into prevention-based approaches (which aim to prevent attacks from reaching the model) and detection-based approaches (which identify and mitigate attacks after they occur). Prevention methods include input filtering, prompt engineering, and sandboxing, while detection methods include perplexity checks, output verification, and anomaly detection. More recent approaches include context sanitization [ 16 ] , instruction isolation [ 17 ] , and baseline defense combinations [ 18 ] . These methods offer promising directions for enhancing robustness but have not been systematically evaluated against sophisticated attacks in evaluation contexts. Despite these advances, no single defense mechanism provides complete protection against sophisticated attacks. As noted by Bhandari et al. [ 19 ] and Zhu et al. [

Read the complete text