MCP Safety Training: Learning to Refuse Falsely Benign MCP Exploits using Improved Preference Alignment

John T. Halloran · 2025

Read MCP Safety Training: Learning to Refuse Falsely Benign MCP Exploits using Improved Preference Alignment by John T. Halloran (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-23634.txt.

Introduction

John Halloran warns that AI agents wired to outside tools can be hijacked by web pages that merely look harmless, in a 2025 paper. The Model Context Protocol links agents to files and services. Earlier work showed attacks needing a victim to download a poisoned file first.

Halloran shows the trap is wider: posting malicious content online can be enough to fool an agent into granting access or stealing credentials. A new data set pairs such falsely benign attacks with truly benign samples. Preference training then taught models to refuse the traps, improving guardrails while keeping normal help intact. Safer refusal habits are urgent as agents gain keys to real systems.

Catalog record

CREATORJohn T. Halloran
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE41 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-23634

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MCP Safety Training: Learning to Refuse Falsely Benign MCP Exploits using Improved Preference Alignment John T. Halloran Abstract The model context protocol (MCP) [ 5 ] has been widely adapted as an open standard enabling the seamless integration of generative AI agents. However, recent work has shown the MCP is susceptible to retrieval-based “falsely benign” attacks (FBAs) [ 38 ] , allowing malicious system access and credential theft, but requiring that users download compromised files directly to their systems. Herein, we show that the threat model of MCP-based attacks is significantly broader than previously thought, i.e., attackers need only post malicious content online to deceive MCP agents into carrying out their attacks on unsuspecting victim’s systems. To improve alignment guardrails against such attacks, we introduce a new MCP dataset of FBAs and (truly) benign samples to explore the effectiveness of direct preference optimization (DPO) for the refusal training of large language models (LLMs). While DPO improves model guardrails against such attacks, we show that the efficacy of refusal learning varies drastically depending on the model’s original post-training alignment scheme–e.g., GRPO-based LLMs learn to refuse extremely poorly. Thus, to further improve FBA refusals, we introduce R etrieval A ugmented G eneration for Pref erence alignment (RAG-Pref), a novel preference alignment strategy based on RAG [ 31 ] . We show that RAG-Pref significantly improves the ability of LLMs to refuse FBAs, particularly when combined with DPO alignment, thus drastically improving guardrails against MCP-based attacks. 1 Introduction The model context protocol (MCP) [ 5 ] has been recently released as an open protocol for connecting generative AI components. By standardizing API calls between large language models (LLMs), supported tools, and data sources, the MCP serves as a universal protocol to seamlessly integrate agents across widely used services/applications, thus replacing the previous fragmented approach of designing application-specific agentic APIs. Subsequently, the MCP has been widely adapted by major services–e.g., Google Cloud [ 21 ] , Slack [ 7 ] , Copilot [ 35 ] , Stripe [ 41 ] , HuggingFace Tiny Agents [ 11 ] –and industry-leading LLMs–e.g., Anthropic’s Claude [ 6 ] , OpenAI’s gpt-4o / o1 / o3 / o4 [ 36 ] , and Google’s Gemma / Gemini [ 20 ] . However, recent work has shown that the MCP enables security risks [ 38 , 28 , 29 ] . In particular, [ 38 ] showed that while aggressive attacks (AAs)–i.e., attack prompts which explicitly state harmful phrases or suspicious text–triggered refusals from both Claude and Llama-3 models, requests which were falsely benign attacks (FBAs)–i.e., attack prompts without harmful phrases which maintain a casual/neutral cadence–were completed by the respective LLMs. Furthermore, refusal mechanisms from both Claude 3.7 , and Llama-3.3-70B were shown to rely heavily on attack cues from AAs which, when removed, resulted in successful FBAs. 1 1 1 Attacks encoded in octal and harmful/cyber-attack phrases (e.g., “hack,” “backdoor,”, “steal”) were directly refused by Claude 3.7 and Llama-3.3-70B , respectively. However, the former encoded in plaintext was successfully performed by Claude 3.7 , and Llama-3.3-70B completed requests when only the harmful/cyber-attack phrase was removed. . By utilizing the lack of refusals for FBAs, [ 38 ] further showed that a class of new retrieval-based attacks, called R etrieval- A gent DE ception (RADE), were possible via the MCP. While effective at enabling various attacks, RADE is inherently limited by the requirement that users must download specific manipulated files onto their systems. However, we show that the threat model of MCP attacks is significantly broader than previously thought . We present a new MCP attack framework, Total Retrieval-Agent DEception (TRADE), wherein an an attacker need only post an FBA online to enable retrieval-based MCP attacks . To combat TRADE and other (as of yet) unknown MCP-related attacks, we explore the use of preference fine-tuning [ 39 ] to increase the ability of LLMs to refuse MCP FBAs. To address the current lack of MCP attack data, we create MCP-FBAs , a new high-quality dataset of FBAs and truly benign (TB) samples. Furthermore, in contrast to LLM refusal [ 9 , 10 , 8 , 45 , 22 ] and agentic attack [ 16 , 23 , 13 ] work, we introduce and compare new LLM refusal metrics which reflect practical LLM inference settings and the immediate impact of MCP-enabled attacks. Using these metrics and MCP-FBAs , we show that widely-used LLMs –many of which have gone through extensive safety alignment [ 22 , 42 , 43 ] – have difficulty refusing FBAs, with an average 8.5% and best 23.8% strict refusal rating across all models . To improve MCP-attack refusal capabilities, we use one of the most widely used alignment algorithms, direct preference optimization (DPO) [ 39 ] , to align a large number of LLMs (varying by instruction-tuning algorithm) to refuse FBAs and comply with TB requests. However, we show that DPO-aligned LLMs display limited FBA-refusal ability (average 87% strict refusal improvement across all models). In particular, GRPO-based reasoning models display especially poor refusal-learning (average 45% strict refusal improvement across such models). To thus further improve the FBA refusal ability of LLMs, we introduce R etrieval A ugmented G eneration for Pref erence alignment (RAG-Pref), a novel RAG algorithm designed to supplement an LLM’s safety alignment knowledge. Compared to offline (training-based) alignment using DPO, RAG-Pref greatly improves the refusal ability of all considered LLMs , resulting in an average 247% strict refusal improvement across all models without any model training . Furthermore, we show that both online alignment (using RAG-Pref) and offline alignment are complimentary to one another, with RAG-Pref increasing the strict refusal ability of DPO-aligned models by an average 465%. Importantly, these successive refusal improvements are reflected by the reasoning models considered, with GRPO-based models displaying strict refusal averages of 323% and 542% using RAG-Pref and RAG-Pref combined with DPO-alignment, respectively. Our main contributions are as follows: • TRADE, a new MCP attack framework with a wide threat model , and successful attack demonstrations on Claude 3.7 Sonnet . • Stricter refusal metrics for LLMs, reflecting the real-world impact and severity of MCP-targeted attacks. • The first high-quality, open-source MCP-attack dataset, MCP-FBAs , containing a large number of training/testing FBAs and TB samples. • The first study on the effectiveness of preference alignment for improving LLM guardrails against MCP-targeted attacks. • RAG-Pref , a new RAG-based preference alignment algorithm, which drastically improves FBA refusal rates for original and DPO-aligned LLMs. 2 Total Retrieval-Agent Deception (TRADE) Figure 1: Threat model for a TRADE attack. An attacker posts content online, in this case, a webpage for a vegan black bean chili recipe. The webpage also contains an FBA with targeted commands (bottom red text) centered around a specific theme (“X” in the figure). The MCP user requests that the website’s content be added to a vector database. When the user requests for content related to these themes, the FBA commands are executed (e.g., in the figure, the attacker is granted remote access to the victim’s system). [ 38 ] demonstrated RADE could successfully use MCP servers for malicious attacks. In RADE, an attacker compromises publicly available files with an FBA curated around a specific topic. When an

Read the complete text