Quantifying Conversation Drift in MCP via Latent Polytope

Haoran Shi, Hongwei Yao, Shuo Shao, Shaopeng Jiao, Ziqi Peng, Zhan Qin, Cong Wang · 2025

Read Quantifying Conversation Drift in MCP via Latent Polytope by Haoran Shi, Hongwei Yao, Shuo Shao, Shaopeng Jiao, Ziqi Peng, Zhan Qin, Cong Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-06418.txt.

Introduction

Haoran Shi and co-authors present a defense for language models that use outside tools in this 2025 paper. A new protocol lets models fetch live data and call tools, but that open setting invites attacks. Planted content can poison a tool or slip in hidden prompts, which can hijack a chat, spread false claims, or leak data.

Their framework, called SecMCP, watches for drift in the chat. It models the model's inner activity as a shape in a hidden space and flags odd shifts in that path. Tests on three leading models and three data sets gave strong detection scores. The authors also sort the main threat types for this protocol.

Catalog record

CREATORHaoran Shi, Hongwei Yao, Shuo Shao, Shaopeng Jiao, Ziqi Peng, Zhan Qin, Cong Wang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-06418

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Quantifying Conversation Drift in MCP via Latent Polytope Haoran Shi, Hongwei Yao, Shuo Shao, Shaopeng Jiao, Ziqi Peng, Zhan Qin, Cong Wang Abstract The Model Context Protocol (MCP) enhances large language models (LLMs) by integrating external tools, enabling dynamic aggregation of real-time data to improve task execution. However, its non-isolated execution context introduces critical security and privacy risks. In particular, adversarially crafted content can induce tool poisoning or indirect prompt injection, leading to conversation hijacking , misinformation propagation , or data exfiltration . Existing defenses, such as rule-based filters or LLM-driven detection, remain inadequate due to their reliance on static signatures, computational inefficiency, and inability to quantify conversational hijacking. To address these limitations, we propose SecMCP , a secure framework that detects and quantifies conversation drift , deviations in latent space trajectories induced by adversarial external knowledge. By modeling LLM activation vectors within a latent polytope space, SecMCP identifies anomalous shifts in conversational dynamics, enabling proactive detection of hijacking, misleading, and data exfiltration. We evaluate SecMCP on three state-of-the-art LLMs (Llama3, Vicuna, Mistral) across benchmark datasets (MS MARCO, HotpotQA, FinQA), demonstrating robust detection with AUROC scores exceeding 0.915 while maintaining system usability. Our contributions include a systematic categorization of MCP security threats, a novel latent polytope-based methodology for quantifying conversation drift, and empirical validation of SecMCP ’s efficacy. Introduction In recent years, large language models (LLMs) such as ChatGPT, Claude, and DeepSeek ( Achiam et al. 2023 ) have demonstrated remarkable success across a wide range of tasks, including language understanding, machine translation, and question answering. Despite these advances, the effectiveness of state‑of‑the‑art (SoTA) models remains constrained by their limited capacity to access external data and interact with real‑world. In practice, LLMs rely heavily on contextual cues provided within the input to infer background knowledge, interpret semantic relations, and capture dependencies among information fragments. This contextual reasoning not only supports more accurate task execution and question answering but also enhances model generalization across diverse downstream domains. To mitigate these limitations, Anthropic recently introduced the Model Context Protocol (MCP) , a framework designed to extend LLM functionality through integration with external tools such as web search engines and knowledge databases. MCP enables LLMs to dynamically aggregate information from multiple contextual streams, thereby supporting real-time decision making and adaptive service delivery. For instance, a web search tool allows retrieval of up-to-date news and wikipedia, while knowledge database tools facilitate access to specialized domain corpora. Despite these advantages, MCP introduces critical security and privacy risks due to its reliance on a non-isolated execution context , where multiple data streams coexist within a shared operational space ( Yao et al. 2025 ) . This design, while optimized for performance, creates an attack surface for adversaries. Malicious servers may exploit this environment by embedding adversarial instructions into retrieved content, leading to tool poisoning or indirect prompt injection ( Yao, Lou, and Qin 2024 ) . Such attacks can result in hijacking of the model’s behavior, the introduction of misleading information, or even the exfiltration of sensitive data, undermining the reliability of MCP-enabled systems. Figure 1: Overall architecture and workflow of the MCP-powered agent system. Existing defense mechanisms remain insufficient ( He et al. 2025a ) . Rule-based methods (e.g., regular expressions or semantic similarity filters) rely heavily on predefined attack signatures, rendering them ineffective against previously unseen threats ( Jacob et al. 2025 ) . Detection approaches that directly leverage LLMs introduce significant computational overhead and often achieve limited success rates. More critically, current techniques fail to quantify the degree of conversational hijacking or hallucination, limiting their utility for fine-grained risk assessment in MCP-powered agent system. To address these challenges, we propose SecMCP , a secure MCP framework that detects and quantifies conversation drift induced by adversarial external knowledge. Our key insight is that adversarial instructions, while often benign in surface text, activate distinct clusters of neurons in the latent space, thereby shifting the trajectory of conversation generation. Building on this observation, SecMCP leverages activation vector representations of LLM queries and models conversational dynamics within a latent polytope space. By quantifying deviations from expected conversational trajectories, SecMCP enables proactive detection of data exfiltration, misleading, and hijacking. We implement MCP with simulated web search and knowledge database tools, and evaluate SecMCP on three SoTA open‑source LLMs—Llama3, Vicuna, and Mistral—across three widely used benchmark datasets: MS MARCO, HotpotQA, and FinQA. Experimental results demonstrate that SecMCP achieves robust security detection, with AUROC scores consistently exceeding 0.915, while preserving normal MCP functionality. The main contributions of this work are as follows: • Systematic Risk Analysis : We provide a comprehensive categorization of security threats in MCP-powered agent systems, identifying three primary risks—hijacking, misleading, and data exfiltration—and establishing a framework for subsequent research. • Secure MCP Framework : We introduce SecMCP , which detects and quantifies conversation drift through latent polytope analysis, enabling effective identification of adversarial manipulations in MCP interactions. • Extensive Evaluation : We validate the effectiveness and robustness of SecMCP through experiments on multiple SoTA LLMs and benchmark datasets, demonstrating both its security benefits and its negligible impact on system usability. Related Works LLM Attacks In the past few years, security risks associated with LLMs have garnered significant attention from the research community. This section provides an in-depth review of existing literature on the subject, with a particular focus on issues related to prompt injection. Prompt injection Prompt injection attacks have emerged as a serious security threat to LLMs, enabling adversaries to manipulate outputs by exploiting the model’s sensitivity to crafted input instructions. Early studies such as ( Perez and Ribeiro 2022 ) demonstrated the feasibility of semantic jailbreaks by appending override instructions to prompts, while later work ( Zou et al. 2024a ) introduced more systematic methods using gradient-based token optimization, creating transferable jailbreak prompts that remain effective across models. Beyond direct prompt manipulation, recent efforts like ( Liu et al. 2023 ) developed black-box injection techniques inspired by web attacks. On the defense side, efforts have diversified into both prevention and detection strategies. Structural approaches like ( Chen et al. 2024 ) aim to isolate model instructions from user data by enforcing rigid input formats. Authentication-based defenses, such as ( Suo 2024 ) , rely on cryptographically signed prompts to ensure input integrity. Dynamic defenses have also gained traction: ( Phute et al. 2024 ) proposed RA-LLM, which employs a secondary LLM to audit outputs for harmful content. ( Zhong et al. 2025 ) applies dynamic information-flow control in TBAS via dependency screening and region masking. Overall, although various defense methods have been proposed to date, most operate by preventing or detecting attacks solely at the LLM’s input or output interfaces, and there exists no mature technique that leverages internal model information (e.g. activation) for defense. MCP security As the MCP protocol has only been recently introduced, discussions surrounding its security are still in the early stages. ( Narajala, Huang, and Habler 2025 ) proposes a Tool Registry system to address issues such as tool squatting—the deceptive registration or misrepresentation of tools. ( Radosevich and Halloran 2025 ) introduces MCPSafetyScanner, an agentic tool designed to assess the security of arbitrary MCP servers. ( Narajala and Habler 2025 ; Hou et al. 2025 ) provide a

Read the complete text