On the Geometric Limits of Transformer Defenses against Obfuscation Attacks: Latent Embedding Collapse & Performance–Robustness Gap
Becky Mashaido and Tapadhir Das · 2026
Read On the Geometric Limits of Transformer Defenses against Obfuscation Attacks: Latent Embedding Collapse & Performance–Robustness Gap by Becky Mashaido and Tapadhir Das (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-19159.txt.
Introduction
Becky Mashaido and Tapadhir Das show that detection scores for prompt-injection defences do not mean robust maps. Doctored prompts mixing look-alike letters, zero-width marks, and punctuation noise can land on the embedding map of clean prompts, a quirk they call latent embedding collapse. On BERT-family encoders of depth and size, detectors score near-perfectly, yet the smallest clean-doctored gap is 1.02, a sign of near-overlap. Doctored embeddings vary more within each class, showing inner instability and a score-robustness gap that metrics miss.
Catalog record
| CREATOR | Becky Mashaido and Tapadhir Das |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 21532 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.19159
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
On the Geometric Limits of Transformer Defenses against Obfuscation Attacks: Latent Embedding Collapse & Performance–Robustness Gap Becky Mashaido and Tapadhir Das Abstract Prompt-injection attacks pose significant risks to language model safety, yet existing defenses are typically evaluated using classification performance. We show that high detection performance does not imply representational robustness. Specifically, multi-operator obfuscated prompts (combining homoglyphs, zero-width characters, and punctuation or emoji noise) can partially collapse onto the embedding manifold of clean prompts, a phenomenon we term l a t e n t latent e m b e d d i n g embedding c o l l a p s e collapse . Results indicate that across multiple BERT-family encoders with varying depth and capacity, detectors achieve near-perfect classification performance, yet the minimal clean–obfuscated margin δ = 1.02 \delta=1.02 , indicating near-overlap of obfuscated and clean embeddings. Obfuscated embeddings further exhibit elevated intra-class variance ( 3.33 ± 6.23 3.33\pm 6.23 ), indicating severe latent-space instability despite high performance. These results reveal a substantial p e r f o r m a n c e – r o b u s t n e s s performance–robustness g a p gap , demonstrating that standard evaluation metrics fail to capture latent embedding collapse and underlying geometric fragility. Our findings show that increasing model capacity does not eliminate latent embedding collapse, motivating geometry-aware robustness analysis as a necessary complement to performance-based evaluation for prompt-injection defenses. Email: [email protected], [email protected] I Introduction Large language models (LLMs) have become a ubiquitous part of our daily personal & professional lives. These are power neural network models that are trained on a wide variety of text and code to comprehend, reproduce, and handle human language [ 8 ] . Due to their widespread usage, the market size of LLMs are expected to hit $35.43 billion by 2030 [ 9 ] . However, despite their extended use, LLMs can be compromised by cyber attacks such as prompt injections [ 13 ] . Prompt-injection attacks pose significant risks to the safety of LLMs by embedding malicious instructions within seemingly benign user inputs. Such attacks can override system directives, manipulate model behavior, or exfiltrate sensitive information without explicit policy violations [ 16 ] . An illustration of prompt injection can be seen in Figure 1 . Fig. 1 : Illustration of Prompt Injection Attack on LLMs As a result, a growing body of work proposes prompt-injection detection mechanisms based on supervised classification, typically evaluated using accuracy, precision, recall, or F1 score [ 6 , 3 ] . While these metrics capture observable misclassification behavior, they provide limited insights into the robustness of the underlying representations learned by transformers. In particular, a detector may achieve near-perfect performance while relying on fragile latent representations that can be exploited by adaptive adversaries. This raises a fundamental question: does high detection performance imply representational robustness against compositional prompt obfuscation? In this paper, we have investigated this research question and have discovered that the answer is No . For this, we demonstrate that multi-operator obfuscated prompts (constructed through combinations of Unicode homoglyphs, zero-width characters, fragmented instructions and punctuation or emoji noise) can intrude into regions of the latent space occupied by clean prompts even when classification performance remains near-perfect. We refer to this phenomenon as latent embedding collapse , characterized by a partial manifold collapse in which obfuscated prompts intrude into regions of the latent space occupied by clean prompts without inducing misclassifications. To the best of our knowledge, this is the first research that is studying embedding-space or geometric analyses in LLM adversarial defense. This makes our focus on latent embedding collapse a novel perspective. The main contributions of this work are threefold: 1. Formalizing the concept of latent embedding collapse and connecting it to partial manifold collapse in transformer embedding spaces. 2. Introducing geometric metrics to quantify representational robustness, providing a complementary evaluation to conventional performance metrics. 3. Demonstrating empirically that increasing model depth or capacity does not resolve latent embedding collapse, indicating that the performance-robustness gap is an inherent phenomenon rather than an artifact of limited model size. The rest of the paper is structured as follows: Section II provides our literature review; Section III introduces the system model; Section IV details the methodology of the research; Section V presents evaluation results for our project; and Section VI provides concluding remarks and future work. II Related Work Prompt-injection attacks in LLMs have attracted increasing attention due to their potential to subtly alter model outputs [ 15 ] [ 2 ] [ 5 ] [ 10 ] . The work in [ 15 ] explored the security vulnerabilities in relation to prompt injection attacks for LLMs. Researchers in [ 2 ] introduced ChatInject, an attack that formatted malicious payloads to mimic native chat templates in LLMs. The work in [ 5 ] demonstrated that LLMs blur the lines between data and instructions during prompt injection attacks. Authors in [ 10 ] introduced Prompt Infection, where malicious prompts self-replicate across interconnected LLM agents. Prior defenses largely adopt a supervised classification approach, detecting malicious prompts using accuracy, precision, recall, or F1 score as evaluation metrics [ 11 ] [ 6 ] [ 3 ] [ 14 ] .In [ 11 ] , authors proposed UniGuardian, the first defense designed to detect prompt injection, backdoor attacks, and adversarial attacks in LLMs. The work in [ 6 ] investigated the underlying mechanisms of prompt injection attacks by analyzing the attention patterns within LLMs. Researchers in [ 3 ] proposed DefensiveToken, a defense with prompt injection robustness comparable to training-time alternatives. Authors in [ 14 ] introduced Jatmo, a defense mechanism for generating task-specific models resilient to prompt injection attacks. Through these works, we observe that performance metrics demonstrate strong detection performance. However, a limitation of these works is that they provide limited insight into the geometry of the learned latent space and the robustness of underlying representations [ 4 ] . Several surveys and critiques highlight the limitations of performance-centric evaluation in this domain [ 17 ] [ 7 ] . Even recently proposed defenses, including data filtering [ 18 ] and representation-based methods [ 12 ] , remain primarily evaluated on observable classification outcomes, rather than examining latent embedding structure. Observations on embedding-space or geometric analyses are rare in adversarial defense, making our focus on latent embedding collapse a novel perspective. This gap motivates our approach: rather than focusing solely on misclassification rates, we analyze the latent embedding collapse , a phenomenon in which obfuscated prompts partially intrude into regions of the latent space occupied by clean prompts without inducing misclassification. By emphasizing geometric properties of transformer representations, our method complements prior performance-based defenses and provides deeper insight into model robustness, revealing vulnerabilities that conventional metrics overlook. III System Model In this section, we are providing the mathematical background for latent embedding collapse. Let \phi(x)\in\mathbb{R}^{d} denote the final-layer embedding of a prompt x produced by a transformer encoder. For each prompt