Exploring Potential Prompt Injection Attacks in Federated Military LLMs and Their Mitigation
Youngjoon Lee, Taehyun Park, Yunho Lee, Jinu Gong, Joonhyuk Kang · 2025
Read Exploring Potential Prompt Injection Attacks in Federated Military LLMs and Their Mitigation by Youngjoon Lee, Taehyun Park, Yunho Lee, Jinu Gong, Joonhyuk Kang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-18416.txt.
Introduction
Youngjoon Lee and co-authors present a 2025 arXiv perspective paper on prompt injection attacks against military language models trained across allied nations. Federated learning lets partners build shared models while keeping their data at home. Crafted prompts, though, could pry out secrets, let a partner free-ride, disrupt the system, or spread false information.
The authors sort the threats into those four classes and propose a joint human-AI defense framework. Technical tools include red and blue team wargaming plus quality checks on shared model weights. Policy tools include joint rule writing and shared verification of security protocols. The paper aims to keep allied trust intact.
Catalog record
| CREATOR | Youngjoon Lee, Taehyun Park, Yunho Lee, Jinu Gong, Joonhyuk Kang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 29 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-18416
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Exploring Potential Prompt Injection Attacks in Federated Military LLMs and Their Mitigation Youngjoon Lee, Taehyun Park, Yunho Lee, Jinu Gong, Joonhyuk Kang Abstract Federated Learning (FL) is increasingly being adopted in military collaborations to develop Large Language Models (LLMs) while preserving data sovereignty. However, prompt injection attacks—malicious manipulations of input prompts—pose new threats that may undermine operational security, disrupt decision-making, and erode trust among allies. This perspective paper highlights four potential vulnerabilities in federated military LLMs: secret data leakage, free-rider exploitation, system disruption, and misinformation spread. To address these potential risks, we propose a human–AI collaborative framework that introduces both technical and policy countermeasures. On the technical side, our framework uses red/blue team wargaming and quality assurance to detect and mitigate adversarial behaviors of shared LLM weights. On the policy side, it promotes joint AI–human policy development and verification of security protocols. Our findings will guide future research and emphasize proactive strategies for emerging military contexts. Index Terms : federated learning, large language model, adversarial attack, military policy I Introduction The rise of LLMs and specialized AI hardware (e.g., Intel Gaudi) has accelerated the use of AI in many defense applications, enabling advanced analytics that were previously out of reach [ 1 ] . FL [ 2 ] provides a framework for allied nations to collaboratively train LLMs while maintaining data sovereignty [ 3 ] , [ 4 ] , as shown in Fig. 1 . By utilizing FL, each participant preserves sensitive information while reducing the risk of unauthorized access [ 5 ] . At the same time, FL facilitates collaborative model development, it also necessitates robust security measures to defend against evolving adversarial tactics [ 6 ] . Specifically, the most pressing concern is the threat of prompt injection attacks, in which adversaries cleverly manipulate or alter input prompts to extract secret data or disrupt mission-critical systems [ 7 ] , [ 8 ] . Such attacks can significantly weaken operational security, interrupt crucial decision-making processes, and undermine the trust [ 9 ] that underlies effective military cooperation among allied nations. Because FL involves a coalition of countries with numerous AI and analysis experts, addressing these threats necessitates both advanced technical solutions and policy frameworks that can accommodate varied defense requirements. Prompt injection attacks [ 10 ] , [ 11 ] , [ 12 ] may appear in four main forms in military context: secret data leakage , free-rider exploitation , system disruption , and misinformation spread , each posing distinct obstacles to the reliability and integrity of federated military LLMs. These threats often remain subtle and thus difficult to detect through traditional monitoring methods [ 13 ] , underscoring the need for specialized methods tailored to multinational contexts. Recognizing these evolving vulnerabilities is essential for creating robust countermeasures, as overlooking them could leave critical operations open to compromise in high-stakes military scenarios. Fig. 1: FL framework for military LLM training across allied nations. The process involves four key stages: (1) initial LLM synchronization, (2) local training with private data, (3) weight exchange, and (4) model aggregation. This iterative process continues until convergence, while mitigating adversarial risks. Blue clouds represent benign nation’s servers while red clouds indicate potentially compromised servers. In this paper we present an insight on how to address these risks by presenting a human–AI collaborative countermeasures from both technical and political perspectives. At first, we propose technical countermeasure process including red/blue team wargaming and continuous quality assurance, which help unveil hidden vulnerabilities within shared LLM infrastructures and reinforce overall system resilience. Then, we propose political countermeasure process, where military policy, domain experts, and AI experts work together to establish proper security policy for the nation. By applying either or both of these approaches, we aim to present a practical solution to counter present and emerging prompt injection threats in federated military environments. Therefore, this ensures sustained operational effectiveness and reinforces trust among coalition partners. The main contributions of this paper are as follows: • We introduce FL and its vulnerabilities to prompt injection attacks, highlighting the potential risks and security challenges in decentralized AI systems. • We present four potential major threat scenarios in military FL, including secret data leakage, free-rider exploitation, system disruption, and misinformation spread. • We propose human-AI collaborative countermeasures in perspective of technical and political way. • Additionally, we emphasize the need for standardized security frameworks and cooperative defense strategies. The remainder of this paper is organized as follows. In Section II , provides background knowledge of FL and prompt injection attacks. Section III introduces the potential challenges in federated military LLMs. Section IV discusses technical and political countermeasures for the potential threats. Finally, section V explores future direction and section VI concludes the paper. II Background In this section, we introduce the key background concepts for FL and prompt injection attacks. We specifically explore their impact on military collaborations among allied nations and the challenges they present in secure AI integration. II-A FL Topology in Military Alliances In data-sensitive fields such as healthcare and finance [ 14 ] , FL has emerged as an established approach for secure collaborative learning due to characteristics depicted in Fig. 2 . The military is also increasingly considering its adoption to enhance operational security and AI-driven decision-making. The FL architecture enables participating nations to contribute to model training while preserving their existing security infrastructure and operational autonomy [ 15 ] . This decentralized approach allows nations to leverage insights that may not be readily available within their own data sources while maintaining strict security controls. For example, the U.S. Department of Defense is actively collaborating with academia, industry, and allied nations to explore the adoption of FL for data management and responsible AI, ensuring alignment with security and operational requirements [ 16 ] . A major benefit of FL in military coalitions is its ability to enhance decision-making by integrating diverse operational experiences and data sources [ 17 ] . Different nations have unique battlefield environments, weapon systems, and threat intelligence, all of which can be incorporated into a shared model without compromising national security. This collective learning improves the adaptability of AI models to varied military scenarios, making them more effective in real-world military operations. Furthermore, FL optimizes communication efficiency by exchanging only model updates instead of raw data, significantly reducing bandwidth requirements. This streamlined communication enables real-time adaptability, allowing AI models to rapidly adjust to evolving threats and dynamic battlefield conditions. By incorporating advanced LLMs into a FL framework, allies can securely merge language data from various sources. This integration enhances cross-linguistic capabilities, strengthening communication protocols in joint operations. Each participant contributes domain-specific text corpora, enriching the shared model with contextual knowledge drawn from diverse military practices. This holistic approach bolsters language comprehension, enabling the LLM to accurately interpret mission-critical directives and intelligence reports. Ultimately, FL-driven LLM training preserves data sovereignty while facilitating faster, more reliable information synthesis for coalition decision-makers. Fig. 2: Illustration of four key advantages in FL: (1) Privacy-preservation capabilities, enabling reduced exposure risk and decentralized data management; (2) Joint learning framework, facilitating shared insights and real-time adaptation; (3) System scalability supporting cross-system expansion and seamless growth; and (4) Regulatory compliance features ensuring adherence to privacy laws and protection