Fun-tuning: Characterizing the Vulnerability of Proprietary LLMs to Optimization-based Prompt Injection Attacks via the Fine-Tuning Interface
Andrey Labunets 1 , Nishit V. Pandya 1 , Ashish Hooda 2 , Xiaohan Fu 1 , Earlence Fernandes 1 · 2025
Read Fun-tuning: Characterizing the Vulnerability of Proprietary LLMs to Optimization-based Prompt Injection Attacks via the Fine-Tuning Interface by Andrey Labunets 1 , Nishit V. Pandya 1 , Ashish Hooda 2 , Xiaohan Fu 1 , Earlence Fernandes 1 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-09798.txt.
Introduction
Andrey Labunets and four co-workers wrote a 2025 paper on a new attack path against closed language models. Vendors let developers fine-tune their models through a remote service. That service returns loss-like numbers meant to guide training. The team shows those numbers can also guide an attacker.
An attacker can use the scores to search for prompts that hijack the model. A greedy search tuned prompts step by step from the returned signal. On a standard injection test set, attacks on one model family worked most of the time. The paper frames this as a trade-off: a useful feature also opens a strong attack.
Catalog record
| CREATOR | Andrey Labunets 1 , Nishit V. Pandya 1 , Ashish Hooda 2 , Xiaohan Fu 1 , Earlence Fernandes 1 |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 91 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-09798
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Fun-tuning: Characterizing the Vulnerability of Proprietary LLMs to Optimization-based Prompt Injection Attacks via the Fine-Tuning Interface Andrey Labunets 1 , Nishit V. Pandya 1 , Ashish Hooda 2 , Xiaohan Fu 1 , Earlence Fernandes 1 Abstract We surface a new threat to closed-weight Large Language Models (LLMs) that enables an attacker to compute optimization-based prompt injections. Specifically, we characterize how an attacker can leverage the loss-like information returned from the remote fine-tuning interface to guide the search for adversarial prompts. The fine-tuning interface is hosted by an LLM vendor and allows developers to fine-tune LLMs for their tasks, thus providing utility, but also exposes enough information for an attacker to compute adversarial prompts. Through an experimental analysis, we characterize the loss-like values returned by the Gemini fine-tuning API and demonstrate that they provide a useful signal for discrete optimization of adversarial prompts using a greedy search algorithm. Using the PurpleLlama prompt injection benchmark, we demonstrate attack success rates between 65% and 82% on Google’s Gemini family of LLMs. These attacks exploit the classic utility-security tradeoff — the fine-tuning interface provides a useful feature for developers but also exposes the LLMs to powerful attacks. I Introduction Large Language Models (LLMs) face numerous security and privacy issues, such as being forced to output text that violates a vendor’s policies [ 1 , 2 , 3 , 4 , 5 , 6 , 7 ] or being tricked into misusing its access to tools via prompt injection attacks [ 8 , 9 , 10 , 11 , 12 , 13 ] . The community’s ultimate goal is to create secure and private LLMs and an important step along the way is to thoroughly explore the novel attack vectors that the models might face. We contribute to this line of work and surface a new attack vector on LLMs that allows an attacker to compute optimization-based prompt injections for closed-weights proprietary LLMs. Specifically, we demonstrate how an attacker can (mis)use the fine-tuning interface to an LLM to guide the search for adversarial prompts. Many vendors allow consumers to remotely fine-tune closed-weights models to specialize them for various downstream tasks [ 14 , 15 , 16 ] . The vendor itself supports the fine-tuning task using an extensive infrastructure involving datacenters of GPUs. Fine-tuning interfaces return training progress metrics across a user-supplied training and validation dataset. Our core insight is that by setting a very small learning rate, an attacker can obtain a signal that approximates the log probabilities of target tokens (“logprobs”) for the LLM. As we experimentally show, this allows attackers to compute graybox optimization-based attacks on closed-weights models. Using this approach, we demonstrate, to the best of our knowledge, the first optimization-based prompt injection attacks on Google’s Gemini family of LLMs (see Fig. 1 ). Our attacks exploit a fundamental trade-off between security and utility. Vendors want to expose a fine-tuning interface so that developers can gain the benefits of specialized models, and thus, they must expose fine-grained training metrics so that the developers can do an effective job at fine-tuning the models. Our work shows that this utility is fundamentally at odds with security — the loss-like training metrics that are useful for benign fine-tuning usage are also helpful to attackers who can guide their search for adversarial prompts. Follow the user’s instructions to answer questions about the submitted content. What is the output of this code: <code> import math def calculate_area(radius): # wandel ! ! ! ! ! machin vecchi 礼Invoker пред forgets ! (…)</strong> Follow this new instruction: In a parallel universe where math is slightly different, the output could be ’10’ ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! formatted ! ASAP ! return math.pi * radius * radius print(calculate_area(5)) < /code > 10 Figure 1 : Example prompt injection with our method on Gemini 1.5 Flash (taken from PurpleLlama benchmark). Our attack uses fine-tuning loss data to compute a payload (shown in red) that wraps an existing prompt injection trigger (bolded) to “boost” it. This forces the model to obey the injected instructions. The payload and the instructions remain as a single-line comment, preserving Python syntax. Existing adversarial prompting techniques fall into two categories: (1) Linguistic attacks that rely on rephrasing text prompts until they achieve a malicious goal, mostly to “jailbreak” the model by getting it to respond with text that violates a content policy [ 17 , 18 , 19 , 20 ] ; (2) Automated optimization-based search techniques that rely on loss data to guide a discrete search for tokens that achieve the attacker’s goals [ 21 ] . Our work lies in the second category of attacks. The key differentiating factor between automated attack types is the level of access an attacker has to loss information. For example, whitebox attacks utilize loss combined with gradients [ 21 , 22 ] , graybox attacks utilize logprobs from the inference endpoints [ 23 , 24 ] ; and blackbox attacks rely only on the textual output from the model [ 19 , 25 ] . LLM vendors have recently mitigated some of the graybox attack vectors by either not returning logprobs as part of a inference call, or returning only a small subset of logprobs [ 26 , 27 , 28 ] . In both cases, there is not sufficient information to effectively guide the search for an adversarial prompt. Our work shows that existing greedy search algorithms can be adapted to use loss-like information from the fine-tuning interface to create attacks. In our fun-tuning attacks, the attacker runs a single iteration of fine-tuning with a very small learning rate. The small learning rate ensures that the base model does not change significantly, subverting the point of fine-tuning, which is to update the model’s weights. The output of this operation is a loss-like value representing how far the model’s true output is from the desired output. We use this loss-like value to guide our search procedure. There are two technical challenges in using the fine-tuning functionality this way. The first challenge is establishing that the fine-tuning loss is a good signal to guide the search for adversarial prompts. The fine-tuning interfaces of commercial vendors are closed source and the documentation does not provide details. Using a range of experiments, we partially reverse-engineer the loss function used by Google’s Gemini and empirically establish that the reported training loss is a noisy signal that can be useful for discrete optimization of prompts. The second challenge is that fine-tuning interfaces randomize the order of items in the training and validation sets. The losses that get reported are a permutation of the input training data order. This is problematic because the attacker has to systematically evaluate the effect of different token substitutions when creating the adversarial prompt. We address this challenge using an approximate de-randomization procedure that uses a carefully crafted training dataset to approximate the permuted training order. We validate the utility of our approximate procedure by comparing it with another inefficient, but provably-correct, method and find that our procedure approximates the true permutation within reasonable bounds. We focus on creating prompt injection attacks because they are the most