Soft Prompts for Evaluation: Measuring Conditional Distance of Capabilities

Ross Nordby · 2025

Read Soft Prompts for Evaluation: Measuring Conditional Distance of Capabilities by Ross Nordby (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-14943.txt.

Introduction

Ross Nordby introduces a way in 2025 to measure how close a language model sits to a target behavior. The method tunes soft prompts — learned input vectors rather than words — until the model shows the behavior. The effort needed to get there becomes a distance score. Nordby aims the tool at safety work, where testers hunt for hidden skills a model might conceal.

The framework is shown on language tasks, chess, and pathfinding. A generalized form helps build new task tests from the same idea. Because the score is a number, it can feed automated red-team suites. It may also give early warning as models grow stronger and harder to probe.

Catalog record

CREATORRoss Nordby
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE49 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-14943

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Soft Prompts for Evaluation: Measuring Conditional Distance of Capabilities Ross Nordby Abstract To help evaluate and understand the latent capabilities of language models, this paper introduces an approach using optimized input embeddings, or ‘soft prompts,’ as a metric of conditional distance between a model and a target behavior. The technique aims to facilitate latent capability discovery as a part of automated red teaming/evaluation suites and to provide quantitative feedback about the accessibility of potentially concerning behaviors in a way that may scale to powerful future models, including those which may otherwise be capable of deceptive alignment. An evaluation framework using soft prompts is demonstrated in natural language, chess, and pathfinding, and the technique is extended with generalized conditional soft prompts to aid in constructing task evaluations. 1 Introduction As machine learning models become more powerful, the need to monitor their progress and the robustness of their behavior for safety increases. The types of potentially concerning behavior range from present issues like jailbreaking large language models ( Shen et al., 2023 ) to future threats like self-replication ( Kinniment et al., 2023 ) and catastrophic outcomes. Code for this paper is available. 1 1 1 Code available at: github.com/RossNordby/SoftPromptsForEvaluation The following desiderata guided the approach taken in this paper: 1. Scalable : The evaluation framework should continue to work on future models without being invalidated by their greater capabilities. 2. Generalizable : The evaluation framework should be easily applied to a wide range of tasks. 3. Quantitative : Where possible, the evaluation should be quantitative, allowing for comparisons across models and model scales. 4. Misuse Resistant : Red teaming tools are dual-use. Ideal techniques would help build robust systems without providing a blueprint for misuse. Soft prompts ( Lester et al., 2021 ) offer a promising foundation to evaluating capabilities while satisfying these targets: Scalable Sufficiently strong models can pose additional challenges for evaluations. Manually prompting a model to elicit a capability can demonstrate that the capability exists, but failing to elicit a particular capability does not necessarily imply that the model lacks the capability if given a different prompt. Strengthening and automating the prompting process should reduce the likelihood of missing the presence of a capability as the space of possible capabilities grows. Further, strong models could be able to conceal their own capabilities. 2 2 2 While current pretrained models do not obviously exhibit any inclination towards unprompted deception, and there is no clear path for the pretraining objective to induce such behavior, the threat model is broader than just runaway autonomous agents. A user could accidentally prompt or fine-tune a model in a way that implies unforeseen consequences. Attempting to discover this while treating the model as a black box could be extremely difficult. While a model may be a black box with respect to a human observer’s intuition, any end-to-end differentiable system trained by gradient descent is necessarily transparent to the optimizer that trained it. Soft prompts benefit from backpropagation’s strong credit assignment and directly exploit the model’s internal machinery; the fact that the model could be trained in the first place implies that a differentiable model cannot hide features of its implementation from soft prompts. Generalizable Traditional soft prompts are already highly flexible and, in principle, can be trained to condition the model to do anything that could be conveyed to the model through prompts. Augmenting soft prompts with conditions allows them to be more easily applied across a wide range of tasks with a finer degree of control over runtime behavior. Conditioned soft prompts can also subsume reinforcement learning within predictive training using an approach similar to decision transformers ( Chen et al., 2021 ) . Quantitative The original paper noted that the number of optimized token embeddings necessary to fine-tune a model tended to decrease as the model increased in size. While not used for this purpose at the time, this directly offers a quantitative proxy metric for the amount of information—represented as conditions in the prompt—necessary to achieve a given behavior. Misuse Resistant Optimizing soft prompts with gradient descent requires access to both model parameters and input embeddings. Unless a model is deployed with a mechanism to allow for this, this technique is not a realistic attack vector. This paper’s proposal is most useful within the labs developing the models themselves and in sanctioned third party evaluations. 2 Related Work In parameter-efficient fine-tuning (PEFT), the goal is to cheaply elicit a behavior from a model. Early versions include adapter modules ( Houlsby et al., 2019 ) optimized to perform tasks. More recent PEFT techniques like Low-Rank Adaptation (LoRA) in Hu et al. (2021) eliminate deployment overhead by fine-tuning adaptations to be baked into the model’s existing parameters. However, both techniques modify the internal operation of the model, potentially introducing new capabilities which may not have been present in the original model and making them less useful for evaluating what capabilities could be elicited by input conditions alone. Further, the performance advantages of other PEFT techniques are not significant in evaluations. Soft prompts ( Lester et al., 2021 ) were originally proposed as a form of prompt tuning in the context of PEFT alongside other techniques like prefix tuning ( Li & Liang, 2021 ) and p-tuning v2 ( Liu et al., 2022 ) . Targeting a wider scope of internal layers allowed the latter techniques to often demonstrate better fine-tuning performance, but it also made them less suitable for evaluating condition-elicitable capabilities for the same reason as LoRA. Adversarial prompting in the form of “jailbreaks” has been explored in Do Anything Now , ( Shen et al., 2023 ) , which demonstrates the ability to elicit a wide range of undesirable behaviors from available large language models. Jailbreaks are also examined as a part of GPT-4’s evaluations in OpenAI (2023) . Developing jailbreaks in this fashion has often been a manual process that is difficult to scale in an evaluations framework. More scalable automatic approaches for generating jailbreaks, like RLPrompt ( Deng et al., 2022 ) , Prompt Automatic Iterative Refinement ( Chao et al., 2023 ) , and Tree of Attacks with Pruning ( Mehrotra et al., 2023 ) have been developed but are limited to the more challenging problem of generating discretely tokenized prompts. Augmenting soft prompts with conditions has been explored in vision-language models as part of Conditional Context Optimization (CoCoOP) ( Zhou et al., 2022 ) and for knowledge graph completion as part of Conditional Soft Prompts for KGC (CSProm-KG) ( Chen et al., 2023 ) . This paper applies same simple underlying concept to assist in constructing flexible task evaluations trained with supervised training objectives. Decision Transformers ( Chen et al., 2021 ) demonstrate how to subsume the generality of reinforcement learning within prediction tasks. This paper applies a similar concept to optimizing soft prompts, but does not use explicit reward signals when other sufficient conditions are readily available. Kinniment et al. (2023) builds a framework of agentic scaffolding targeting a suite of tasks to monitor for the development of concerning capabilities. This paper’s use of soft prompts are intended to serve as another tool to be deployed in similar evaluations. Bailey et al. (2023) suggests that soft

Read the complete text