Do Prompts Reshape Representations? An Empirical Study of Prompting Effects on Embeddings

Cesar Gonzalez-Gutierrez, Dirk Hovy · 2025

Read Do Prompts Reshape Representations? An Empirical Study of Prompting Effects on Embeddings by Cesar Gonzalez-Gutierrez, Dirk Hovy (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-19694.txt.

Introduction

Cesar Gonzalez-Gutierrez and Dirk Hovy test a widespread assumption about prompting language models: that instructions relevant to a task improve the model's internal representations. Prompting is the standard way to use these models without extra training, yet why it works is poorly understood. The authors run probing experiments on prompt embeddings across combinations of templates for zero-shot classification.

Prompting did change representation quality, but the changes did not reliably track how relevant the prompt was to the task. In other words, a well-matched prompt does not guarantee better internal representations, contrary to intuition. The paper then examines other factors that might explain the pattern, offering a caution for anyone designing prompts or interpreting model behavior.

Catalog record

CREATORCesar Gonzalez-Gutierrez, Dirk Hovy
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-19694

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Do Prompts Reshape Representations? An Empirical Study of Prompting Effects on Embeddings Cesar Gonzalez-Gutierrez, Dirk Hovy Abstract Prompting is a common approach for leveraging LMs in zero-shot settings. However, the underlying mechanisms that enable LMs to perform diverse tasks without task-specific supervision remain poorly understood. Studying the relationship between prompting and the quality of internal representations can shed light on how pre-trained embeddings may support in-context task solving. In this empirical study, we conduct a series of probing experiments on prompt embeddings, analyzing various combinations of prompt templates for zero-shot classification. Our findings show that while prompting affects the quality of representations, these changes do not consistently correlate with the relevance of the prompts to the target task. This result challenges the assumption that more relevant prompts necessarily lead to better representations. We further analyze potential factors that may contribute to this unexpected behavior. 1 Introduction In recent years, language model (LM) prompting has emerged as the dominant model application paradigm in NLP, with LMs like GPT-3 ( Brown et al. 2020 ) , LLaMA ( Touvron et al. 2023 ) , PaLM ( Chowdhery et al. 2024 ) , among many others. In this framework, the model architecture remains unchanged, and the downstream task is verbalized transforming tasks into text prompts which are inputted to the model to elicit the desired response. Prompting leverages LMs performing language modeling tasks through conditional text generation or clause-style completion. A key advantage of this approach is its generality. A single model architecture can ideally handle various tasks without further supervised training, eliminating the need for task-specific fine-tuning. This method also offers great flexibility, allowing tasks to be switched simply by changing the prompt. This could in principle enable the model to address tasks it was not explicitly trained on, a phenomenon often referred to as model’s emergent abilities ( Brown et al. 2020 ; Schaeffer et al. 2023 ) . Although LMs have high computational demands, prompting also offers advantages in low-annotation scenarios, by leveraging their encoded knowledge to address under-resourced tasks ( Mosbach et al. 2023 ) . The model’s ability to solve a task processing a prompt containing the target sample is known as in-context learning (ICL). Tasks can be approached including solved examples in the prompt ( few-shot learning), or by directly instructing the model without explicit examples ( zero-shot learning). Contrary to supervised training, ICL does not need to update the model weights, leveraging LM’s pre-training without further adaptation, and achieving performance solely by contextualizing the sample in a prompt. For a prompt to be effective for a task, the language used in the context of such task must have been encountered during pre-training ( Gonen et al. 2024 ) . This exposure would allow the model to learn patterns and recognize language structures relevant to the particular task. Conversely, if a LM has not seen patters analogous to those relevant for a given task, this should result in lower performance when processing the prompt. As an example, similar to the argument made by Radford et al. 2019 on how ICL might work in practice, consider the prompt “Is the following review positive or negative?: The movie was great!”. In this case, the review itself is the sample to evaluate. The question, explicitly states the task objective (sentiment analysis) and refers to the review for evaluation. The language patterns that appear while expressing or commenting on others opinions (e.g., “Did you enjoy the movie? I think it was awesome.”), although formulated less explicitly, have a similar structure. These patters are common in language and their structure can be learned by LMs during pre-training. This illustrates how a LM can build useful representations from seemingly unrelated contexts for ICL, provided that the corpus is sufficiently large and general. If LMs are capable of performing a task without modifying the pre-trained weights, then all that occurs during ICL are representational changes via processing of the prompt tokens. At the representation level, ICL can be understood as contextualization of samples via prompting. This raises the question: What happens to the sample representation when we contextualize it in a prompt? Can we measure any improvement in the representation when it’s contextualized in a prompt that makes it more suitable for a particular task? If ICL is driven by language patterns shared between a task and the data seen during pre-training, then prompts that are relevant to the task should improve the quality of the resulting representations. In contrast, irrelevant prompts should produce no improvement or even worsen quality. Based on this assumption, we hypothesize that changes in representational quality induced by prompting (relative to a specific task) will align with the relevance of the prompt to the task. Moreover, such representational changes should be measurable using representation analysis tools. Additionally, if ICL capabilities in transformer models result solely from language modeling tasks, then these capabilities should not depend on the specific type of pre-training (as long as the model can effectively model the input distribution). In our experiments, we test quality changes using both masked language models (MLM) and autoregressive models. This property should also be present in contextual models of any size, and its effects detectable in smaller models. Therefore, if the effects are significant, we should be able to study them in models with only a few hundred million parameters, small enough to run on mid-sized hardware. In this empirical study, we employ probing techniques ( Ettinger et al. 2016 ; Adi et al. 2017 ) to analyze the task-relevant information encoded within sentence representations. Specifically, we compute task-specific probe performance on prompt embeddings, computed from the target samples along task-specific instructions (zero-shot prompting). We then compare probe performance between different prompt templates, some being relevant to the probe task while others unrelated. In summary, this work studies the effect of prompting at the embedding level. We hypothesize that prompting contextualizes sentence representations, leading to measurable changes in their embeddings, as reflected in probing task performance. We expect probe performance to improve when prompts are relevant to the task, and decrease when they are irrelevant. We anticipate that similar effects may also be observed using alternative metrics of task-specific representation quality. The main contributions of this paper are: • We conduct an empirical analysis comparing the quality of sentence representations across several tasks, using prompt templates that are either relevant or irrelevant to the target task. • We show that prompting alters sample’s sentence representations through contextualization. • We find that changes in prompt embedding quality due to prompt relevance do not follow a consistent or predictable pattern across tasks and models. 2 Experimental Setup 2.1 Learning Setting Prompting In a prompting pipeline ( Liu et al. 2023 ) , the input text x is first modified using a prompting function p . This function applies a template or prefix to produce a prompt x^{\prime}=p(x) . The prompt may include several examples of the task (few-shot learning), or none (zero-shot learning), along with task instructions. This prompt is then used as input to the model for language modeling, such as masked token prediction or next token prediction. The highest-scoring answer is selected from the model output, according to some LM

Read the complete text