Spurious Prompts: Can Irrelevant Prompts Steer Large Language Models?

Paweł Batorski, Abtin Pourhadi, Jerzy Sarosiek, Przemysław Spurek, Paul Swoboda · 2026

Read Spurious Prompts: Can Irrelevant Prompts Steer Large Language Models? by Paweł Batorski, Abtin Pourhadi, Jerzy Sarosiek, Przemysław Spurek, Paul Swoboda (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-29678.txt.

Introduction

Pawel Batorski and four colleagues ask whether prompts with no link to a task can still steer a language model. They coin the term spurious prompts for such unrelated nudges and build a simple black-box hunt to find them. Across reasoning and question-answering tests, models from under one billion to twenty-seven billion weights, in three families, often gained from these prompts, at times matching tuned baselines. The same nudges could also push models toward quirks, like always picking the first choice or returning an even number unasked.

Catalog record

CREATORPaweł Batorski, Abtin Pourhadi, Jerzy Sarosiek, Przemysław Spurek, Paul Swoboda
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE109781
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.29678

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Spurious Prompts: Can Irrelevant Prompts Steer Large Language Models? Paweł Batorski, Abtin Pourhadi, Jerzy Sarosiek, Przemysław Spurek, Paul Swoboda Abstract Large language models are highly sensitive to prompts, but this sensitivity is usually studied through task-relevant instructions, demonstrations, or reasoning cues. In this paper, we study a different form of prompt sensitivity: whether prompts that are semantically unrelated to the task can nevertheless steer model behavior. We call them spurious prompts and show their surprising efficacy. We also propose a simple black-box search procedure for discovering them. Across reasoning and question-answering benchmarks, using models ranging from 0.8B to 27B parameters and spanning three model families, we show that spurious prompts can improve performance, often matching or outperforming standard prompting baselines and task-aware prompt optimization. We further show that they can steer models toward unintended behaviors, such as repeatedly selecting the first answer option, producing incorrect answers, returning an even, prime or small number without explicitly instructing the model to do so. These findings reveal a new kind of prompt sensitivity: LLMs can be systematically steered by prompts that are unrelated to the task they are asked to solve. Our code is available at https://github.com/Batorskq/spurious. Figure 1: Right: High-level illustration of Spurious Prompting. We search for prompts that are unrelated to the target task but can nevertheless solve it. Left: Average accuracy over seven benchmarks comparing Spurious Prompting with PromptWizard and zero-shot chain-of-thought prompting across four models ranging from 0.8B to 27B parameters. Spurious Prompting achieves performance comparable to, and in some cases better than, the two baselines despite using prompts that are completely unrelated to the task being solved. 1 Introduction Large language models (LLMs) are designed to be instruction-following engines, leading to an entire discipline dedicated to crafting a perfect task description. While it is well documented that LLMs are highly sensitive to superficial changes in prompt wording, formatting, and demonstration order ( 34 ; 16 ; 19 ; 24 ; 21 ; 37 ; 6 ) , a core, unquestioned assumption underscores almost all of this research: that prompt variations must remain task-preserving . The prevailing paradigm dictates that to improve a model’s performance, we must ask it to solve that exact task more clearly, methodically, or with better context. In this paper, we challenge this foundational assumption and expose a counterintuitive form of prompt sensitivity. Instead of refining task descriptions, we show that prompts whose surface content is deliberately unrelated to the target task can still drive model behavior and improve performance. We introduce these as spurious prompts : natural-language instructions that avoid task-relevant vocabulary, domain cues, and explicit solution strategies, while still requiring the model to answer the user’s query directly. Importantly, their influence is not limited to accuracy improvements. We also show that spurious prompts can steer models toward unintended behaviors, such as repeatedly selecting the first answer option, choosing incorrect answers, or, for mathematical questions, producing even or prime-number outputs, without explicitly instructing the model to do so. We further develop a simple black-box search procedure for discovering spurious prompts requiring no hidden states or logits. We evaluate spurious prompts across mathematical reasoning, narrative reasoning, and knowledge-intensive question-answering benchmarks, comparing them with standard task-agnostic prompting strategies and PromptWizard ( 1 ) , a task-aware prompt optimizer. Across multiple model–benchmark pairs, spurious prompts match or outperform these baselines, even though they are deliberately stripped of task-specific vocabulary, domain cues, and explicit reasoning instructions. This shows that prompt effectiveness need not arise only from better task descriptions: superficially unrelated instructions can also induce behaviors that substantially alter model performance. Further we demonstrate that spurious prompts are semantically indistinguishable from random, unrelated text, proving they are not merely disguised task descriptions. Their effectiveness is also not a byproduct of length, as they are often significantly shorter than task-aware optimized prompts. Furthermore, transfer experiments show these prompts exploit highly specific, idiosyncratic interactions between the model, the benchmark, and the prompt’s latent control structure. Together, these results suggest a new view on prompt sensitivity of LLMs: model behavior and reasoning capabilities can be powerfully steered by latent features completely detached from the intended task. This also poses new question of how instruction following really works for LLMs. To summarize, our contributions are as follows: Spurious prompts: We introduce the notion of spurious prompts, i.e. prompts that are deliberately unrelated to the target task on the surface, yet still heavily dictate downstream model performance. Constrained search procedure: We propose a novel, simple procedure that searches over spurious system prompts while explicitly aggressively excluding task descriptions, domain vocabulary, and common reasoning cues. Our search operates in a fully black-box setting, requiring only model outputs and no access to gradients, hidden states, model weights, or other internal model information. Empirical and diagnostic analysis: We empirically demonstrate that spurious prompts can substantially boost accuracy across benchmarks and models. Our experiments further suggest that they can steer model predictions toward specific behaviors, such as selecting the first option, choosing incorrect answers, or producing even or prime-number outputs, without explicitly instructing the model to do so. Spurious prompts can be found across different LLM families and sizes. 2 Related Work Prompt sensitivity. A complementary line of work shows that LLM performance can vary substantially under prompt changes that are usually intended to preserve the task. Early studies found few-shot performance to be sensitive to prompt format, demonstration choice, and demonstration order, motivating calibration and ordering methods ( 34 ; 16 ; 19 ; 31 ; 9 ; 3 ; 2 ) . Other work studies sensitivity to spurious formatting features ( 24 ) , answer-option order ( 21 ) , prompt-level sensitivity metrics ( 37 ; 6 ; 15 ) , and worst-case prompt performance ( 4 ) . Recent work also argues that some reported sensitivity may be amplified by evaluation artifacts such as rigid answer matching or log-likelihood scoring ( 10 ) . Closest to our motivation, Webson and Pavlick ( 28 ) show that models can perform well even with irrelevant or misleading prompts. Our work differs by actively searching for high-performing spurious prompts under explicit lexical constraints, turning this diagnostic observation into a controlled prompt-search setting. Adversarial prompts. Adversarial prompting is closely related, but this literature mainly studies jailbreaking: prompts designed to bypass safety mechanisms or elicit harmful behavior ( 20 ; 14 ; 17 ; 30 ) . Other work studies jailbreaks under multilingual or cipher-based transformations ( 8 ; 33 ) , and benchmarks such attacks systematically ( 5 ) . Our work differs in both goal and setting: we do not aim to bypass safety policies, but instead study whether task-irrelevant prompts can spuriously steer ordinary benchmark behavior. Also our prompts are natural language and semantically meaningful ones, unlike many adversarial prompts that produce cryptic text. 3 Spurious Search Figure 2: Top: Overview of our fully black-box search procedure. An LLM generator first proposes candidate prompts and is explicitly instructed to make them unrelated to the target task. These candidates are then passed to a prompt validator, which filters out prompts that contain task-relevant content. The remaining prompts are evaluated on a subset of the training data, after which the top- K prompts are mutated and the best

Read the complete text