Auto-Prompt Generation is Not Robust: Prompt Optimization Driven by Pseudo Gradient
Zeru Shi, Zhenting Wang, Yongye Su, Weidi Luo, Hang Gao, Fan Yang, Ruixiang Tang, Yongfeng Zhang · 2024
Read Auto-Prompt Generation is Not Robust: Prompt Optimization Driven by Pseudo Gradient by Zeru Shi, Zhenting Wang, Yongye Su, Weidi Luo, Hang Gao, Fan Yang, Ruixiang Tang, Yongfeng Zhang (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-18196.txt.
Introduction
Zeru Shi and colleagues show that automatic prompt writing is fragile, then offer a fix, in work from 2024. Automatic methods now write prompts for large language models, but their strength under messy input was poorly known. Small prompt changes can swing model output by a large amount.
A new benchmark, called PertBench, tests prompt methods under many input changes. Tests found clear weakness in current strategies under even minor changes. The team's fix, called PGO, uses change types as guide signals in place of true gradients. Models then write prompts that hold up under noise. Wide tests across tasks and models showed steadier performance than prior methods when inputs were disturbed.
Catalog record
| CREATOR | Zeru Shi, Zhenting Wang, Yongye Su, Weidi Luo, Hang Gao, Fan Yang, Ruixiang Tang, Yongfeng Zhang |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 71 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-18196
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Auto-Prompt Generation is Not Robust: Prompt Optimization Driven by Pseudo Gradient Zeru Shi, Zhenting Wang, Yongye Su, Weidi Luo, Hang Gao, Fan Yang, Ruixiang Tang, Yongfeng Zhang Abstract While automatic prompt generation methods have recently received significant attention, their robustness remains poorly understood. In this paper, we introduce PertBench , a comprehensive benchmark dataset that includes a wide range of input perturbations, designed to systematically evaluate the robustness of current auto-prompting techniques. Our analysis reveals substantial vulnerabilities in existing prompt generation strategies, where even minor modifications to the prompt can lead to significant differences in model output. To address this issue, we propose PGO , a gradient-free prompt generation framework that leverages perturbation types as pseudo-gradient signals to guide LLMs in producing more robust prompts. In contrast to existing methods that assess prompt quality only on clean, well-structured inputs, our approach explicitly emphasizes robustness under noisy and perturbed conditions. Extensive experiments across diverse tasks and multiple LLMs show PGO consistently outperforms previous methods in maintaining performance under input perturbations. 1 Introduction Figure 1: An example of adding a perturbation to the input and its result in classification task, where the top one indicates that the prompt gets the correct output under normal input, the middle one indicates that the prompt gets the wrong result under perturbed input, the bottom indicates the PGO gets correct result under perturbed input. Large Language Models (LLMs) have demonstrated impressive capabilities across a wide spectrum of tasks. Abburi et al. (2023) ; Ge et al. (2024) ; Laban et al. (2023) . Given their broad applicability, a growing body of research has focused on strategies to more effectively elicit and amplify these capabilities. Among them, prompt engineering has emerged as a central technique, with recent advances increasingly relying on LLMs themselves to automate the prompt optimization process. Representative approaches include instruction fine-tuning Ouyang et al. (2022) ; Ziegler et al. (2019) , Chain-of-Thought (CoT) prompting Shum et al. (2023) ; Zhang et al. (2022) , and inference-guided prompt generation Wang et al. (2023b) ; Liu et al. (2024) . Despite their success on clean, well-structured inputs, these methods share a key limitation: they assume idealized input conditions, and thus fail to account for the noisy, imperfect inputs commonly encountered in real-world scenarios, such as typographical errors, ambiguous phrasing, or factual imprecision. These input perturbations can significantly degrade model performance, revealing a critical robustness gap in current prompt optimization techniques. As shown in Figure. 1 , even subtle input corruptions (e.g., a minor typo) can mislead an LLMs and trigger incorrect predictions in tasks like classification. Figure 2: Comparison of the execution results of all different methods on PertBench for the three tasks. Perturbation level Name. Explanation Type Character C1 Change words to have typos P1 C2 Change Letters. P1 C3 Add extraneous characters P1 Word W1 Change word to synonyms. P2 W2 Delete meaningless words. P2 W3 Add neutral words. P2 Sentence S1 Add meaningless handle P1 S2 Paraphrase the sentence. P2 S3 Change the syntactic structure. P2 Table 1: Explanation of different kinds of perturbation. In this work, we introduce PertBench , a comprehensive benchmark designed to evaluate the robustness of prompt-based methods under diverse input perturbations. Our benchmark spans three representative natural language processing tasks, text simplification, summarization, and classification—capturing a wide spectrum of practical use cases. In contrast to prior datasets such as NoiseBench Merdjanovska et al. (2024) ; Moradi and Samwald (2021) ; Dong et al. (2023) , which primarily focus on character-level noise within a single task or short form content (e.g., dialogues), PertBench provides broader coverage across multiple granularity levels, including character-level, word-level, and sentence-level perturbations. Moreover, we extend the perturbation setting to long form text and ensure a uniform and comprehensive perturbation strategy across all tasks. The specific categories of perturbations are summarized in Table 1 . This design enables a more systematic and rigorous evaluation of prompt robustness across tasks and text lengths. Our results demonstrate that existing prompt generation methods experience substantial performance degradation under perturbed inputs, exposing their vulnerability and limited generalization. To mitigate this issue, we propose PGO (Pseudo Gradient Optimization), a gradient-free framework for robust discrete prompt optimization. Unlike conventional approaches that rely on model gradients to update prompts, PGO leverages perturbation information as guidance to guide prompt refinement. We categorize perturbations into two types P1 and P2 depending on whether they cause significant semantic shifts. Correspondingly, PGO introduces two complementary optimization strategies tailored to these categories and iteratively refines prompts via feedback-driven updates. Through this design, PGO effectively enhances prompt robustness against a broad range of input disturbances. Our main contributions are summarized as follows: • We present PertBench , a comprehensive benchmark for evaluating prompt robustness under diverse input perturbations. It includes nine perturbation types across three core task categories for systematic robustness assessment. • We propose PGO , a pseudo gradient–based prompt optimization framework that enhances robustness across perturbations without requiring model internals or true gradients. • Extensive experiments show that existing methods degrade notably on PertBench, while PGO consistently improves robustness across tasks and perturbation settings. 2 Benchmark Construction 2.1 Details of Benchmark 2.1.1 Basic Datasets In the classification tasks , we focus on six datasets to apply and test perturbations. These include sentiment classification datasets: SST-2 Socher et al. (2013) , CR Hu and Liu (2004) , SST-5 Socher et al. (2013) , and MR PaNgB (2005) , as well as topic classification datasets: AG’s News Zhang et al. (2015) and TREC Voorhees and Tice (2000) . In language generation tasks, we utilize the Asset Alva-Manchego et al. (2020) for text simplification , which includes multiple benchmarks for reference translations. For the text summarization task , we use the XSum Narayan et al. (2018) , which consists of concise summaries generated from longer texts. More details of PertBench in the Appendix A . 2.1.2 Perturbation Types We adopt a widely used approach following Xu et al. (2023) , which form the basis for the perturbations in our experiments (See Table. 1 ). These disturbances are categorized into two groups: P1 and P2 . P1 introduce typographical errors and non-sensical strings into the text, without significantly altering the underlying semantic structure of the sentence. P2 alter the semantic structure of sentences by modifying their syntactic composition, while maintaining their core meaning. These modifications can result in a bias in how LLMs interpret the input, potentially affecting their understanding. We compare the similarity between the perturbed data set and the original data set, and the experiment shows that we complete the perturbation of the data while maintaining the original similarity. The results in the Appendix A . 2.2 Benchmark Construction When constructing the dataset, we apply nine different perturbations to each type of task to construct separate perturbed datasets. To be more specific, we introduce various types of perturbation-specific guide words as a fixed gradient g , and use it to guide the perturbations on x . y is the standard output. The performance of the original prompt on the perturbed inputs is evaluated using a loss function \mathcal{L}_{adv} . The score is minimized when perturbed. After generating the perturbed sample prompt p , we compute either its Levenshtein distance