Automatic Prompt Generation via Adaptive Selection of Prompting Techniques
Yohei Ikenoue, Hitomi Tashiro, Shigeru Kuroyanagi · 2025
Read Automatic Prompt Generation via Adaptive Selection of Prompting Techniques by Yohei Ikenoue, Hitomi Tashiro, Shigeru Kuroyanagi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-18162.txt.
Introduction
Yohei Ikenoue, Hitomi Tashiro, and Shigeru Kuroyanagi automate the craft of writing prompts for language models. Good prompting needs expert knowledge of many techniques and of the task at hand. Their method instead starts from a plain task description given by the user, with no template library in sight.
A knowledge base links clusters of similar tasks to the techniques that suit them. A new task is matched to its cluster, and a prompt is built from the linked techniques. Tests on twenty-three hard benchmark tasks judge the output. Casual users get a route to strong prompts without studying prompt engineering.
Catalog record
| CREATOR | Yohei Ikenoue, Hitomi Tashiro, Shigeru Kuroyanagi |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 45 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-18162
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Automatic Prompt Generation via Adaptive Selection of Prompting Techniques Yohei Ikenoue, Hitomi Tashiro, Shigeru Kuroyanagi Abstract Prompt engineering is crucial for achieving reliable and effective outputs from large language models (LLMs), but its design requires specialized knowledge of prompting techniques and a deep understanding of target tasks. To address this challenge, we propose a novel method that adaptively selects task-appropriate prompting techniques based on users’ abstract task descriptions and automatically generates high-quality prompts without relying on pre-existing templates or frameworks. The proposed method constructs a knowledge base that associates task clusters, characterized by semantic similarity across diverse tasks, with their corresponding prompting techniques. When users input task descriptions, the system assigns them to the most relevant task cluster and dynamically generates prompts by integrating techniques drawn from the knowledge base. An experimental evaluation of the proposed method on 23 tasks from BIG-Bench Extra Hard (BBEH) demonstrates superior performance compared with standard prompts and existing automatic prompt-generation tools, as measured by both arithmetic and harmonic mean scores. This research establishes a foundation for streamlining and standardizing prompt creation, enabling non-experts to effectively leverage LLMs. 1 Introduction Large language models (LLMs) have significantly advanced natural language processing, achieving human-level performance on many benchmarks and surpassing humans on certain tasks. To achieve reliable and effective outputs from LLMs, the design of input instructions, known as prompts, is crucial. The systematic development of these prompts, referred to as prompt engineering, plays a central role in determining output quality. However, current approaches to prompt engineering remain highly manual, dependent on expert knowledge, and reliant on extensive trial and error, which creates a substantial barrier to broader LLM adoption. For general users without expertise in prompt design, it is particularly challenging to translate abstract objectives into concrete prompts that yield effective results. This limitation creates demand for new approaches that allow users, regardless of their background, to leverage LLM capabilities through simple, non-technical input. To address this challenge, we propose a prompt generation method that automatically applies task-appropriate prompting techniques based on user-provided task descriptions. The goal is to streamline and standardize prompt creation, enabling diverse users to exploit LLM capabilities through simple, high-level input. Specifically, the proposed method constructs a knowledge base that associates task clusters, characterized by semantic similarity across diverse tasks, with their corresponding prompting techniques. When users input task descriptions, the system assigns them to the most relevant task cluster and dynamically generates high-quality, task-adapted prompts by integrating techniques from the knowledge base, without relying on pre-existing templates or frameworks. 2 Related Work As the importance of prompt engineering has grown, research in this field has advanced rapidly, with a strong focus on automation and optimization. These studies aim to support the discovery of effective prompts and facilitate broader use of LLMs. A variety of prompting techniques have been proposed, including Chain-of-Thought (CoT) prompting for step-by-step reasoning ( Wei et al. 2023 ) , few-shot prompting for providing examples ( Brown et al. 2020 ) , and emotion prompting for incorporating affective instructions ( Li et al. 2023 ) . While such techniques have demonstrated effectiveness in specific problem settings, identifying which technique, or combination of techniques, is optimal for a given task still requires specialized knowledge and extensive trial and error. This limitation highlights the need for frameworks that can automatically select and apply prompting techniques. Automatic Prompt Engineer (APE) ( Zhou et al. 2023b ) is a prominent example of research on automatic prompt generation. APE discovers effective prompts by having LLMs generate and select instructions from task input–output examples, achieving notable results in prompt optimization for specific tasks. However, most existing automatic prompt generation methods, including APE, assume the availability of substantial input–output examples or seed prompts. In contrast, the method proposed in this paper generates task-specific prompts without relying on pre-existing templates, starting instead from abstract, high-level information such as users’ objectives or tasks to be streamlined. Other approaches attempt to generate prompts directly from natural language instructions. For example, Anthropic’s Prompt Generator ( Anthropic 2024 ) allows users to describe task overviews and desired goals, then produces candidate prompts. This method similarly does not require seed prompts or I/O examples, making it an appropriate baseline for our evaluation. Accordingly, our experiments use Anthropic’s Prompt Generator as one of the comparison methods. Building on this background, the novelty of this research lies in proposing a framework that generates effective prompts from scratch by: (1) analyzing and classifying the latent semantic structure of tasks from users’ simple descriptions, (2) selecting from a pre-constructed knowledge base of diverse prompting techniques, and (3) dynamically combining techniques adapted to task characteristics. The aim is to realize a system that produces practical prompts from non-technical input, without requiring specialized expertise. 3 Methodology 3.1 System Overview This section outlines the architecture and implementation of the proposed adaptive prompt generation system, which operates in two main phases: knowledge base construction and prompt generation. In the construction phase, diverse tasks are clustered by semantic similarity, and a knowledge base is created that links each cluster to effective prompting techniques. In the generation phase, user-provided task descriptions are analyzed, and the system dynamically produces high-quality, task-adapted prompts by selecting and integrating techniques from the knowledge base. The following subsections describe the system configuration and processing pipeline in detail. 3.2 System Configuration This research employs multiple large language models (LLMs) in constructing the prompt generation system, assigning each model to roles aligned with its performance characteristics and processing requirements. • LLM (large language model): gemini-2.5-pro • Embedding Model: gemini-embedding-exp-03-07 In subsequent sections, references to “LLM” or “Embedding Model” refer to these specific models. 3.3 Knowledge Base Construction Phase This phase constructs the knowledge base required for prompt generation. Figure 1: Processing flow diagram of Knowledge Base Construction Phase 3.3.1 Task Cluster Definition and Semantic Space Construction The core of this method is the definition of task clusters that semantically classify and organize diverse user tasks. These clusters provide the foundation for selecting appropriate prompting techniques. Task Cluster Derivation Process: To ensure reproducibility, this study combines quantitative clustering with LLM-based semantic interpretation to derive task clusters. • Task Vectorization : Each task’s name and description are combined into a text representation and converted into high-dimensional vectors using the Embedding Model. This embeds the semantic features of each task in vector space. • Quantitative Determination of Optimal Cluster Number (K) : The optimal number of clusters K is determined automatically using quantitative indicators. Specifically, k-means clustering ( MacQueen 1967 ) is executed across candidate values of K (ranging from 3 to fewer than the total number of tasks), and silhouette scores ( Rousseeuw 1987 ) are computed for each. The clustering result with the highest silhouette score is adopted. • LLM-based Cluster Semantic Assignment : For each cluster identified, the LLM is provided with the list of tasks and generates a description of shared characteristics and required capabilities. This yields both detailed descriptions ( cluster_description ) and short identifiers ( cluster_id ) that express each cluster’s features in human-readable form. • Task Cluster Semantic Space Construction : The Embedding Model is then applied to each cluster’s detailed description ( cluster_description ) to map the clusters into high-dimensional vector space. 3.3.2