The Prompt is Mightier than the Example
Shengzhe Xu, Nikhil Muralidhar, Naren Ramakrishnan · 2025
Read The Prompt is Mightier than the Example by Shengzhe Xu, Nikhil Muralidhar, Naren Ramakrishnan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-18485.txt.
Introduction
Shengzhe Xu, Nikhil Muralidhar, and Naren Ramakrishnan ask how much know-how in a prompt can stand in for worked samples. Big language models can fake rows of a data table when shown a few true rows. Top-grade fakes often need many such rows, which may be dear or scarce. At the same time, the models hold vast built-in lore that might fill the gap.
Their method, Knowledge-Guided Prompting, writes field lore straight into the prompt. Tests then trade lore against sample count in a set way. The trade follows a clean law: as lore grows and samples shrink, fake quality moves in a path that can be foretold. The law held across their tests on mixed data tables. Prompt lore can thus serve as a cheap stand-in for scarce samples, or boost them when both are at hand.
Catalog record
| CREATOR | Shengzhe Xu, Nikhil Muralidhar, Naren Ramakrishnan |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-18485
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
The Prompt is Mightier than the Example Shengzhe Xu, Nikhil Muralidhar, Naren Ramakrishnan Abstract . Numerous recent prompt optimization approaches like chain-of-thought, tree-of-thought prompting, have been demonstrated to significantly improve the quality of content generated by large language models (LLMs). In-context learning (ICL), a recent paradigm where a few representative examples guide content generation has also led to strong and consistent improvements in generation quality of LLM generated content. This idea has been applied to great effect in synthetic tabular data generation, where LLMs, through effective use of ICL and prompt optimization, can generate data that approximate samples from complex, heterogeneous distributions based on representative examples. However, ensuring high-fidelity synthetic data often requires a very large number of ICL examples which may be unavailable or costly to obtain. At the same time, as LLMs get larger and larger, their in-built prior knowledge becomes vast and can potentially substitute for specific data examples. In this paper, we introduce Knowledge-Guided Prompting (KGP) as a new knob in prompt optimization and explore the ability of KGP-based prompt optimization to offset the cost of ICL. Specifically, we explore the question ‘how many examples can a prompt substitute for?’ and explore knowledge-guided prompting (KGP) where domain knowledge, either inferred or available, is explicitly injected into the prompt, reducing dependence on ICL examples. Our experiments systematically explore the trade-off between ICL and KGP, revealing an empirical scaling law that quantifies how quality of generated synthetic data varies with increasing domain knowledge and decreasing example count. We classify prior knowledge into strong knowledge (e.g., symbolic constraints, statistical priors) versus weaker knowledge (e.g., monotonicity constraints, dependency relationships) and explore relationships between both forms and in-context examples. Our results demonstrate that knowledge-guided prompting can be a scalable alternative, or addition, to in-context examples, unlocking new approaches to synthetic data generation. DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06 1. Introduction Synthetic data generation is a key ingredient in many KDD pipelines, e.g., to help overcome privacy limitations ( Abay et al., 2019 ; Jordon et al., 2018 ) , to support machine learning in domains where there are imbalanced classes ( He and Garcia, 2009 ) , to enable data augmentation when real data is scarce ( Choi et al., 2017 ) , and to simulate rare or extreme events that are difficult to capture in real-world datasets ( Esteban et al., 2017 ) . Many powerful ML algorithms, e.g., generative adversarial networks (GANs) ( Goodfellow et al., 2020 ; Hui et al., 2023 ; Wang et al., 2020a ; Xie and Huang, 2019 ; Xu et al., 2019 ) rely on synthetic data generation as a key ingredient to their workflow. Recently, large language models (LLMs), especially the latest variants such as GPT-4o and the LLaMA series, have been examined for their potential as structural data regressors or generators. Most modern LLMs are based on the transformer architecture ( Vaswani et al., 2017 ) with parameters ranging from few millions to billions ( Hoffmann et al., 2022 ) , and researchers have developed creative ways to harness LLMs in traditional machine learning and data contexts. For instance, LIFT ( Dinh et al., 2022 ) transforms table rows of raw numerical data into sentences such as ‘An Iris plant with sepal length 5.1cm, sepal width 3.5cm…’, and employs an LLM to solve traditional machine learning taasks like classification, regression, and generation. GReaT ( Borisov et al., 2022 ) fine-tunes an LLM for synthetic data generation and show that even small-scale models such as Distill-GPT ( Radford et al., 2019 ) are capable of synthetic data generation ( Borisov et al., 2022 ) . While the above works fine-tune an LLM to support data generation, newer variants support synthetic data generation out-of-the-box, i.e., with in-context learning (ICL) ( Seedat et al., 2024 ) . After prompt engineering paradigms are carefully designed to facilitate in-context learning, just a few example rows in the context window can enable an LLM to generate synthetic data that conforms to the inferred properties of the supplied rows. However ensuring fidelity to complex, heterogeneous distributions by finding representative examples remains a challenge and requires careful experimentation. Just as sampling points on a curve might require more points where there are shifts in behavior (versus regions where there is more normalcy), we will require more ICL examples for some regions versus others to support improved generalization. In this paper, we investigate if prompt optimization by injecting prior knowledge into the prompt , can help in synthetic data generation and, more specifically, whether it can replicate behavior that previously required an inordinate number of ICL examples. We ask the question: ‘how many examples can a knowledge-guided prompt substitute for?’ and aim to capture this tradeoff by defining knowledge levels and studying their interplay with the number of ICL examples. Through our experiments, we show how prompt optimization with explicit domain priors including symbolic, statistical priors, can be infused into prompts to reduce or even eliminate ICL examples. Our approach is dubbed knowledge-guided prompting (KGP), where domain knowledge, either inferred or available, is explicitly injected into the prompt (serving as an additional knob for prompt optimization), reducing the dependence on ICL examples. This enables new approaches to synthetic data generation than purely data-driven or purely knowledge-driven approaches. Our contributions are: (1) We propose a new approach for prompt optimization called knowledge-guided prompting (KGP), to improve the quality of structured data generation while at the same time limiting the number of ICL examples required. This approach is especially valuable in scenarios where there is data paucity or we wish to reduce the number of tokens while maintaining synthetic data generation quality. (2) The KGP approach proposed here systematizes prior knowledge into strong knowledge (e.g., symbolic constraints, statistical priors) versus weaker knowledge (e.g., monotonicity constraints, dependency relationships), and explores their interplay w.r.t. the number of ICL examples. (3) Through numerous experiments, we demonstrate that our KGP approach yields better generation quality than using purely ICL examples, unlocking new hybrid approaches to synthetic data generation. Most importantly, we demonstrate how the KGP framework provides a framework to think about scaling laws that predict the number of examples needed for given levels of prior knowledge. (a) (b) Figure 1 . (a) a traditional synthetic tabular data generation pipeline using LLMs encodes sample data as in-context learning examples to drive the generation process. (b) Our prompt optimization approach based on knowledge-guided prompting (KGP), incorporates automatically inferred domain knowledge, providing the LLM-based generator a complementary context in addition to ICL examples. Our experimental findings indicate that such global property conditioning via. KGP leads to a significant improvement in synthetic data generation quality, indicating that KGP can indeed be employed as a useful knob for prompt optimization. 2. Related Work Tabular data synthesis and representation learning for tables have been extensively studied ( Du et al., 2021 ; Wang et al., 2021 ; Zhang et al., 2023b ; Margeloiu et al., [n. d.] ; Fang et al., 2024 ; Patki et al., 2016 ; Du and Li, 2024 ; McKenna et al., 2019 ; Park et al., 2018 ; Wang et