Monte Carlo Sampling for Analyzing In-Context Examples
Stephanie Schoch, Yangfeng Ji · 2025
Read Monte Carlo Sampling for Analyzing In-Context Examples by Stephanie Schoch, Yangfeng Ji (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-22002.txt.
Description
Stephanie Schoch and Yangfeng Ji develop a Monte Carlo sampling method to analyze in-context examples for large language models. Their 2025 paper deals with a known brittleness: learning from examples in the prompt depends on order, number, and choice of examples.
Ablation studies that vary one factor can miss how the factors interact. Their sampling method varies the number of examples while accounting for order and selection at the same time. Past guidance on how many examples to pick did not always generalize across sets and orderings. Whether one example beats none depended heavily on which example was chosen. They also apply the method to select examples, with a negative result.
Catalog record
| CREATOR | Stephanie Schoch, Yangfeng Ji |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 26 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-22002