Monte Carlo Sampling for Analyzing In-Context Examples

Stephanie Schoch, Yangfeng Ji · 2025

Read Monte Carlo Sampling for Analyzing In-Context Examples by Stephanie Schoch, Yangfeng Ji (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-22002.txt.

Description

Stephanie Schoch and Yangfeng Ji develop a Monte Carlo sampling method to analyze in-context examples for large language models. Their 2025 paper deals with a known brittleness: learning from examples in the prompt depends on order, number, and choice of examples.

Ablation studies that vary one factor can miss how the factors interact. Their sampling method varies the number of examples while accounting for order and selection at the same time. Past guidance on how many examples to pick did not always generalize across sets and orderings. Whether one example beats none depended heavily on which example was chosen. They also apply the method to select examples, with a negative result.

Catalog record

CREATORStephanie Schoch, Yangfeng Ji
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE26 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-22002

Browse the full library · About the Tower of Babel