What Matters More For In-Context Learning under Matched Compute Budgets: Pretraining on Natural Text or Incorporating Targeted Synthetic Examples?
Mohammed Sabry, Anya Belz · 2025
Read What Matters More For In-Context Learning under Matched Compute Budgets: Pretraining on Natural Text or Incorporating Targeted Synthetic Examples? by Mohammed Sabry, Anya Belz (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-22947.txt.
Introduction
Mohammed Sabry and Anya Belz test whether planted copy patterns make models learn better in context. Their data rewrite, Bi-Induct, slips short copy snippets into normal training text. Some snippets copy forward, some backward, and some mix both. All models got the same compute budget. Sizes ran from small to one billion parameters.
The rewrite did raise copy-head activity, as designed. Yet few-shot scores barely moved on standard tests. On function-style probes, plain training won at the largest size. Removing top copy heads hurt plain models most, which shows their circuits were more load-bearing. The lesson: waking a circuit up is not the same as making the system depend on it.
Catalog record
| CREATOR | Mohammed Sabry, Anya Belz |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 95 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-22947
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
What Matters More For In-Context Learning under Matched Compute Budgets: Pretraining on Natural Text or Incorporating Targeted Synthetic Examples? Mohammed Sabry, Anya Belz Abstract Does explicitly exercising the induction circuit during pretraining improve in-context learning (ICL), or is natural text sufficient, when compute is held constant (iso-FLOPs)? To test whether targeted synthetic data can accelerate the emergence of induction heads and enhance ICL performance, we introduce Bi-Induct , a lightweight curriculum that injects forward-copy ( Induction ), backward-copy ( Anti , as a control), or a balanced mix, into the pretraining stream. We conduct iso-FLOPs pretraining across models from 0.13B to 1B parameters, evaluating effects across three axes: (i) few-shot performance on ICL benchmarks, (ii) head-level telemetry, and (iii) held-out language modeling perplexity. Our findings challenge the intuition that early induction circuit activation directly translates to better ICL. While Bi-Induct accelerates induction head emergence at smaller scales, this does not consistently yield better few-shot generalization. On standard LM benchmarks, Bi-Induct matches natural-only training; on function-style ICL probes, the 1B natural-only model performs best. Stress tests (e.g., label permutation, HITS@1 vs. HITS@3, 1 vs. 10 shots) preserve these trends. Telemetry reveals that larger models trained only on natural text develop broader and earlier-peaking induction heads, despite seeing no explicit induction patterns. Anti-induction data fails to elicit meaningful activation. Perplexity penalties from synthetic data shrink with scale, suggesting that larger models can absorb non-natural patterns with minimal cost. Crucially, ablating the top 2% of induction heads per layer degrades ICL more than random ablations, especially for natural-only models, indicating more centralized, load-bearing circuits. Bi-Induct variants exhibit more redundant induction activity, pointing to different circuit utilization patterns. Overall, we find that inducing activation is not sufficient: improvements in ICL hinge on whether these circuits become functionally necessary. These results underscore the importance of mechanism-aware pretraining diagnostics and data mixtures that foster load-bearing , not merely present, structure. 1 Introduction Transformer language models learn a simple copy pattern early in training: when a token A reappears in context, the model increases the probability of the token that followed the previous A . Prior work identified a two-head motif implementing this behavior and linked it to in-context learning on pattern-matching tasks ( Olsson et al. 2022 ) . Despite its simplicity, this motif typically emerges only after many billions of tokens, well after the first training-loss plateau. Theoretical and empirical studies frame the delay as a phase transition ( Chen et al. 2024 ; Edelman et al. 2024 ) . Accelerating the onset of this transition could reduce compute and expose internal circuits for analysis earlier. A practical way to approach this is to intervene on the data rather than the architecture or objective . In contrast to synthetic-task-only plateau-shortening studies ( Kim et al. 2025 ) and objective-level interventions such as multi-token prediction ( Gloeckle et al. 2024 ) , we adopt a data-rewrite perspective that is easy to deploy at scale: inject a small fraction of targeted inputs into the pretraining stream that selectively excite the putative induction mechanism while keeping the architecture and objective fixed. Concretely, we replace a small slice of natural tokens with synthetic copy snippets that cleanly exercise the copier-selector behavior of induction (forward copy) and anti-induction (backward copy). Copy-style cues are the canonical probe for the induction circuit and are widely used to measure it ( Olsson et al. 2022 ; Nanda & Bloom 2022 ) . While other synthetic families have been studied (for example n-gram statistics ( Edelman et al. 2024 ) , p-hop tasks ( Sanford et al. 2024 ) , and intrinsic tasks ( Gu et al. 2023 ) ), copy snippets align most directly with the hypothesized mechanism and with distributional properties such as burstiness that correlate with the rise of in-context learning ( Chan et al. 2022 ) . These considerations lead to a single, testable question: Under iso-FLOPs, what is more effective for in-context learning—pure natural text, or natural text with a small fraction replaced by directional copy snippets that directly exercise the induction circuit? To answer this, we introduce Bi-Induct , a lightweight curriculum that interleaves short duplicate-span snippets with natural text during early training. We evaluate three variants under matched compute: forward induction, backward anti-induction, and a balanced mix where the direction is chosen independently at each injection . We assess the impact of our pretraining strategies on three axes: (i) downstream ICL performance on standard few-shot benchmarks; (ii) a mechanistic metric, namely the concentration of the top 2% most active attention heads per layer; and (iii) language modeling quality via held-out perplexity. We conduct experiments at three model scales (0.13B, 0.5B, 1B) under iso-FLOPs constraints, also using the 0.13B model as a design space to explore curriculum parameters, including data mix ratios, and span lengths effects. Method and scope: We implement Bi-Induct as a lightweight curriculum, introducing a linearly annealed mixture of synthetic directional-copy data during early pretraining which we compare against a pure-natural baseline under iso-FLOPs conditions. To disentangle model capability from calibration, we evaluate few-shot ICL performance alongside two diagnostics: held-out perplexity (as a quality guardrail) and head-level telemetry measuring attention to copy patterns. Our objective is not to optimize benchmark scores, but to understand when and how targeted data augmentation induces load-bearing behavior for ICL. Contributions: • Matched - compute comparison across scales: We conduct an iso - FLOPs comparison of pure-natural pretraining vs. a small directional-copy replacement, run at 0.13B, 0.5B, and 1B parameters on the same corpus with identical token budgets and training steps per scale. We report per - task and composite few-shot ICL results alongside a perplexity guardrail and attention-head telemetry (forward/anti-copy scores) at the final checkpoint. • Key empirical finding: Early induction \neq better ICL. While Bi-Induct accelerates and broadens induction-head activity at 0.13B and 0.5B parameters, this does not translate into superior ICL endpoints; at 1B, the natural - only baseline outperforms Bi-Induct on function - style probes. • Induction - head ablations revealing circuit load-bearing: Ablating the top 2% induction heads per layer reduces ICL more than matched random ablations at all scales. The largest relative drop is seen in the natural - only baseline , while Bi-Induct variants degrade less, suggesting a more redundant or distributed use of the induction circuit under synthetic augmentation. • Design - lab ablations (0.13B): We use the 0.13B model as a design lab to sweep curriculum parameters, including copy-span length, initial synthetic-to-natural ratio, and annealing schedules, to identify scaling-relevant settings. • ICL robustness diagnostics: We evaluate model robustness through shot-count variation, label- permutation stress tests, and decision - rule sensitivity via HITS@ k on function - style probes. For a concise glossary of terms and symbols, see Appendix A . 2 Related Work Induction heads and the mechanics of ICL: The induction-head motif—a two-head circuit that matches a repeated cue and copies its following token—was introduced by Olsson et al. 2022 , who provided multiple converging tests linking it to the rise of in-context learning (ICL). Follow-up theory and controlled synthetic-task studies characterize the behavior as a phase transition: on Markov-chain data, models move from uniform predictions to unigram heuristics and then abruptly to bigram induction ( Edelman et al. 2024