Model Capability Dominates: Inference-Time Optimization Lessons from AIMO 3

Natapong Nitarach · 2026

Read Model Capability Dominates: Inference-Time Optimization Lessons from AIMO 3 by Natapong Nitarach (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-27844.txt.

Introduction

Nitarach reports that prompt tricks fail next to raw model strength. Tests come from the AIMO 3 math contest setup. Three models face fifty IMO-level problems. Over twenty-three experiments try diverse prompt mixes. Every prompt-level fix falls short in the runs. High-temperature sampling already spreads errors enough. Weaker strategies cut accuracy more than correlation. An eight-point capability gap dominates all tuning. A verifier selector could recover selection loss.

Catalog record

CREATORNatapong Nitarach
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE32223
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.27844

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Model Capability Dominates: Inference-Time Optimization Lessons from AIMO 3 Natapong Nitarach Abstract Majority voting over multiple LLM attempts improves mathematical reasoning, but correlated errors limit the effective sample size. A natural fix: assign structurally different reasoning strategies to different voters to decorrelate errors. We test this — Diverse Prompt Mixer — in the AIMO 3 competition ( 6 ) : 3 models, 23+ experiments, and 50 IMO-level problems on a single H100 80 GB with a 5-hour limit. Every intervention fails. High-temperature sampling already decorrelates errors sufficiently; weaker prompt strategies reduce per-attempt accuracy more than they reduce correlation. Across a 17-point model capability gap and every inference-time optimization we tried, model capability dominates by an order of magnitude. 1 Introduction Majority voting across N inference attempts — self-consistency ( 14 ) — is the standard approach for competition mathematics. It rests on the Condorcet Jury Theorem ( 4 ) : with per-attempt accuracy p>0.5 and independent errors, the majority converges to the correct answer as N grows. In practice, errors are correlated. The same model with the same prompt makes the same systematic mistakes regardless of random seed. The effective sample size becomes N_{\text{eff}}=\frac{N}{1+(N-1)\rho} (1) where \rho is the mean pairwise error correlation. At \rho=0.3 , eight attempts reduce to 2.6 effective votes. The natural fix: assign different reasoning strategies to different voters. If “small cases first” and “work backwards” lead the model through different paths, their errors should be less correlated. We call this Diverse Prompt Mixer . The idea is intuitive. It is also wrong. This paper explains why (Figure 1 ). Figure 1: Model capability dominates. Per-attempt accuracy \hat{p} vs. expected majority-vote score under Binomial voting at N{=}8,16,32 . Four models at their empirical (\hat{p},\,\text{score}) . The 17-point gap between gpt-oss-120b ( \hat{p}{=}0.69 ) and all alternatives dwarfs every prompt optimization tested ( \pm 2 points). Inference-time tricks cannot bridge a capability gap. 2 System Architecture Our system runs entirely on a single H100 80 GB within Kaggle’s 5-hour wall-clock limit. No external APIs, no multi-GPU setups, no pre-computed solutions. 2.1 Model and Serving We use gpt-oss-120b : 116.8B total parameters, 5.1B active via Mixture-of-Experts (MoE). The model is served locally via vLLM ( 7 ) with FP8 quantization for both weights and KV cache. Model weights consume {\sim}75 GB of GPU memory; the remaining {\sim}5 GB supports KV cache for up to N{=}8 concurrent sequences at 65536-token context. Server startup takes {\sim}160 s (including weight loading to page cache). We pre-load weights via sequential file reads before launching vLLM to avoid cold-start latency during inference. 2.2 Inference Pipeline Each problem passes through a 5-stage pipeline: 1. Budget allocation : Divide remaining wall-clock time equally among remaining problems. No bonus, no inflation — this guarantees total solving time \leq notebook limit. 2. Parallel attempts : Launch N{=}8 concurrent inference threads, each with a different random seed but identical system prompt and temperature T{=}1.0 . 3. Tool-integrated reasoning : Each attempt can generate Python code via <tool_call> tags. Code executes in a persistent Jupyter kernel sandbox with access to sympy , numpy , mpmath . Results feed back into the conversation for multi-turn reasoning. 4. Early stopping : If 4 of 8 attempts agree on a non-trivial answer ( >1 ), remaining attempts are cancelled. 5. Entropy-weighted voting : Final answer selected by weighted majority vote, where weight w=1+1/(\text{entropy}+0.1) . Low-entropy (confident) attempts contribute more. 2.3 Time Management The 5-hour limit is the binding constraint. Our budget system: • Total: 18000 s. Reserve 540 s for infrastructure variance, 360 s for startup. • Solving budget: 17100 s for 50 problems. • Per-problem budget: \text{time\_left}/\text{problems\_remaining} . Pure equal division — no bonus from saved time, no inflation. Mathematically guarantees completion. • Hard deadline: if < 30 s remain, return 0 immediately. This ensures the inference server answers all 50 problems before Kaggle’s hard kill. 3 Diverse Prompt Mixer We design four complementary strategies, each a different system prompt with all other parameters identical: 1. Original : Step-by-step with exploration, planning, and verification. 2. Small Cases First : Enumerate n{=}1,2,3,\ldots , discover patterns, conjecture, prove. 3. Work Backwards : List constraints, narrow search space, construct. 4. Classify Then Solve : Identify problem type, recall canonical techniques, apply. Three ensemble configurations plus isolated strategies: Configuration Orig Small Back Class N LB Score Baseline (13 runs) 8 0 0 0 8 39.7 mean Conservative 5 1 1 1 8 40 Aggressive 3 2 2 1 8 40 Equal 2 2 2 2 8 38 Isolated strategies (8 \times each): 8 \times Small Cases 0 8 0 0 8 37 8 \times Work Backwards 0 0 8 0 8 39 8 \times Classify 0 0 0 8 8 36 8 \times Code-first (code-first strategy) 8 41, 38, 34 Formalize-First (F-1) (equations before code) 8 39 Table 1: Prompt diversity configurations on gpt-oss-120b . More diversity = worse performance. Code-first: 41, 38, 34 across three runs (mean 37.7 — below baseline). More diversity = worse performance. The relationship is monotonic: replacing more Original prompts with diverse strategies never helps and eventually hurts (Figure 2 ). Every individual strategy is weaker than the Original. Code-first (41/50) appeared promising but confirmation runs scored 38 and 34, giving a 3-run mean of 37.7 — clearly below baseline. Figure 2: Prompt diversity vs. score on gpt-oss-120b . Blue circles: individual baseline runs ( N{=}13 ). Black diamonds: configuration means. Shaded band: baseline \pm 1\sigma . More diversity monotonically degrades performance. 4 Why It Fails 4.1 Temperature Already Decorrelates At T{=}1.0 , the model explores substantially different reasoning paths even with identical prompts: Temperature LB Score \Delta from baseline T{=}0.5 38 -1.7 T{=}0.8 40 +0.3 T{=}1.0 39.7 — (baseline, 13-run mean) T{=}1.2 , min_p = 0.03 37 -2.7 Adding prompt diversity on top of high temperature is redundant — stochastic diversity already achieves most of the achievable decorrelation. 4.2 Pairwise Correlation Is Already Near Zero Equation 1 assumes constant pairwise correlation \rho across attempts. We measure \hat{\rho} directly using the method-of-moments estimator for a binary exchangeable sequence: \hat{\rho}=\frac{v_{c}(v_{c}-1)/[N(N-1)]-\hat{p}^{2}}{\hat{p}(1-\hat{p})} where v_{c} is the number of correct votes and \hat{p}=v_{c}/N . We compute this across all three models: gpt-oss-120b ( N{=}8 ), Qwen3.5-35B-A3B ( N{=}16 ), and Nemotron-Super-120B ( N{=}3 ). Problems with \hat{p}\in\{0,1\} yield undefined \hat{\rho} and are excluded. This gives 8 computable points (Figure 3 ). All eight are negative. The five estimates from models with N\geq 7 are \hat{\rho}\in\{-0.167,-0.167,-0.100,-0.067,-0.067\} , mean -0.113 . The three Nemotron points all give \hat{\rho}=-0.500 , which is the mathematical minimum for N{=}3 , v_{c}{=}1 (any single-correct-vote outcome at N{=}3 forces this value); they confirm sign but not magnitude. Across all 8 points the mean is -0.258 . When the model is wrong, wrong answers scatter across many distinct values rather than clustering—so two draws are less likely to share the correct answer than \hat{p}^{2} predicts. This leaves no correlation headroom for diversity strategies to exploit. Diverse prompts cannot decorrelate what is already at or below zero. Figure 3: Per-problem \hat{\rho} vs. \hat{p} across three models (10 problems each). Circles = Qwen3.5-35B-A3B ( N{=}16 ); squares = gpt-oss-120b ( N{=}8 ); hollow triangles = Nemotron-Super-120B ( N{=}3 , hollow because \hat{\rho}{=}-0.500 is mathematically

Read the complete text