The Growing Pains of Frontier Models: When Leaderboards Stop Separating and What to Measure Next

Adil Amin · 2026

Read The Growing Pains of Frontier Models: When Leaderboards Stop Separating and What to Measure Next by Adil Amin (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-18840.txt.

Introduction

Adil Amin studies why leaderboards stop separating frontier models, using paired coding and question-answering scores across thirty-four models from ten labs. Capabilities cooperate overall, yet cooperation varies fivefold by lab, and labs pivot between coding-first and reasoning-rich stances. His per-release residual diagnoses each lab emphasis from just two public scores. Pretraining shifts prove lasting while post-training shifts reverse within a release, a split that tells builders whether to retrain or wait.

Catalog record

CREATORAdil Amin
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE46346
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.18840

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

The Growing Pains of Frontier Models: When Leaderboards Stop Separating and What to Measure Next Adil Amin Abstract Leaderboards rank frontier models on independent axes but do not reveal whether capabilities reinforce or trade off across releases—and at the frontier, this interaction is the more informative signal. We decompose paired SWE-bench and GPQA Diamond scores into a population coupling trend and per-release residual ( h h -field) that diagnoses capability emphasis from two public benchmark scores. Across 34 models from 10 labs (2024–2026), capabilities cooperate ( r = + 0.72 r=+0.72 , p < 10 − 6 p<10^{-6} ), but cooperation varies systematically: per-lab coupling slopes span 5 × 5\times (Google 1.15 1.15 vs. DeepSeek 0.23 0.23 ), and labs pivot—DeepSeek reversed from reasoning-rich to coding-first ( Δ ​ h = 15.9 \Delta h=15.9 pp); Anthropic oscillates between coding excursions and recovery. The population regression serves as an isocline phase boundary: the same ( a / b ) ⋅ B 1 \sqrt{(a/b)\cdot B_{1}} classifier that identifies the base-scale coupling transition 2 classifies frontier models and already detects mixed-phase behavior at the next transition (two models below the GPQA–IFEval isocline). The h h -field is not just diagnostic—it tells you what to change. Pretraining establishes coupling at 0.871 0.871 while RLHF adds 0.081 0.081 2 : pretraining-level shifts are permanent (DeepSeek’s four-release reversal persists), post-training shifts are reversible (Anthropic’s three coding excursions each recover within one release), and inference compute alone shifts h h by + 7.8 +7.8 pp without retraining. Knowing which component dominates determines whether to retrain or wait. We provide a three-step diagnostic (locate, classify, predict), a per-lab measurement-priority table, and seven falsifiable predictions with timestamped criteria. Five post-cutoff releases fall within the 95% prediction interval. Code, data, and an interactive dashboard: https://zehenlabs.com/cape/ . 1 Introduction When a frontier lab releases a new model, the community asks two questions: how well does it code, and how well does it reason? These questions are always asked separately. SWE-bench measures coding; GPQA Diamond measures reasoning; each gets its own leaderboard row, its own trajectory, its own narrative. But no one asks whether improving one helps or hurts the other—and at the frontier, this interaction turns out to be the more informative signal. We test it. Using paired benchmark scores (SWE-bench Verified and GPQA Diamond) across 34 frontier models from 10 labs, we measure whether capabilities reinforce or undermine each other across a lab’s release sequence. Capabilities cooperate ( r=+0.72 , p<10^{-6} ), but the pattern of cooperation varies by lab and over time. Each lab leaves a fingerprint—a one-number diagnostic that summarizes whether its models are becoming better reasoners, better coders, or both. Leaderboards answer “who is ahead?” CAPE answers a different question: what kind of progress is this? A release can climb the leaderboard by moving along the population trend, by specializing away from it, or by exhausting an axis that no longer separates frontier models—a problem now documented across half of all active benchmarks 1 . Those cases require different responses. We operationalize this with three quantities available from public scores alone: population coupling ( r ), release residual ( h ), and saturation ratio ( \sigma ), so that any practitioner can reproduce the diagnosis from the same model-card numbers. Where 2 establishes that the coupling transition is engineerable at base scale, showing that architecture, curation, and distillation shift the critical threshold, the present work asks: once models are deep in the cooperative regime, what determines their trajectory through coupling space, and what can practitioners do about it? Contributions. 1. We measure cooperative frontier coupling on the largest cross-lab panel to date (34 models, 10 labs), with a matched core set and explicit data-provenance curation. 2. We introduce the h -field as a per-lab diagnostic that tracks release-level capability emphasis and trajectory changes over time. 3. We show that cooperation cascades: five open-weight families independently confirm a second capability transition at 30–72B, the isocline phase classifier generalizes across cascade levels, and per-lab coupling slopes vary 5\times . 4. We provide a three-step diagnostic (locate, classify, predict), a per-lab measurement-priority table, and seven timestamped falsifiable predictions for the next 12 months. Terminology. The companion paper 2 measures local coupling \gamma_{12}(N)=\Delta B_{2}/\Delta B_{1} between consecutive model sizes within a family—a derivative that tracks how capabilities interact as models scale. This paper primarily uses population coupling r(B_{1},B_{2}) , the Pearson correlation across a panel of models, which captures cooperative structure across labs. Both measure the same underlying coupling at different resolutions: \gamma_{12} detects sign changes within families; r confirms the cooperative regime persists across the full frontier population. 2 Framework: Two Scores, One Diagnostic The diagnostic requires only two benchmark scores per model and produces three outputs: regime state, lab-relative residual, and transition risk. All definitions are self-contained; no external reference is required. 2.1 Capability coupling Let B_{1} and B_{2} be two benchmark scores measured on the same model. The population coupling is the Pearson correlation r(B_{1},B_{2}) computed across a panel of models. When r>0 , capabilities reinforce each other (“cooperative”); when r<0 , they trade off (“antagonistic”). For frontier measurement, we use B_{1}=\text{SWE-bench Verified} (autonomous coding) and B_{2}=\text{GPQA Diamond} (graduate-level scientific reasoning). These axes were chosen because they are widely reported, represent distinct capability dimensions (code generation vs. knowledge-intensive reasoning), and have sufficient dynamic range at frontier scale. 2.2 The h -field: lab-relative residual Given the population regression \hat{B}_{2}=\beta_{1}B_{1}+\beta_{0} , the h -field for model i is the signed residual: h_{i}=B_{2,i}-(\beta_{1}B_{1,i}+\beta_{0}) (1) A positive h_{i} indicates reasoning-rich performance relative to the coding–reasoning trend; a negative h_{i} indicates coding-rich. The per-lab mean \bar{h}_{\rm lab} summarizes a lab’s characteristic deviation from the population. For example, Claude Opus 4.6 has SWE =80.8 and GPQA =91.3 . Using Eq. 3 , h=91.3-(0.513\times 80.8+46.4)\approx+3.4 , placing it slightly reasoning-rich relative to the population trend. The h -field is descriptive, not causal: it summarizes where a model sits relative to the population trend, not why. It is useful for comparing releases within and across labs without access to proprietary training details. Decomposition. The aggregate h has internal structure. In the phase-transition framework, h decomposes into contributions analogous to distinct thermodynamic control parameters: h_{i}\;=\;h_{\rm pre}+h_{\rm post}+h_{\rm arch} (2) where h_{\rm pre} reflects pretraining choices (data composition, curriculum—permanent changes that shift the phase boundary), h_{\rm post} reflects post-training interventions (RLHF, distillation, inference compute—reversible changes that deflect the system but relax when removed), and h_{\rm arch} reflects architectural decisions (width, depth, routing—structural and permanent). The physics analogy is precise: h_{\rm pre} acts like doping (changes the material), h_{\rm post} acts like an applied field (changes what is done to the material), and h_{\rm arch} acts like strain (changes the lattice); Appendix E develops the correspondence. At base scale, these are partially separable: Gemma-4-E4B shows pretraining coupling 0.871 and RLHF post-training adds +0.081 2 —pretraining contributes {\sim}10{:}1 over post-training alignment. Operationally, h_{\rm pre} shifts persist across releases (they change the model); h_{\rm post} shifts may reverse (they change what is done to the model). Section 4 provides two tests: the +7.8 pp compute-tier shift isolates h_{\rm post} , and Anthropic’s three oscillation cycles demonstrate that post-training excursions relax within one release. 2.3 Phase classification We assign each model to

Read the complete text