Scaling Trends for Multi-Hop Contextual Reasoning in Mid-Scale Language Models
Brady Steele, Micah Katz · 2026
Read Scaling Trends for Multi-Hop Contextual Reasoning in Mid-Scale Language Models by Brady Steele, Micah Katz (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-04254.txt.
Introduction
Brady Steele and Micah Katz run a controlled comparison of rule-based matching and multi-agent language models on multi-hop reasoning. Multi-hop tasks need facts combined across documents, not just found in one place. Their synthetic framework holds the task fixed while methods and models vary, which isolates what each approach truly adds.
Across one hundred twenty trials with four mid-scale models, a clear split appears. Rules score perfectly on structured retrieval yet almost fail on cross-document reasoning, while multi-agent systems show the reverse pattern and lead on reasoning. Extra agents help only models that already reason well enough, which suggests amplification rather than rescue. Active parameter count, not total count, predicts mixture-of-experts performance, and a smaller newer model beats a larger older one, pointing to training quality.
Catalog record
| CREATOR | Brady Steele, Micah Katz |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 37 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-04254
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Scaling Trends for Multi-Hop Contextual Reasoning in Mid-Scale Language Models Brady Steele, Micah Katz Abstract We present a controlled study of multi-hop contextual reasoning in large language models, providing a clean demonstration of the task-method dissociation : rule-based pattern matching achieves 100% success on structured information retrieval but only 6.7% on tasks requiring cross-document reasoning, while LLM-based multi-agent systems show the inverse pattern, achieving up to 80% on reasoning tasks where rule-based methods fail. Using a synthetic evaluation framework with 120 trials across four models (LLaMA-3 8B, LLaMA-2 13B, Mixtral 8 × \times 7B, DeepSeek-V2 16B), we report three key findings: (1) Multi-agent amplification depends on base capability : statistically significant gains occur only for models with sufficient reasoning ability ( p < 0.001 p<0.001 for LLaMA-3 8B, p = 0.014 p=0.014 for Mixtral), with improvements of up to 46.7 percentage points, while weaker models show no benefit, suggesting amplification rather than compensation; (2) Active parameters predict reasoning performance : Mixtral’s performance aligns with its ∼ \sim 12B active parameters rather than 47B total, consistent with the hypothesis that inference-time compute drives reasoning capability in MoE architectures; (3) Architecture quality matters : LLaMA-3 8B outperforms LLaMA-2 13B despite fewer parameters, consistent with known training improvements. Our results provide controlled quantitative evidence for intuitions about multi-agent coordination and MoE scaling, while highlighting the dependence of multi-agent benefits on base model capability. We release our evaluation framework to support reproducible research on reasoning in mid-scale models. 1 Introduction Large language models (LLMs) have demonstrated remarkable capabilities across a wide range of tasks, from text generation to code synthesis to mathematical reasoning ( Brown et al. 2020 ; Chowdhery et al. 2022 ; OpenAI 2023 ) . Understanding how these capabilities vary with model size is crucial for both capability prediction and resource allocation in model development ( Kaplan et al. 2020 ; Hoffmann et al. 2022 ) . While prior work has established power-law relationships between model size and performance on many benchmarks, the behavior of more complex cognitive abilities, particularly those requiring multi-step reasoning across disparate information sources, remains an active area of investigation. Multi-hop contextual reasoning presents a particularly interesting case for scaling analysis. Unlike tasks that can be solved through pattern matching or single-step retrieval, multi-hop reasoning requires models to: (1) identify relevant pieces of information distributed across a context, (2) recognize implicit relationships between these pieces, and (3) synthesize them to reach conclusions not explicitly stated in any single source. This capability is fundamental to real-world applications including document understanding, scientific discovery, and security analysis. In this work, we investigate the scaling properties of multi-hop contextual reasoning using a controlled synthetic evaluation framework. Our framework generates structured inference tasks that require connecting multiple pieces of contextual information, for example, linking a family member’s name with a birth year to infer a plausible password pattern. Critically, our evaluation uses entirely synthetic data with no real user information, enabling rigorous analysis without privacy concerns. Our work makes the following contributions: 1. Controlled demonstration of task-method dissociation: We provide a clean, quantitative confirmation that rule-based methods achieve 100% on pattern-matching tasks but only 6.7% on reasoning tasks, while LLM agents show the inverse, offering a controlled synthetic setting to study a well-known phenomenon. 2. Multi-agent amplification depends on base capability: We show that multi-agent coordination provides large, statistically significant improvements on reasoning tasks (up to +46.7 percentage points, p<0.001 ), but only for models with sufficient base reasoning capability. Weaker models show no benefit, suggesting multi-agent systems amplify existing capability rather than compensate for its absence. 3. Active parameters predict MoE reasoning: We provide evidence that Mixtral’s reasoning performance aligns with its active parameter count ( \sim 12B) rather than total parameters (47B), consistent with the hypothesis that inference-time compute drives reasoning capability in MoE architectures. 4. Accessible evaluation framework: We release a synthetic evaluation framework enabling reproducible research on multi-hop reasoning using consumer hardware, with all data generated synthetically to avoid privacy concerns. 2 Related Work 2.1 Scaling Laws for Language Models The study of neural scaling laws has revealed consistent relationships between model size, data, compute, and performance. Kaplan et al. 2020 established power-law relationships for language model loss, showing smooth improvement with scale across many orders of magnitude. Hoffmann et al. 2022 refined these findings, demonstrating that optimal compute allocation requires scaling data proportionally with parameters. However, subsequent work has shown that different capabilities may scale differently. Wei et al. 2022b documented emergent abilities, capabilities that appear abruptly at certain scales rather than improving gradually. These include arithmetic, multi-step reasoning, and instruction following. Schaeffer et al. 2023 challenged whether these emergences are fundamental or artifacts of metric choice, sparking ongoing debate about the nature of capability scaling. Our work contributes to this literature by providing detailed scaling analysis for multi-hop contextual reasoning, a capability not systematically studied in prior scaling work. 2.2 Emergent Capabilities in Large Language Models The concept of emergence in LLMs has generated significant interest and debate. Wei et al. 2022b identified numerous tasks exhibiting emergent behavior, where performance remains at chance level until a threshold model size, then rapidly improves. Examples include multi-digit arithmetic, word unscrambling, and Persian QA. Ganguli et al. 2022 argued that unpredictable emergence poses challenges for AI safety, as dangerous capabilities might appear suddenly during scaling. Conversely, Schaeffer et al. 2023 demonstrated that some apparent emergences disappear with continuous metrics, suggesting they may be measurement artifacts. Recent theoretical work has sought to explain emergence through lens of circuit formation ( Olsson et al. 2022 ) , phase transitions in loss landscapes ( Power et al. 2022 ) , and capability composition ( Arora & Goyal 2023 ) . Our observations are consistent with phase transition interpretations, though our limited model range does not allow definitive conclusions. 2.3 Multi-Agent LLM Systems Multi-agent architectures have emerged as a powerful paradigm for enhancing LLM capabilities on complex tasks. Hong et al. 2024 introduced MetaGPT, using Standard Operating Procedures to coordinate agents on software engineering tasks. Wu et al. 2023 developed AutoGen for customizable multi-agent conversations, demonstrating improvements on coding and math benchmarks. Du et al. 2023 showed that multi-agent debate improves factuality and reasoning, while Liang et al. 2023 found that diverse agent personas enhance problem-solving. Chen et al. 2024 demonstrated emergent social behaviors in multi-agent LLM systems. However, the relationship between base model capability and multi-agent effectiveness has received limited systematic attention. Our work addresses this gap by providing controlled evidence that multi-agent benefits depend critically on base model reasoning ability, a finding with practical implications for deployment decisions. 2.4 Compositional and Multi-Hop Reasoning Multi-hop reasoning requires combining multiple pieces of information to reach conclusions. Benchmarks including HotpotQA ( Yang et al. 2018 ) , MuSiQue ( Trivedi et al. 2022 ) , and StrategyQA ( Geva et al. 2021 ) evaluate this capability, though with natural language rather than the controlled synthetic setting we employ. Press et al. 2023 studied compositional reasoning in LLMs, finding systematic failures on tasks requiring combining learned facts. Dziri et al. 2023 analyzed reasoning chains and found that LLMs often rely on shortcuts rather than genuine multi-step reasoning. Ofir et al. 2024 proposed theoretical frameworks for