LLM Jaggedness Unlocks Scientific Creativity

Shray Mathur, J. Anibal Boscoboinik · 2026

Read LLM Jaggedness Unlocks Scientific Creativity by Shray Mathur, J. Anibal Boscoboinik (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-10574.txt.

Introduction

Shray Mathur and J. Anibal Boscoboinik study the jaggedness of AI progress through science ideas. Models are not improving at the same pace; skills grow unevenly across tasks and scales. Their 2026 paper introduces SciAidanBench, a benchmark of science questions measuring the science idea skills of language models.

Given a question, models generate as many unique, sound ideas as possible, with the valid total as a proxy for promise. Testing 19 models across 8 makers, the authors find it across and within models. Gains in general idea skills do not transfer evenly to science skills. Stronger models do not improve evenly per prompt, showing bursts on some questions and weak performance on others. Models display uneven strengths across fields. The authors show this jaggedness can be used: compute, pooling, and brainstorms combine models into teams that beat any model.

Catalog record

CREATORShray Mathur, J. Anibal Boscoboinik
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.10574

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

LLM Jaggedness Unlocks Scientific Creativity Shray Mathur, J. Anibal Boscoboinik Abstract 1 Abstract As artificial intelligence advances, models are not improving uniformly. Instead, progress unfolds in a jagged fashion, with capabilities growing unevenly across tasks, domains, and model scales. In this work, we examine this dynamic jaggedness through the lens of scientific idea generation. We introduce SciAidanBench, a benchmark of open-ended scientific questions designed to measure the scientific creativity of large language models (LLMs). Given a scientific question, models are asked to generate as many unique and coherent ideas as possible, with the total number of valid responses serving as a proxy for creative potential. Evaluating 19 base models across 8 providers (30 total variants including reasoning versions), we find that jaggedness manifests both across models and within models. First, in a cross-task comparison between general and scientific creativity, improvements in general creativity do not translate uniformly to scientific creativity, revealing divergent capability profiles across models. Second, at the prompt level, stronger models do not improve uniformly; instead, they exhibit high variability, with bursts of creativity on some questions and limited performance on others. Third, at the domain level, individual models display uneven strengths across scientific subfields, reflecting fragmented internal capability profiles. Finally, we show that this jaggedness can be harnessed. We explore mechanisms of inference-time compute , knowledge pooling , and brainstorming to combine models effectively and construct meta-model ensembles that outperform any single model. Our results position jaggedness not as a limitation, but as a resource, a structural feature of AI progress that, when understood and leveraged, can amplify LLM-driven scientific creativity. 2 Introduction Recent research has revealed that progress in artificial intelligence is not uniform across capabilities but rather follows a jagged frontier [ 5 ] : an uneven landscape of competence. Although the notion of jaggedness has only recently been introduced explicitly, multiple studies implicitly reveal this phenomenon across distinct dimensions of model characteristics. Diverse forms of jaggedness have been identified (Table 1 ), including: across tasks, where AI assistance improves human performance in some task types while worsening it in others [ 5 ] ; across model scale, where larger models may perform worse on certain benchmarks [ 17 ] ; across reasoning effort, where allocating more computational steps can lead to declining accuracy [ 9 ] ; and across problem complexity, where performance can collapse as task difficulty increases [ 24 ] . More recently, behavioral jaggedness has been observed, where small differences in model alignment or value priors produce large behavioral divergences [ 31 ] . Together, these results suggest that AI competence forms a highly irregular surface, one that varies across scale, compute, complexity, and specification. A systematic characterization of such jaggedness remains at an early stage. Table 1: Summary of different dimensions of jaggedness in AI capabilities across recent literature. Paper Jaggedness Dimension Description Dell’Acqua et al. [ 5 ] Task Frontier AI performance varies sharply across task types—high gains within the frontier but degradation on tasks beyond it. McKenzie et al. [ 17 ] Model Scale Increasing model size can reduce performance on certain tasks, revealing non-monotonic scaling effects. Gema et al. [ 9 ] Reasoning Depth Allowing more reasoning steps or compute time can degrade accuracy, showing limits to deeper reasoning. Shojaee et al. [ 24 ] Problem Complexity Model accuracy collapses abruptly beyond certain complexity thresholds, exposing cliffs in reasoning ability. Zhang et al. [ 31 ] Model Behaviour Small differences in model specifications or normative values cause large behavioral divergences. Building on these observations, we aim to study this jaggedness more explicitly and formally through the lens of scientific creativity. Creativity is an ideal measure for revealing differences because it is open-ended and generative, giving models the freedom to explore solution spaces. We build on AidanBench [ 18 ] , a benchmark designed to evaluate general creativity of LLMs via sustained idea generation to open-ended questions. We adapt the methodology to the scientific domain and present SciAidanBench [ 15 ] , a benchmark dataset of 155 open-ended scientific questions, spanning 6 scientific domains. We focus on scientific creativity for studying jaggedness for several reasons. Scientific creativity is closely related to general creativity but with the stricter constraints of scientific reasoning and coherence. Because both tasks share a similar formulation, where models are asked to generate as many valid and meaningful responses as possible, they provide a natural basis for comparison. This allows us to study jaggedness across tasks (general vs scientific creativity), revealing how model capabilities may shift under similar but different task types. The hierarchy of scientific sub-domains offers a way to evaluate jaggedness across topics. Finally, the growing use of large language models (LLMs) in scientific discovery [ 14 , 30 , 21 ] , from literature review [ 23 , 11 ] to experimental design and operation [ 29 , 12 , 16 ] , makes this investigation timely. Understanding jaggedness in this context not only sheds light on the unevenness of model capabilities but also offers insights that could be leveraged to smoothen out these jagged edges and enhance the scientific capabilities of AI systems. Previous efforts to improve LLM diversity include either computationally intensive methods like training interventions [ 4 , 32 ] , decoding strategies [ 3 , 19 ] , or lightweight prompting methods [ 31 ] . Specifically there have been efforts to improve scientific idea generation capabilities of LLMs [ 25 , 22 , 2 ] which rely on multi-llm systems. While these approaches have been shown to improve idea generation capabilities, they remain constrained by structured workflows that limit open-ended generation. Moreover, most multi-LLM systems typically use replicas of the same model, inheriting shared biases and limiting diversity. Through SciAidanBench , we provide a benchmark for evaluating scientific idea generation capabilities of LLMs. The benchmark consists of a diverse set of open-ended scientific questions drawn from across the sciences (physics, chemistry, nanoscience, neuroscience, biology and environmental science). Given a question such as “Propose an experiment to detect a new fundamental force of nature” or “Propose a new method for delivering CRISPR machinery into specific cell types,” models are prompted to generate as many unique ideas as possible, while coherently answering the query. This measures their ability to explore conceptual spaces, and acts as a proxy measure for the need for creatvitiy in science more generally. We use this measure to shed light on jaggedness of LLMs. While prior work has attributed jaggedness to factors such as data composition, training methodology, and post-training alignment, little attention has been given to how it might be mitigated or harnessed. Studying creativity in this setting provides a means to do so, enabling the identification and combination of strengths across models to smoothen out the jaggedness of this developing intelligence frontier. 3 Methodology 3.1 Dataset We construct a dataset comprising 155 open-ended scientific questions designed to evaluate the creative reasoning and sustained ideation capabilities of large language models (LLMs). Each question requires domain-relevant scientific reasoning and open-ended concept generation rather than simple retrieval of factual knowledge. Questions are phrased using templates such as “ Propose a method for… ” or “ Suggest a way to… ” or “ Design an experiment that

Read the complete text