Quantifying the Reproducibility of Astrophysical Methods with Large Language Models and Information Theory: A Case Study in Spectral Reconstruction
Hsing Wen Lin ∗ (林省文), Zong-Fu Sie (謝宗富) · 2026
Read Quantifying the Reproducibility of Astrophysical Methods with Large Language Models and Information Theory: A Case Study in Spectral Reconstruction by Hsing Wen Lin ∗ (林省文), Zong-Fu Sie (謝宗富) (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-11154.txt.
Introduction
Hsing Wen Lin and Zong-Fu Sie measure how reproducible astrophysical methods are from their written descriptions, using language models and information theory. Published method texts often lack the detail needed for computational reproduction, yet no framework has measured how strongly text pins down valid implementations.
Treating algorithmic reconstruction as a distribution made by large language models, the team uses Shannon entropy and Jensen-Shannon divergence to measure constraint on the hypothesis space. A case study of Trans-Neptunian Object spectral reconstruction from sparse photometry prompts frontier models with title, abstract, and methods text at rising levels. More text clarifies overall structure but leaves implementation-level variance, an entropy floor showing many divergent implementations fit explicit instructions. Executable pipelines confirm the gap. Astrophysicists get a number for reproducibility, not just a hope.
Catalog record
| CREATOR | Hsing Wen Lin ∗ (林省文), Zong-Fu Sie (謝宗富) |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 91787 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.11154
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Quantifying the Reproducibility of Astrophysical Methods with Large Language Models and Information Theory: A Case Study in Spectral Reconstruction Hsing Wen Lin ∗ (林省文), Zong-Fu Sie (謝宗富) Abstract Modern astrophysical studies rely heavily on complex data analysis pipelines; however, published descriptions often lack the detail required for computational reproducibility. In this work, we present an information-theoretic framework to quantify how effectively a method can be reconstructed from its written description. By treating algorithmic reconstruction as a probability distribution generated by Large Language Models (LLMs), we utilize Shannon entropy and Jensen–Shannon divergence to measure how strongly text constrains the hypothesis space of valid implementations. We demonstrate this approach through a case study of Trans-Neptunian Object (TNO) spectral reconstruction from sparse photometry. By prompting frontier LLMs with varying levels of manuscript text (Title, Abstract, and Methods), we find that while increasing text successfully clarifies the overall algorithmic structure, it fails to eliminate variance at the implementation level. This persistent variance establishes an “entropy floor,” demonstrating that multiple divergent implementations remain consistent with explicit instructions. To evaluate practical reproducibility, we convert these reconstructed algorithms into executable pipelines. Our results reveal that, while LLMs easily recover core functional methodologies, they systematically fail to infer the tacit expert knowledge required for strict scientific calibration. This pilot study demonstrates that LLMs can be repurposed as a zero-shot diagnostic tool to audit methodological transparency, helping authors identify missing structural constraints and preserve scientific integrity in an era of automated research. I Introduction Reproducibility is essential to science, yet many published results remain difficult to replicate ( 7 ; 18 , e.g.) . Large-scale surveys point to several key drivers of irreproducibility, including selective reporting, publication pressure, weak statistical rigor, and inadequate oversight ( 6 ) . This suggests that the problem stems from a combination of experimental design, social incentives, and methodological practices. Specifically, vague or incomplete methodological descriptions are a solvable part of this problem. Even when results are reported accurately, missing details about computational procedures, parameters, and implementation can prevent others from replicating the work. Unlike social or cultural factors, the information within a paper can be directly analyzed and quantified. Despite the growing complexity of data pipelines, reproducibility in astronomy and astrophysics has received relatively little systematic study. While a decade-long community effort has successfully promoted open-source practices ( 3 ; 4 ; 2 ; 1 ; 25 ; 29 ) , sharing code alone does not guarantee reproducibility. Software is often unavailable; even when public, it may rely on hidden assumptions or specific configurations that hinder its accessibility and reuse. Recent advances in Large Language Models (LLMs) offer new ways to address these challenges. By generating code from natural language descriptions ( 15 ; 8 ) , LLMs provide a means to reconstruct scientific methods directly from published papers ( 24 ) . If successful, this approach would significantly increase transparency and accessibility in astronomical research. However, the rapid integration of generative AI into astronomical workflows has sparked intense community debate regarding the limits of automated research and the true nature of scientific understanding ( 21 ; 27 ; 28 ) . As recent perspectives highlight, LLMs act as a powerful diagnostic mirror for the field. While they excel at executing well-structured computational tasks, they expose the profound extent to which functional science relies on tacit, domain-specific physical intuition that is rarely formalized in print ( 21 ; 27 ) . From this viewpoint, if a frontier LLM cannot reconstruct a pipeline from a manuscript, the failure may not lie with the model’s capabilities, but rather indicates that the necessary informational constraints are fundamentally absent from the text itself. While completely disentangling LLM capabilities from textual limits remains an open challenge, the observation that multiple distinct frontier models converge on the exact same structural omissions strongly implies a fundamental absence of information in the text, rather than isolated model hallucinations. Measuring this informational gap is central to the broader challenge of quantifying knowledge. 10 suggests that scientific knowledge reduces uncertainty by linking data with an explanatory framework. In this view, knowledge acts as information compression: a successful theory or method narrows the range of outcomes consistent with the data. However, this model assumes the explanatory framework is already known, whereas in many cases, it must be extracted from the text itself. In practice, the explanatory framework itself must often be inferred during text-based reconstruction. Since LLMs operate using a vast prior of potential solutions, the role of the input text is to narrow this space toward the intended approach. This suggests that the core challenge lies not only in the amount of information provided, but also in identifying the correct framework within a broad hypothesis space. While existing frameworks quantify knowledge as information compression ( 10 ) , we still lack a way to measure how much information a paper provides for reconstructing its method. Furthermore, there is no quantitative approach to characterize how this information constrains the range of possible solutions. This gap makes it difficult to assess a study’s reproducibility, even when all documentation is available. In this work, we present an information-theoretic framework to quantify the information in scientific texts and the feasibility of reconstructing their methods. By treating reconstruction as a probability distribution over algorithms generated by LLMs, we measure uncertainty using Shannon entropy ( H ) and alignment using Jensen–Shannon divergence ( JSD ). We apply this framework to an astrophysical case study in spectral reconstruction, demonstrating how textual completeness limits the recovery of scientific workflows from the literature. The rest of this paper is organized as follows. Section II defines the information-theoretic framework used to quantify textual information. Section III details our experimental design for LLM-based reconstruction, while Section IV applies this approach to an astrophysical case study. We discuss the results and their implications in Section V and summarize our findings in Section VI . II Information-Theoretic Framework II.1 Problem Formulation We frame the reconstruction of scientific methods as an inference problem over an algorithm space. Let X represent the information in a scientific text, and Y denote the original method. Given X , a large language model (LLM) generates candidate reconstructions \hat{Y} , which we treat as samples from a conditional distribution: P(Y\mid X). (1) This distribution represents the set of plausible algorithms consistent with the text. When X lacks detail, P(Y\mid X) is broad, reflecting a wide range of possible interpretations. In information-theoretic terms, this corresponds to high information entropy or Shannon entropy ( 26 , hereafter, entropy,) . As we add specific constraints to X , the distribution becomes more concentrated, and the entropy decreases. II.2 The Stochastic Sampler and Inductive Bias In practice, the idealized distribution P(Y\mid X) is accessed through an empirical sampler (in this study, a Large Language Model). To account for the fact that any computational agent introduces its own internal constraints, we define the effective distribution modeled by the agent as: P_{\Phi,\mathcal{T}}(Y\mid X), (2) where \Phi represents the inductive bias of the agent (including its architecture and pre-trained priors) and \mathcal{T} denotes the inference configuration (comprising prompting and decoding strategies). Under this formulation, the candidate reconstructions generated by the model, denoted as \hat{Y} , are treated as independent