AstroMLab 4: Benchmark-Topping Performance in Astronomy Q&A with a 70B-Parameter Domain-Specialized Reasoning Model

Tijmen de Haan 1,2,∗ , Yuan-Sen Ting 3,4 , Tirthankar Ghosal 5 , Tuan Dung Nguyen 6 , Alberto Accomazzi 7 , Emily Herron 5 , Vanessa Lama 5 , Rui Pan 8 , Azton Wells 9 , Nesar Ramachandra 9 · 2025

Read AstroMLab 4: Benchmark-Topping Performance in Astronomy Q&A with a 70B-Parameter Domain-Specialized Reasoning Model by Tijmen de Haan 1,2,∗ , Yuan-Sen Ting 3,4 , Tirthankar Ghosal 5 , Tuan Dung Nguyen 6 , Alberto Accomazzi 7 , Emily Herron 5 , Vanessa Lama 5 , Rui Pan 8 , Azton Wells 9 , Nesar Ramachandra 9 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-17592.txt.

Introduction

Ten authors led by Tijmen de Haan tune a 70-billion-part language model into a star-space pro that tops a hard quiz board. Broad models fumble fine points of such fields, which blocks their use as lab aides. Their model drank deep of star-space text first, then took lessons with thought chains spelled out. It can now answer at once or show its work step by step.

On a held-back set of near four thousand quiz items, the model scored 89 percent. That mark ties the best closed giant models in the run, at less run cost. Years of prior smaller builds fed this one in the team line. Open star aides can thus stand with paid giants.

Catalog record

CREATORTijmen de Haan 1,2,∗ , Yuan-Sen Ting 3,4 , Tirthankar Ghosal 5 , Tuan Dung Nguyen 6 , Alberto Accomazzi 7 , Emily Herron 5 , Vanessa Lama 5 , Rui Pan 8 , Azton Wells 9 , Nesar Ramachandra 9
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE27 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-17592

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

AstroMLab 4: Benchmark-Topping Performance in Astronomy Q&A with a 70B-Parameter Domain-Specialized Reasoning Model Tijmen de Haan 1,2,∗ , Yuan-Sen Ting 3,4 , Tirthankar Ghosal 5 , Tuan Dung Nguyen 6 , Alberto Accomazzi 7 , Emily Herron 5 , Vanessa Lama 5 , Rui Pan 8 , Azton Wells 9 , Nesar Ramachandra 9 Abstract General-purpose large language models (LLMs), despite their broad capabilities, often struggle with specialized domain knowledge, a limitation particularly pronounced in more accessible, lower-parameter versions. This gap hinders their deployment as effective agents in demanding fields such as astronomy. Building on our prior work with AstroSage-Llama-3.1-8B, this study introduces AstroSage-Llama-3.1-70B, a significantly larger and more advanced domain-specialized natural-language AI assistant. It is designed for research and education across astronomy, astrophysics, space science, astroparticle physics, cosmology, and astronomical instrumentation. Developed from the Meta-Llama-3.1-70B foundation, AstroSage-Llama-3.1-70B underwent extensive continued pre-training (CPT) on a vast corpus of astronomical literature, followed by supervised fine-tuning (SFT) and model merging. Beyond its 70-billion parameter scale, this model incorporates refined datasets, judiciously chosen learning hyperparameters, and improved training procedures, achieving state-of-the-art performance on complex astronomical tasks. Notably, we integrated reasoning chains into the SFT dataset, enabling AstroSage-Llama-3.1-70B to either answer the user query immediately, or first emit a human-readable thought process. Evaluated on the AstroMLab-1 benchmark ( Ting et al. 2024 ) —comprising 4,425 questions from literature withheld during training—AstroSage-Llama-3.1-70B achieves state-of-the-art performance. It surpasses all other tested open-weight and proprietary models, including leading systems like o3, Gemini-2.5-Pro, Claude-3.7-Sonnet, Deepseek-R1, and Qwen-3-235B, even those with API costs two orders of magnitude higher. This work demonstrates that domain specialization, when applied to large-scale models, can enable them to outperform generalist counterparts in specialized knowledge areas like astronomy, thereby advancing the frontier of AI capabilities in the field. I Introduction Astronomy and its related fields demand sophisticated tools that can process vast amounts of specialized knowledge. Large Language Models (LLMs) have emerged as promising assistants for this domain, offering capabilities as research collaborators, educational resources, and knowledge repositories. Domain-specialized models demonstrate particular cost-effectiveness in such contexts, as their parameters can be optimized for specific knowledge domains rather than distributed across the breadth of general internet content. Our previous work, AstroSage-Llama-3.1-8B ( de Haan et al. 2025 ) (hereafter TdH25), established that a relatively modest 8-billion parameter LLM, when extensively trained on astronomical content, could match or exceed the performance of much larger general-purpose models on astronomical knowledge tasks ( Ting et al. 2024 ) . This finding highlighted the potential of domain specialization for creating efficient, high-performing AI assistants. We now introduce AstroSage-Llama-3.1-70B, a 70-billion parameter language model that represents an advancement in specialized AI for astronomy. Our central research question asks whether domain specialization merely improves efficiency or can enable specialized models to outperform even the largest commercial alternatives. Following TdH25’s approach with Meta-Llama-3.1-8B, we applied similar domain specialization techniques to the larger Meta-Llama-3.1-70B foundation. Beyond the increased parameter count, we implemented several key enhancements: expanded and refined datasets for both continued pre-training (CPT) and supervised fine-tuning (SFT); optimized learning hyperparameters based on public benchmarks and our own experimentation; and an explicit reasoning capability that enables step-by-step analytical processes before generating answers. The core hypothesis driving this work is that a larger specialized model can elevate AI performance across astronomy, astrophysics, space science, astroparticle physics, cosmology, and astronomical instrumentation—collectively referred to as “astronomy” throughout this paper. While AstroSage-Llama-3.1-8B successfully matched larger models’ performance, AstroSage-Llama-3.1-70B aims to surpass even advanced commercial alternatives. Beyond testing this hypothesis, we are making our trained model openly available to serve as a resource for researchers, educators, and students in the field, as well as non-scientists interested in astronomy. This paper is structured as follows: Section II details the multi-stage training pipeline for AstroSage-Llama-3.1-70B, including improvements to our CPT and SFT corpora and procedures, and the model merging strategy for the final release. Section III discusses potential applications. Section IV evaluates AstroSage-Llama-3.1-70B against the updated Ting et al. 2024 benchmark, comparing it with other recent proprietary and open-weight models. II Model Architecture and Training Methodology AstroSage-Llama-3.1-70B is derived from the Meta-Llama-3.1-70B architecture ( Dubey et al. 2024 ) . This base model was selected for consistency with TdH25, which chose Meta-Llama-3.1-8B for its state-of-the-art general capabilities and permissive licensing. The tokenizer from Meta-Llama-3.1-70B-Instruct was used without modification. Following the methodology of TdH25, our development process comprised three main stages: continued pre-training (CPT), supervised fine-tuning (SFT), and model merging. II.1 Continued Pre-training (CPT) The objective of CPT is to imbue the base model with extensive domain-specific knowledge from astronomical literature. The CPT dataset for AstroSage-Llama-3.1-70B builds upon the comprehensive corpus developed in TdH25, which included approximately 250,000 arXiv preprints (astro-ph, gr-qc from 2007-2024), nearly 30,000 Wikipedia articles related to astronomy, and internet-available textbooks. The knowledge cutoff for the astro-ph papers included in the CPT dataset remains January 2024. This dataset was further enhanced: • We applied ftfy ( Speer 2019 ) for consistent Unicode text normalization and some rule-based repetition removal to correct OCR failures, supplementing the perplexity-based cleaning methods detailed in TdH25. • To preserve general language understanding and mitigate catastrophic forgetting due to specialization, we incorporated a random selection of samples from the FineWeb dataset ( Penedo et al. 2023 ) into each training epoch. This addition of previous pretraining tokens during CPT is sometimes known as “replay”. Notably, the specific FineWeb samples were varied for each epoch, ensuring diverse exposure to general web text. The CPT phase was conducted on the Oak Ridge Leadership Computing Facility (OLCF) Frontier supercomputer utilizing AMD Instinct MI250X GPUs. The Meta-Llama-3.1-70B base model underwent 2.5 epochs of CPT, an extensive training process that utilized approximately 168,000 GPU-hours. Several learning hyperparameters were modified compared to TdH25. Firstly, the learning rate was scaled relative to that in Dubey et al. 2024 to maintain the same learning rate per token . Secondly, the warm-up period was extended to 0.15 epochs. Finally, we increased the weight decay regularization term to 0.1, aligning with Dubey et al. 2024 . II.2 Supervised Fine-tuning (SFT) Following CPT, the model underwent SFT to develop its instruction-following and conversational capabilities, and to instill behaviors such as chain-of-thought and self-reflection (collectively termed “reasoning”). Figure 1: Composition of the AstroSage-Llama-3.1-70B SFT training dataset. The combination of reasoning-focused datasets (41.8%) with domain-specific astronomy Q&A (30.8%) reflects our strategy to develop a model that combines analytical thinking with specialized knowledge. General instruction-following data (OpenHermes 2.5) helps maintain versatility while preserving domain expertise. File sizes represent uncompressed UTF-8 encoded text including system and user prompts. Figure 1 illustrates the composition of the SFT dataset. Its largest component is NVIDIA’s Llama-Nemotron-Post-Training-Dataset ( Bercovich et al. 2025 ) , which was used to train the Llama-3_1-Nemotron series. This series consistently demonstrates excellent performance on the LMArena ( Chiang et al. 2024 ) , suggesting it is a strong dataset for eliciting reasoning and aligning with human preferences. The reasoning components cover science, code, mathematics, and general chat, providing a foundation for analytical thinking across different domains. We also included the OpenHermes 2.5 dataset, which helps build general instruction-following capabilities and adherence to the system prompt. To enhance domain expertise, we incorporated the custom domain-specific Q&A datasets from both TdH25

Read the complete text