The Galaxy’s Guide to the Tokenizer: A Benchmark for Scientific Foundation Models

Sogol Sanjaripour, Michael J. Smith, Manuel Pérez-Carrasco, Juan Rafael Martínez-Galarza, Bahram Mobasher, Gabriela Canalizo · 2026

Read The Galaxy’s Guide to the Tokenizer: A Benchmark for Scientific Foundation Models by Sogol Sanjaripour, Michael J. Smith, Manuel Pérez-Carrasco, Juan Rafael Martínez-Galarza, Bahram Mobasher, Gabriela Canalizo (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-25610.txt.

Introduction

Before a transformer can learn from galaxy images, the images must be chopped into tokens, and that chopping shapes what the model can know. Sogol Sanjaripour and colleagues compare four tokenisation strategies in 2026 inside one shared astronomy transformer, using six hundred forty thousand galaxy images from a major sky survey. Two scores matter: how faithfully the tokens rebuild the image, and how well the learned features predict independently measured physics like mass and star formation rate.

No method won both contests. A flow based tokenizer rebuilt images best, while a vector quantised one gave the strongest features for physical properties. Two others preserved local shape detail better. Reconstruction quality and representation quality came apart, the authors stress, so tokenizers should be benchmarked on the science they serve, not on looks alone.

Catalog record

CREATORSogol Sanjaripour, Michael J. Smith, Manuel Pérez-Carrasco, Juan Rafael Martínez-Galarza, Bahram Mobasher, Gabriela Canalizo
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE28 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.25610

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

The Galaxy’s Guide to the Tokenizer: A Benchmark for Scientific Foundation Models Sogol Sanjaripour, Michael J. Smith, Manuel Pérez-Carrasco, Juan Rafael Martínez-Galarza, Bahram Mobasher, Gabriela Canalizo Abstract Tokenization is central to adapting scientific data for transformer-based foundation models, yet its impact on learned representations remains poorly understood. We compare four tokenization strategies—Affine, AIM, JetFormer, and VQ-VAE—within a unified transformer framework for astronomical imaging. Using 640 000 640\,000 galaxy images from the DESI Legacy Survey and a shared AstroPT backbone, we evaluate each method on reconstruction fidelity and prediction of physical properties. Our results reveal trade-offs across approaches. The flow-based JetFormer achieves higher reconstruction quality, while VQ-VAE yields strong probe performance for galaxy physical properties. Affine and AIM better preserve localized morphological information. We find that reconstruction and representation quality are decoupled, and no single method consistently performs best across the tasks considered here. By grounding our evaluation in independently measured physical quantities, we hope this study serves to highlight the potential of scientific data as a basis for constructing interpretable benchmarks for foundation models. Figure 1 : Comparison of image reconstruction quality using different tokenization strategies with a shared transformer backbone (AstroPT). The top row shows the input galaxy images, while the bottom row presents the reconstructions produced by each tokenizer. For the AIM and Affine models, both input and reconstructed images are shown in their patch-wise normalized representation. 1 From pixels to physics The Big Data Era of astronomy has arrived. Surveys such as Euclid ( Amiaux et al. 2012 ) , LSST ( Ivezić et al. 2019 ) , and DESI ( Dey et al. 2019 ) are producing imaging data at unprecedented scale, and transformer-based foundation models have emerged as a leading paradigm for extracting scientific insight ( Smith & Geach 2023 ; Leung & Bovy 2023 ; Koblischke & Bovy 2024 , e.g.) . Yet, while architectural choices have been widely studied, far less attention has been paid to tokenization —the process by which observations are converted into sequences that transformers can process—and how it shapes learned representations. Scientific domains provide a natural setting for this question, as observations are governed by known physical processes and accompanied by independently measured quantities that serve as objective ground truth. In astronomy, these include continuous properties such as redshift, stellar mass, and star formation rate, enabling physically grounded evaluation of representations beyond task-specific metrics ( Sanjaripour et al. 2024 ; Sanjaripour et al. 2025 ; Hemmati et al. 2026 ) . These questions also have practical relevance. Astronomical foundation models increasingly aim to unify multiple modalities within a shared framework ( The Multimodal Universe Collaboration 2024 ; Euclid Collaboration et al. 2025 ; Parker et al. 2025 , see e.g.) , and tokenization strategies that perform well for one modality may not generalize to others. Moreover, representations optimized for reconstruction may discard information relevant for scientific inference, a concern also observed in machine learning ( Black Forest Labs 2025 ; Mousavi et al. 2025 , e.g.,) . We systematically compare four tokenization strategies within a controlled framework: Affine (linear) projection; MLP-based ( El-Nouby et al. 2024 , AIM-like;) ; flow-based ( Tschannen et al. 2024 , Jetformer-like;) ; and VQ-VAE-based approaches ( Esser et al. 2021 ; Yu et al. 2021 ) . Our goal is to characterize how tokenization choices influence learned representations and their relation to scientific inference. 2 AstroPT backbone, tokenization strategies, and experimental setup We adopt AstroPT ( Smith et al. 2024 ) as our shared backbone: AstroPT is a decoder-only transformer ( Vaswani et al. 2017 ; Radford et al. 2019 ) trained autoregressively on sequences of astronomical data. We choose an autoregressive, decoder-only architecture because it is the dominant paradigm in current foundation models and imposes minimal inductive bias on the representation, making it a neutral testbed for comparing tokenization strategies. Our four tokenization strategies span a range of design choices: linear versus non-linear mappings, continuous versus discrete representations, and deterministic versus probabilistic decoding. We go into further detail about each strategy below. The Affine tokenizer defines the simplest possible mapping; a linear projection from image patches to the embedding space, with a second linear projection for reconstruction. This makes Affine the minimal baseline in our suite: a single learned linear map in and out, with all representational work left to the backbone. The AIM tokenizer replaces the linear projection with an MLP, following ( El-Nouby et al. 2024 ) and the original AstroPT framework. The MLP head gives the tokeniser a non-linear, learnable mapping from patch pixels to the embedding space, in principle letting it capture structure within a patch that a single linear projection cannot. Following ( El-Nouby et al. 2024 ) and ( Smith et al. 2024 ) , we apply z -score normalization to each Affine and AIM tokenizer patch to stabilize training. The JetFormer tokenizer , motivated by ( Tschannen et al. 2024 ) , produces continuous tokens through an invertible normalising flow. Inputs are uniformly dequantised, then a two-dimensional flow produces a latent image preserving all input information. Because the flow is invertible, the same module acts as both encoder and decoder and requires no separately pretrained tokeniser, instead being trained end-to-end to maximise the likelihood of the raw images. These continuous tokens are modelled with a Gaussian-mixture prediction head rather than a categorical distribution over a discrete codebook, placing JetFormer at the continuous end of our design axis. The VQ-VAE tokenizer discretizes images into a learned codebook of visual tokens ( van den Oord et al. 2017 ) . An encoder network maps image patches to a continuous latent space, where each latent vector is quantized to its nearest neighbor in a learned codebook. Following the literature, our VQ-VAE tokenizer is pre-trained on in-distribution galaxy imagery and then frozen during AstroPT training. We must note here that our tokenization comparison is not entirely ablative, and that this is a genuine methodological tension. However, we believe it is endemic to any realistic comparison of tokenizers: each strategy requires its own training procedure to be evaluated competitively, and forcing an entirely uniform regime would produce unrealistic and unfavorable setting for some methods. We therefore instead follow the training methodologies described in the literature as faithfully as possible. Data selection and pre-processing. We use a publicly available galaxy imaging dataset 1 1 1 https://hf.co/datasets/Smith42/galaxies derived from the DESI Legacy Surveys (DESI-LS) Data Release 8 ( Dey et al. 2019 ; Walmsley et al. 2023 ) , comprising 8.6 million galaxies, from which we select a representative subset of 640\,000 galaxies for training. DESI-LS was constructed to provide the targeting imagery for the DESI spectroscopic survey, and as such offers homogeneous, wide-field coverage well suited to large-scale representation learning. For each galaxy we extract 256\times 256 pixel g,r,z –band postage stamps at a resolution of 0.262~\text{arcsec\,pixel}^{-1} . The DR8 photometry is derived from model-based source fitting with the Tractor pipeline, which provides the magnitudes and colours we use as probe targets ( Dey et al. 2019 ) . Hyperparameters and pre-training routines. All experiments employ an 89 M-parameter AstroPT backbone 2 2 2 https://github.com/Smith42/astroPT with 12 transformer layers, model dimension 768 , and 12 attention heads, with a context length

Read the complete text