Mars-Bench: A Benchmark for Evaluating Foundation Models for Mars Science Tasks

Mirali Purohit, Bimal Gajera, Vatsal Malaviya, Irish Mehta, Kunal Kasodekar, Jacob Adler, Steven Lu, Umaa Rebbapragada, Hannah Kerner · 2025

Read Mars-Bench: A Benchmark for Evaluating Foundation Models for Mars Science Tasks by Mirali Purohit, Bimal Gajera, Vatsal Malaviya, Irish Mehta, Kunal Kasodekar, Jacob Adler, Steven Lu, Umaa Rebbapragada, Hannah Kerner (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-24010.txt.

Introduction

Mirali Purohit and eight co-authors introduce Mars-Bench, the first standardized benchmark for machine learning on Mars science tasks. Foundation models have transformed fields with plentiful labeled benchmarks, but Mars research lacks shared test sets, which has slowed its own foundation models. Mars-Bench fills the gap with twenty datasets drawn from orbital and surface imagery.

The tasks span classification, segmentation, and object detection, targeting the features geologists care about: craters, cones, boulders, and frost. All datasets ship in ready-to-use form with standardized splits. The authors baseline them with models pretrained on natural images, on Earth satellite data, and with leading vision-language models. Across the analyses, Mars-specific pretraining looks advantageous over general-domain starting points. The benchmark, data, and code are public, giving the community a common yardstick. The authors hope it sparks the same cycle of progress that benchmarks enabled for Earth observation.

Catalog record

CREATORMirali Purohit, Bimal Gajera, Vatsal Malaviya, Irish Mehta, Kunal Kasodekar, Jacob Adler, Steven Lu, Umaa Rebbapragada, Hannah Kerner
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE105 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-24010

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Mars-Bench: A Benchmark for Evaluating Foundation Models for Mars Science Tasks Mirali Purohit, Bimal Gajera, Vatsal Malaviya, Irish Mehta, Kunal Kasodekar, Jacob Adler, Steven Lu, Umaa Rebbapragada, Hannah Kerner Abstract Foundation models have enabled rapid progress across many specialized domains by leveraging large-scale pre-training on unlabeled data, demonstrating strong generalization to a variety of downstream tasks. While such models have gained significant attention in fields like Earth Observation, their application to Mars science remains limited. A key enabler of progress in other domains has been the availability of standardized benchmarks that support systematic evaluation. In contrast, Mars science lacks such benchmarks and standardized evaluation frameworks, which have limited progress toward developing foundation models for Martian tasks. To address this gap, we introduce Mars-Bench, the first benchmark designed to systematically evaluate models across a broad range of Mars-related tasks using both orbital and surface imagery. Mars-Bench comprises 20 datasets spanning classification, segmentation, and object detection, focused on key geologic features such as craters, cones, boulders, and frost. We provide standardized, ready-to-use datasets and baseline evaluations using models pre-trained on natural images, Earth satellite data, and state-of-the-art vision-language models. Results from all analyses suggest that Mars-specific foundation models may offer advantages over general-domain counterparts, motivating further exploration of domain-adapted pre-training. Mars-Bench aims to establish a standardized foundation for developing and comparing machine learning models for Mars science. Our data, models, and code are available at: https://mars-bench.github.io/ . \faIconFromMacro{faEnvelope} \faIconFromMacro{faEnvelope} footnotetext: Corresponding Author: [email protected] * * footnotetext: Equal Contribution 1 Introduction Over the past few years, foundation models have revolutionized specialized domains such as medical imaging moor2023med ; parmar022boxbart , Earth Observation (EO) klemmer2023satclip ; reed2023scale ; astruc2024anysat , law colombo2024saullm ; cui2023chatlaw , and astronomy lanusse2023astroclip ; nguyen2023astrollama ; slijepcevic2024radio . These models, pre-trained on large and diverse datasets, offer strong generalization capabilities and enable efficient fine-tuning on downstream tasks with minimal data. The EO community has embraced foundation models in the last 3-4 years, with an explosion of methods, datasets, and benchmarks aimed at improving performance across a wide range of geospatial tasks. The key driver of progress in these domains has been the development of high-quality, standardized benchmarks. For example, BigBio fries2022bigbio and MIMIC-IV johnson2023mimic have accelerated model advancements by providing consistent evaluation protocols for medical applications. Benchmarks like Geo-Bench lacoste2023geo and PANGAEA marsocci2024pangaea have accelerated progress in EO applications by providing a suite of standardized classification and segmentation tasks for evaluating geospatial foundation models. Geo-Bench enables model developers to assess generalization across diverse data sources and use cases, creating a pathway for systematic progress. However, no such benchmark exists for Martian applications. Machine learning research for Mars science applications thus lags behind other science domains azari2020integrating . Although recent studies have presented machine learning solutions for a range of Martian applications, including crater detection malvi2023automated ; zheng2025automatic ; delatte2019segmentation , landmark classification wilhelm2020domars16k ; wagstaff2021mars , and cone segmentation purohit2024conequest ; yang2024mapping , these solutions and datasets lack standardization and interoperability. This results in task-specific models or datasets that cannot be easily evaluated as downstream tasks for foundation models or other machine learning advances. This results in limited evaluation of proposed Mars foundation model approaches on 1-2 downstream tasks, limiting the ability to assess model generalization or robustness vincent2024clover ; ward2022improving ; wang2023semi ; goh2023self ; purohit2024investigating . (a) Classification (b) Object Detection (c) Segmentation Figure 1 : Representative samples from selected Mars-Bench datasets, from all three task categories. This gap is particularly surprising given the richness of available Mars data. Orbiters such as the Mars Reconnaissance Orbiter (MRO) zurek2007overview and Mars Odyssey have captured millions of images over the last 20-25 years, while surface rovers like Curiosity and Perseverance have amassed petabytes of high-resolution images. These datasets offer immense potential to study critical questions of planetary science, such as the past presence of water on Mars and the planet’s habitability. Yet, the full value of these datasets remains untapped by the ML community due to their lack of standardization, incomplete documentation, and inconsistent formatting for ML workflows. We introduce Mars-Bench, the first comprehensive benchmark designed to systematically evaluate machine learning models across a diverse set of Mars-related tasks using both orbital and surface imagery. To create this benchmark, we curated and revamped existing datasets, performing quality checks and corrections where necessary and standardizing them in a unified, ML-ready format. The goal of Mars-Bench is to provide a common framework to assess and compare the performance of foundation models on Martian data, facilitating reproducibility and accelerating scientific discovery in planetary science. Our key contributions are as follows: • Diverse task coverage: Mars-Bench includes 20 datasets, summarized in Table 1 , spanning three task types: classification, segmentation, and object detection. We also provide a few-shot and partitioned versions of each dataset for evaluation under varying training sample sizes. • Scientific relevance: Mars-Bench covers a wide range of geologic features commonly studied in Mars science, including craters, cones, boulders, landslides, dust devils, atmospheric dust, etc. These tasks reflect real scientific use cases relevant to planetary scientists and geologists, who co-developed the Mars-Bench. Samples from few Mars-Bench datasets are shown in Figure 1 . • Comprehensive evaluation: Since no standardized pre-trained model exists for Mars data, we benchmarked performance using ImageNet-pretrained models under different training settings. We analyzed model behavior with different training set sizes. We also evaluated Mars-Bench using pre-trained EO models as well as proprietary vision-language models, including Gemini and GPT. • Code, reproducibility, and baseline models: We release full code support for all experiments in this paper, along with tools for dataset handling and results visualization. To facilitate community adoption and reproducibility, we also provide well-documented guidelines and publicly release all baseline models evaluated on Mars-Bench. These models can serve as strong starting points for future applications; for example, generating initial global maps of specific geologic features (e.g., cones), which experts can later refine with minimal annotation effort. 2 Related Work Over the past decade, evaluation benchmarks have played a fundamental role in identifying the limitations of existing foundation models, steering their progress in natural language processing (NLP) and computer vision (CV). For instance, general-purpose natural language understanding (NLU) benchmarks wang2019superglue ; wang2024mmlupro ; srivastava2023beyond have facilitated the development of large language models (LLMs) such as GPT brown2020language , LLaMA touvron2023llama , and Gemini team2023gemini . Even in specialized domains, including medical parmar022boxbart ; fries2022bigbio ; johnson2023mimic , legal fei2023lawbench ; guha2023legalbench , scientific discovery majumder2024discoverybench ; chen2024scienceagentbench , security bhusal2024secure , and finance islam2023financebench , various benchmarks have driven progress in building domain-specific foundation models. Thus, development of quality evaluation benchmarks is necessary for building better foundation models. In the remote sensing domain, Geo-Bench lacoste2023geo has defined standardized evaluation protocols for a broad set of EO tasks and has quickly become a de facto benchmark. Since its release, Geo-Bench has been used to evaluate most foundation models proposed for EO over the past two years, enabling consistent comparisons across models. Other notable efforts include SustainBench yeh2021sustainbench , which targets seven sustainable development goals, AiTLAS dimitrovski2023current , which aggregates 22 EO datasets focused solely on classification tasks, and PANGAEA marsocci2024pangaea , which includes 11 evaluation datasets covering diverse satellite sensors. Despite substantial progress in other domains toward foundation models

Read the complete text