EO-Gym: A Multimodal, Interactive Environment for Earth Observation Agents

Sai Ma, Zhuang Li, Sichao Li, Xinyue Xu, Ruibiao Zhu, Tony Boston, John A. Taylor · 2026

Read EO-Gym: A Multimodal, Interactive Environment for Earth Observation Agents by Sai Ma, Zhuang Li, Sichao Li, Xinyue Xu, Ruibiao Zhu, Tony Boston, John A. Taylor (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-01250.txt.

Introduction

Sai Ma and six co-authors present EO-Gym, an interactive environment for Earth observation agents that mirrors how analysts actually work. Real analysis often requires expanding a region, retrieving historical observations, and switching between optical and radar sensors, while most benchmarks collapse the process into fixed-input, single-turn tasks. EO-Gym formulates analysis as a local geospatial workspace backed by a large indexed file collection and dozens of specialized tools.

On that environment they build a benchmark dataset with thousands of trajectories and reasoning steps grounded in public datasets and satellite imagery. Ten vision-language models still struggle with interactive reasoning, especially temporal and cross-modal workflows. A fine-tuned small model improves the main pass rate substantially, providing a reference baseline.

Catalog record

CREATORSai Ma, Zhuang Li, Sichao Li, Xinyue Xu, Ruibiao Zhu, Tony Boston, John A. Taylor
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE109 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.01250

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

EO-Gym: A Multimodal, Interactive Environment for Earth Observation Agents Sai Ma, Zhuang Li, Sichao Li, Xinyue Xu, Ruibiao Zhu, Tony Boston, John A. Taylor Abstract Earth Observation (EO) analysis is inherently interactive: resolving uncertainty often requires expanding the region of interest, retrieving historical observations, and switching across sensors such as optical and Synthetic Aperture Radar. However, most EO benchmarks collapse this process into fixed-input, single-turn tasks. To address this gap, we present EO-Gym , a controlled executable framework for multimodal, tool-using EO agents that formulates EO analysis as a Gymnasium-style local geospatial workspace backed by more than 660 660 k multimodal files indexed by location, time, and sensor type, with 35 35 EO-specialized tools spanning six task families. Built on this environment, we construct EO-Gym-Data , a benchmark of 9,078 9{,}078 trajectories and 34,604 34{,}604 reasoning steps, and grounded in eight public EO datasets together with Landsat and Sentinel-2 imagery. Evaluating 10 10 open and closed VLMs shows that strong general-purpose models still struggle with interactive EO reasoning, especially on temporal and cross-modal workflows. As a reference baseline, EO-Gym-4B , obtained by fine-tuning Qwen3-VL-4B-Instruct on EO-Gym-Data , improves overall Pass@3 from 0.49 0.49 to 0.74 0.74 under the main evaluation setting. EO-Gym provides a reproducible environment for interactive EO agents, operationalizing EO as an evidence-gathering problem that requires planning across geospatial, temporal, and sensing modality. Code and data are available at https://huggingface.co/datasets/paperuploadacount/EO-Gym . 1 Introduction Earth Observation (EO) underpins planetary-scale monitoring, disaster response, and environmental change analysis. However, many EO questions cannot be resolved from fixed observations. Analysts can reduce uncertainty by expanding the region of interest, retrieving historical observations, changing spatial scale, or switching sensing modalities, for example, from optical imagery to Synthetic Aperture Radar (SAR) during periods of cloud cover or at night [ 1 , 2 , 3 ] . Moreover, because decades of prior work remain siloed within current operational frameworks, accessing this historical data requires systems capable of automatically interfacing with disparate legacy architectures [ 4 ] . Consequently, EO analysis is not merely a perception problem, but an interactive evidence-acquisition process that requires stateful tools to navigate space, reason over time, switch modalities, and synthesize heterogeneous observations for geospatial decision-making. Recent progress in Vision Language Models (VLMs) and tool-using agents presents a promising avenue for automating these analytical workflows [ 5 , 6 , 7 , 8 , 9 ] . However, the dominant large-scale evaluation paradigms in EO remain fundamentally static, as existing corpora are restricted to fixed-input comprehension rather than dynamic, sequential evidence gathering [ 10 , 11 , 12 , 13 , 14 ] . While emerging EO agent initiatives have started to address this limitation [ 15 , 16 , 17 , 18 , 19 ] , the community critically lacks a controlled and reproducible multimodal benchmark. To effectively support the interactive, cross-modal evidence acquisition required for planetary-scale analysis, advancing this domain requires a comprehensive framework. This environment must simultaneously provide executable interactions across space and time, as well as scalable trajectory generation for training and evaluation. To address these requirements, we introduce EO-Gym , a Gymnasium-style workspace designed to directly support executable interactions across EO tasks [ 20 ] . Backed by over 660 k multimodal indexed files and 35 specialized tools, this environment allows agents to perform complex EO reasoning by dynamically accessing data to reduce uncertainty. Utilizing this framework, we ensure scalable trajectory generation by constructing EO-Gym-Data . This comprehensive benchmark comprises 9,078 trajectories derived from diverse satellite sensors, spanning true-color, multispectral, and SAR imagery [ 4 , 21 , 22 , 23 , 24 , 25 , 26 , 27 , 28 ] . Moving beyond static predictions, each trajectory moves beyond static input by linking tool calls, observations, and reasoning into an explicit path for interactive temporal, geospatial, and cross-modal evidence gathering. To assess whether current models can gather temporal, geospatial, and cross-modal evidence directly from an EO environment, we evaluate 10 open and closed VLMs in EO-Gym and find that general-purpose models struggle with these dynamic workflows. We then fine-tune Qwen3-VL-4B-Instruct to obtain EO-Gym-4B , which improves Pass@3 from 0.49 to 0.74 under the Verified/Skill/Simple protocol. This gain shows that training in an interactive EO environment substantially improves tool use and core EO reasoning. In summary, our contributions are threefold: • Controlled executable EO environment. We introduce EO-Gym , a multimodal geospatial workspace that supports spatial navigation, temporal retrieval, cross-modal switching, and EO-specific tool use in a fully executable local setting. • EO multimodal trajectory benchmark. We construct EO-Gym-Data , a benchmark of 9{,}078 EO tool-use trajectories across six EO tasks, grounded in eight public EO datasets and the EO-Gym data-gathering execution space. • Unified benchmark and reference baseline. We evaluate 10 open and closed VLMs and provide EO-Gym-4B as a strong EO-specialized reference baseline. 2 Related Work General benchmarks for tool-using agents. Agent benchmarking increasingly relies on executable environments rather than static prompts. General tool-use benchmarks such as ToolBench , GAIA , and GTA evaluate tool selection, invocation, and coordination across realistic tasks [ 7 , 29 , 30 ] . Recent Gym-style suites extend this paradigm: MLGym frames AI research as episodic agent tasks, while R2E-Gym provides executable software-engineering environments with verifiable outcomes [ 31 , 32 ] . Interactive benchmarks such as VisualWebArena , WebArena , OSWorld , and FormFactory evaluate sequential behavior in web, GUI, form-filling, and computer-use settings [ 8 , 33 , 9 , 34 ] . Related simulators such as AerialGym support Gym-like aerial perception and control, but target embodied UAV learning rather than satellite catalog reasoning [ 35 ] . Collectively, these works show that realistic agent evaluation requires stateful, executable interaction. However, they do not address EO, where reasoning requires evidence acquisition across space, time, and modality, and therefore lacks the specialized tools and environmental grounding needed for EO workflows. From static EO benchmarks to interactive agents. The EO community has developed extensive datasets for perception and vision-language learning, including xView , DOTA , DIOR , FAIR1M , SARDet-100K , and VRSBench [ 36 , 37 , 38 , 39 , 40 , 12 ] . Recent multimodal resources such as EarthScape , EarthGPT , SkyEyeGPT , and EarthDial further expand EO supervision across multi-scale geospatial analysis, multi-sensor comprehension, and dialogue-style learning [ 41 , 10 , 11 , 14 ] . In parallel, infrastructure and representation-learning work has improved reusable EO data access and fixed-input reasoning: TorchGeo supports geospatial datasets, samplers, transforms, and pretrained models; CSP learns geospatial-visual representations from image-location contrast; and satellite time-series methods model temporal attention, panoptic sequence structure, and explainability for crop and parcel analysis [ 42 , 43 , 44 , 45 , 46 ] . These resources are important for pretraining and specialist EO perception, but remain centered on fixed-input prediction rather than autonomous environmental interaction. Recent agentic EO systems address this gap along complementary axes. Kao et al. 2025 study EO agents through code generation and API execution. ThinkGeo introduces structured tool use and multi-step planning [ 16 ] . Earth-Bench cover optical and spectral tools with trajectory-level assessment [ 17 ] . OpenEarthAgent emphasizes supervised trajectory learning and deterministic replay validation [ 18 ] , while

Read the complete text