GeoR-Bench: Evaluating Geoscience Visual Reasoning
Yushuo Zheng, Zicheng Zhang, Huiyu Duan, Chunyi Li, Zijian Chen, Ziheng Jia, Yue Shi, Ke Gu, Xiongkuo Min, Guangtao Zhai · 2026
Read GeoR-Bench: Evaluating Geoscience Visual Reasoning by Yushuo Zheng, Zicheng Zhang, Huiyu Duan, Chunyi Li, Zijian Chen, Ziheng Jia, Yue Shi, Ke Gu, Xiongkuo Min, Guangtao Zhai (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-11541.txt.
Introduction
Zheng and nine colleagues present GeoR-Bench, a benchmark for geoscience visual reasoning through reasoning-led visual editing tasks. Geoscience tools should grasp, reason about and predict earth system change for disaster response, yet current benchmarks stay narrow. GeoR-Bench holds 440 samples across six geoscience fields and 24 task types, covering earth observation images plus maps and diagrams. Outputs score on reasoning, consistency and quality. Of 21 closed- and open-source models tested, the best reaches only 42 point 7 percent exact accuracy, while the best open-source models reach only 10 point 3 percent.
Catalog record
| CREATOR | Yushuo Zheng, Zicheng Zhang, Huiyu Duan, Chunyi Li, Zijian Chen, Ziheng Jia, Yue Shi, Ke Gu, Xiongkuo Min, Guangtao Zhai |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 34458 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.11541
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
GeoR-Bench: Evaluating Geoscience Visual Reasoning Yushuo Zheng, Zicheng Zhang, Huiyu Duan, Chunyi Li, Zijian Chen, Ziheng Jia, Yue Shi, Ke Gu, Xiongkuo Min, Guangtao Zhai Abstract Geoscience intelligence is expected to understand, reason about, and predict earth system changes to support human decision-making in critical domains such as disaster response, climate adaptation and environmental protection. Although current research has shown promising progress on specific geoscience tasks, such as remote sensing interpretation, geographic question-answering, existing benchmarks remain largely task-specific which failing to capture the open-ended real world geoscience problems. As a result, it remains unclear how far current AI systems are from achieving genuine geoscience intelligence. To address this gap, we present GeoR-Bench , a Bench mark for evaluating Geo science visual R easoning through reasoning informed visual editing tasks. GeoR-Bench contains 440 curated samples spanning 6 geoscience categories and 24 task types, covering earth observation imagery and structured scientific representations such as maps and diagrams. We evaluate outputs along three dimensions, including reasoning, consistency, and quality. Benchmark results of 21 closed- and open-source multimodal models reveal that geoscience reasoning remains a critical bottleneck. The highest-performing model achieves 42.7% overall strict accuracy, while the best open-source models only get 10.3%. Notably, the visual consistency and image quality of the outputs frequently surpass their scientific accuracy. Ultimately, these findings indicate that current models generate superficially plausible results but fail to capture underlying earth science processes. $\dagger$ $\dagger$ footnotetext: Project Lead. 1 Introduction Geoscience intelligence is the capacity of an AI system to understand, reason about, and predict Earth-system changes in ways that can inform human decision-making in domains such as disaster response, climate adaptation, and environmental protection. Recent multimodal models have shown encouraging progress on narrow geoscience tasks, including remote sensing interpretation and geographic question answering. Yet a central question remains open, i.e. , how do we evaluate geoscience intelligence in a way that reflects the open-ended, visually grounded nature of real-world Earth-system problems? Existing benchmarks remain largely task-specific and measure isolated recognition or captioning abilities, leaving it unclear how far current models are from genuine geoscience intelligence. In practice, geoscience reasoning must operate over intermediate visual artifacts, including satellite images, maps, cross-sections, and scientific diagrams, and produce outputs that are not merely plausible but scientifically valid. A principled evaluation of geoscience intelligence must therefore probe whether a model can convert Earth-system understanding into a correct, visually grounded generation, rather than only verifying surface-level visual realism or textual recall. To make geoscience intelligence measurable, we propose to evaluate it through reasoning-informed visual editing: given an input image and a task instruction, a model must infer the relevant Earth-system state and produce an output image that reflects that inference. This setting is challenging because correctness depends on reasoning about process, geometry, scale, topology, and domain conventions at the same time; predicting delta growth requires understanding sediment transport and channel branching, modifying a cyclone image requires respecting circulation structure, and completing a tectonic diagram requires correct labels and an interpretable layout. Pure recognition benchmarks do not force a model to externalize such understanding in a testable visual artifact, while generic generation benchmarks reward plausibility or instruction following without verifying scientific validity. Reasoning-informed visual editing closes this gap: the edited image becomes direct evidence of whether the model can turn geoscience reasoning into a grounded visual generation, making editing the measurement interface through which geoscience intelligence becomes observable. Figure 1: GeoR-Bench evaluates geoscience intelligence via reasoning-informed visual editing across six categories: geomorphology, GIS & spatial geometry, crustal science, atmosphere, cryosphere, and hydrology. Each task requires inferring the underlying Earth-system process from an input image and producing a scientifically valid output. However, existing benchmarks do not directly evaluate this capability. General image-editing benchmarks emphasize natural-image manipulation, instruction following, and broad visual reasoning 26 . More reasoning-aware suites such as KRIS-Bench and RISEBench move beyond low-level editing quality, but they still focus primarily on natural scenes and generic structural reasoning rather than scientific correctness ( 22 ; 24 ) . Scientific and multimodal knowledge benchmarks such as MMMU evaluate expert-level understanding ( 27 ) , while scientific illustration benchmarks test technical drawing ability ( 1 ) . These works are important, but they rarely require a model to express domain understanding through a visually grounded geoscience generation. Therefore, in this work, we address this gap by evaluating whether multimodal models can convert geoscience understanding into a correct edited or generated image. GeoR-Bench is built around this principle, which contains 440 samples spanning 6 categories and 24 task types, with 4 task types per category. The benchmark covers geomorphology, hydrology, atmosphere and ocean dynamics, cryosphere science, GIS and spatial geometry, and crustal science. Its tasks span both real-world Earth observation imagery and formal scientific representations, including river delta evolution, wetland fluctuation, cyclone generation under Coriolis constraints, snowline prediction, cross-view geospatial reconstruction, earthquake epicenter triangulation, and geological labeling. This breadth is deliberate: geoscience visual reasoning must operate across natural imagery, spatial products, and structured scientific diagrams rather than within a single visual modality. Figure 1 illustrates the benchmark scope. To evaluate model outputs, we use a three part protocol to judge the generated images: reasoning , consistency , and quality . Reasoning evaluates whether the generated output reflects the intended physical, spatial, or process-based generation by jointly comparing the input image, ground-truth output, model prediction, and the task-specific rubric. Consistency evaluates whether the generated output preserves the required visual relationship between the input image and the target output. Quality evaluates whether the generated image is visually usable as an artifact. We also report accuracy , which counts a sample as correct only when all three dimensions are satisfied simultaneously. These metrics support claims about benchmark-defined geoscience visusal resosing ability. Our contributions are as follows: • We articulate a comprehensive evaluation principle for geoscience intelligence: it should be tested through reasoning-informed visual editing tasks that require models to convert Earth-system understanding into scientifically correct visual outputs. • We introduce GeoR-Bench, a benchmark of 440 curated samples across 6 categories and 24 task types. Each sample follows a standardized format, each task has a dedicated judging rubric, and all evaluative claims are clearly documented. • We provide experiments and analysis showing that current 21 multi-modal models remain substantially weaker at geoscience reasoning than at preserving visual appearance or producing readable outputs, yielding concrete evidence about the present limits of multimodal geoscience visual reasoning. 2 Related Work 2.1 Large Language Models and Multimodal Benchmarks Recent progress in multimodal large models has produced a new generation of evaluation benchmarks for image editing, instruction-following generation, and visually grounded reasoning. General-purpose editing benchmarks such as ImgEdit assess instruction alignment, local content preservation, and visual plausibility across diverse natural scenes ( 26 ) , but strong performance there does not imply fidelity to deeper causal or domain constraints. Reasoning-oriented benchmarks including KRIS-Bench and RISEBench show that visually attractive edits often violate temporal, spatial, causal, or logical requirements ( 22 ; 24 ) , while WorldGenBench tests whether text-to-image systems convert implicit world knowledge into faithful outputs ( 28 ) , and broader knowledge benchmarks such as MMMU and SrdBench highlight persistent difficulty with expert knowledge and technical visual conventions ( 27 ; 1 ) . Yet