Benchmarking Multimodal Understanding and Complex Reasoning for ESG Tasks DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Information systems Document representa

Lei Zhang 1 , Xin Zhou 1 , Chaoyue He 1 , Di Wang 1 , Yi Wu 1 , Hong Xu 1 , Wei Liu 2 , Chunyan Miao 1 · 2025

Read Benchmarking Multimodal Understanding and Complex Reasoning for ESG Tasks DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Information systems Document representa by Lei Zhang 1 , Xin Zhou 1 , Chaoyue He 1 , Di Wang 1 , Yi Wu 1 , Hong Xu 1 , Wei Liu 2 , Chunyan Miao 1 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-18932.txt.

Introduction

Lei Zhang and co-authors introduce MMESGBench, the first benchmark for multimodal understanding and hard reasoning on ESG documents. Environmental, social and governance reports are long, varied in layout, and mix dense text with tables and figures. No dedicated test for this domain existed, though firms rely on these reports for sustainability and compliance checks.

The set holds 933 checked question and answer pairs drawn from forty-five ESG documents of seven types. Pairs were drafted by a multimodal model, screened by a second model, then validated by domain experts. Questions may rest on one page, span pages, or be unanswerable. Multimodal and retrieval-augmented models beat text-only baselines by much, above all on visual and cross-page tasks.

Catalog record

CREATORLei Zhang 1 , Xin Zhou 1 , Chaoyue He 1 , Di Wang 1 , Yi Wu 1 , Hong Xu 1 , Wei Liu 2 , Chunyan Miao 1
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE28 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-18932

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Benchmarking Multimodal Understanding and Complex Reasoning for ESG Tasks DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Information systems Document representation CCS: Information systems Language models Lei Zhang 1 , Xin Zhou 1 , Chaoyue He 1 , Di Wang 1 , Yi Wu 1 , Hong Xu 1 , Wei Liu 2 , Chunyan Miao 1 Abstract . Environmental, Social, and Governance (ESG) reports are essential for evaluating sustainability practices, ensuring regulatory compliance, and promoting financial transparency. However, these documents are often lengthy, structurally diverse, and multimodal, comprising dense text, structured tables, complex figures, and layout-dependent semantics. Existing AI systems often struggle to perform reliable document-level reasoning in such settings, and no dedicated benchmark currently exists in ESG domain. To fill the gap, we introduce MMESGBench , a first-of-its-kind benchmark dataset targeted to evaluate multimodal understanding and complex reasoning across structurally diverse and multi-source ESG documents. This dataset is constructed via a human-AI collaborative, multi-stage pipeline. First, a multimodal LLM generates candidate question-answer (QA) pairs by jointly interpreting rich textual, tabular, and visual information from layout-aware document pages. Second, an LLM verifies the semantic accuracy, completeness, and reasoning complexity of each QA pair. This automated process is followed by an expert-in-the-loop validation, where domain specialists validate and calibrate QA pairs to ensure quality, relevance, and diversity. MMESGBench comprises 933 validated QA pairs derived from 45 ESG documents, spanning across seven distinct document types and three major ESG source categories. Questions are categorized as single-page, cross-page, or unanswerable, with each accompanied by fine-grained multimodal evidence. Initial experiments validate that multimodal and retrieval-augmented models substantially outperform text-only baselines, particularly on visually grounded and cross-page tasks. MMESGBench is publicly available as an open-source dataset at https://github.com/Zhanglei1103/MMESGBench. 1. Introduction Understanding Environmental, Social, and Governance (ESG) factors has become increasingly critical in driving sustainable development, responsible investment, and global regulatory compliance ( krabbe2015aligning ; xiao2025impact ) . This trend has spurred the proliferation of standardized ESG reporting, yielding vast quantities of semantically rich, multimodal data that require interpretation by diverse stakeholders, including governments, corporates, and financial institutions. Concurrently, advancements in large language models (LLMs) and multimodal foundation models offer considerable potential for automating complex ESG-related tasks such as scoring, investment analysis, risk detection, and compliance monitoring ( zou2025esgreveal ; zhang2025optimizing ; wu2024susgen ) . However, the development and deployment of such automated solutions, which depend on reliable multimodal reasoning, are currently impeded by the scarcity of high-quality, task-specific datasets and benchmarks. One of the fundamental challenges in advancing AI for ESG applications lies in the inherent complexity of ESG documents, which can be summarized in the fellowing three interrelated dimensions. 1) Multi-source : ESG documents originate from a broad array of sources, including corporate ESG reports ( e.g. , annual ESG reports and CDP Climate Responses), ESG standards and frameworks ( e.g. , GRI ( gri2023standards ) , TCFD ( fsb2017tcfd ) , SASB ( sasb2023standards ) ), and publications from governments and international organizations ( e.g. , IPCC ( ipcc2023synthesis ) , SDGs ( un2015agenda2030 ) ). This diversity leads to substantial heterogeneity in document formats, structures, and semantic conventions. 2) Multimodality : ESG reports contain rich multimodal content, integrating narrative text, tabular data, analytical charts, visual figures, and layout-aware cues. Effective understanding requires models to jointly process and reason across these heterogeneous modalities. 3) Structural complexity : ESG documents often span hundreds to thousands of pages and exhibit nested structures, inter-referenced sections, and long-range dependencies, posing challenges for both local comprehension and global document-level reasoning. Despite the increasing demand for automated ESG analysis, existing LLM-based systems remain inadequate for such settings. To the best of our knowledge, no current benchmark captures the full extent of the multimodal and structural reasoning challenges posed by ESG documents. To address these challenges, we introduce MMESGBench , the first multimodal benchmark specifically designed for ESG document understanding and reasoning. MMESGBench is constructed through a human-AI collaborative pipeline that combines automatic generation, model-based validation, and expert refinement. Specifically, a multimodal LLM first generates candidate question-answer (QA) pairs by jointly interpreting textual content, tables, figures, and layout cues. These pairs are then filtered by an LLM verifier, which checks for factual correctness, completeness, and reasoning validity. Finally, ESG and AI experts review and refine the produced examples to ensure quality and domain relevance. This process enables scalable generation of fine-grained, evidence-grounded QA data with high fidelity. MMESGBench comprises 933 validated QA pairs across 45 ESG documents, covering seven document types and all three major ESG source categories. Each QA pair is labeled as either single-page, cross-page, or unanswerable, and is associated with detailed multimodal evidence, reflecting realistic document-level reasoning scenarios. MMESGBench is poised to benefit diverse applications within the ESG realm, ranging from ESG document understanding and reasoning to the fine-tuning of specialized ESG models and the implementation of advanced retrieval-augmented generation (RAG) methodologies. To showcase the capabilities of MMESGBench, we benchmark the ESG document reasoning performance of multiple representative models, including text-only LLMs, multimodal LLMs, and RAG pipelines. Results show that multimodal LLMs consistently outperform text-only models, particularly on layout-sensitive and visually grounded questions. In addition, RAG-based methods substantially improve performance on cross-page reasoning tasks by enhancing long-range information access. These findings highlight the importance of multimodal integration and retrieval-aware modeling for reliable ESG document understanding. Thus, MMESGBench is curated as a challenging and practical testbed for future research in this domain. In summary, MMESGBench bridges the gap between modern multimodal AI systems and the complex reasoning demands of ESG document understanding. Our contributions are mainly threefold: • We release MMESGBench , the first dataset for multimodal QA over real-world ESG documents, covering diverse source types, formats, and modalities. • We propose a quality-controlled human-AI collaborative QA generation and verification pipeline, enabling the creation of accurate, diverse, and evidence-grounded QA pairs at scale. • We establish a comprehensive benchmark with strong baseline results and actionable insights, enabling future research on multimodal document-level understanding. Figure 1 . The human-AI collaborative multi-stage QA generation framework. 2. Related Work 2.1. Document Understanding Benchmarks Document understanding has become a critical research area in multimodal AI, leading to the development of numerous benchmarks aimed at evaluating the capabilities of multimodal LLMs. Early efforts such as DocVQA ( mathew2021docvqa ) and InfographicsVQA ( mathew2022infographicvqa ) focus on single-page visual QA, while TableQA ( sun2020tableqa ) , ChartQA ( masry2022chartqa ) , and PlotQA ( methani2020plotqa ) address isolated modalities such as tables or charts. MP-DocVQA ( tito2023hierarchical ) extends DocVQA to multi-page documents but lacks cross-page and unanswerable questions. DUDE ( van2023document ) introduces a small number of cross-page questions, with relatively short documents and annotations collected via crowd-sourcing. SlideVQA ( tanaka2023slidevqa ) includes longer documents (around 20 pages) and cross-page questions but is limited by the slide-deck format and sparsity in reasoning graphs. FinanceBench ( islam2023financebench ) provides long-context, domain-specific financial reports with expert-designed questions, but its open-ended answer format requires manual evaluation. More recent efforts such as MMLongBench ( ma2025mmlongbench ) and LongDocURL ( deng2024longdocurl ) explore long-document multimodal QA and retrieval over web documents and academic papers, focusing on page-level localization and layout-aware reasoning.

Read the complete text