Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements Conference: ACM International Conference on Information and Knowledge Management; Submission Draft; CCS: Computing methodologies Natural language generation CCS: Information systems Data management systems

Ryo Kanazawa, Koyo Hidaka, Teppei Miyamoto, Takayuki Kato, Tomoki Ando, Chenguang Wang, Dayuan Jiang, Naofumi Fujita, Shuhei Saitoh, Atomu Kondo, Koki Arakawa, Daiho Nishioka · 2026

Read Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements Conference: ACM International Conference on Information and Knowledge Management; Submission Draft; CCS: Computing methodologies Natural language generation CCS: Information systems Data management systems by Ryo Kanazawa, Koyo Hidaka, Teppei Miyamoto, Takayuki Kato, Tomoki Ando, Chenguang Wang, Dayuan Jiang, Naofumi Fujita, Shuhei Saitoh, Atomu Kondo, Koki Arakawa, Daiho Nishioka (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-22079.txt.

Introduction

Ryo Kanazawa and co-authors present Ishigaki-IDS-Bench, a benchmark for generating Information Delivery Specification documents from building information requirements. Building Information Modeling projects use IDS to formalize requirements in machine-checkable XML, but authoring IDS conditions grounded in Industry Foundation Classes vocabulary demands expertise in concepts, validation tools, and property sets. The benchmark scores facet-level macro-F1 and content fidelity. Across 10 language models in zero-shot tests, GPT-5.5 reaches the highest Facet F1 at 65.6 percent, while Claude Opus 4.5 reaches the highest Content pass rate at only 33.1 percent.

Catalog record

CREATORRyo Kanazawa, Koyo Hidaka, Teppei Miyamoto, Takayuki Kato, Tomoki Ando, Chenguang Wang, Dayuan Jiang, Naofumi Fujita, Shuhei Saitoh, Atomu Kondo, Koki Arakawa, Daiho Nishioka
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE31552
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.22079

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements Conference: ACM International Conference on Information and Knowledge Management; Submission Draft; CCS: Computing methodologies Natural language generation CCS: Information systems Data management systems Ryo Kanazawa, Koyo Hidaka, Teppei Miyamoto, Takayuki Kato, Tomoki Ando, Chenguang Wang, Dayuan Jiang, Naofumi Fujita, Shuhei Saitoh, Atomu Kondo, Koki Arakawa, Daiho Nishioka Abstract . Large language models (LLMs) are widely used to generate structured outputs such as JSON, SQL, and code, yet public resources remain limited for evaluating generation that must simultaneously satisfy industry-standard XML and domain vocabulary constraints. This paper presents Ishigaki-IDS-Bench, a benchmark for evaluating the ability to generate Information Delivery Specification (IDS) XML from Building Information Modeling (BIM) information requirements. The benchmark contains 166 BIM/IDS expert-authored and verified examples created by expanding 83 practical scenarios into Japanese and English, corresponding gold IDS files, and metadata for input format, language, turn setting, IFC version, and construction domain. Its evaluation combines IDSAuditTool-based Processability, Structure, and Content audits with content-agreement evaluation against gold IDS files. In zero-shot evaluation over 10 LLMs, the best model reaches 65.6% macro F1 for content agreement, while only 27.7% of outputs pass the Content audit. These results show that current LLMs can express part of the information requirements as IDS, but still struggle to stably generate XML that satisfies the IDS standard and IFC vocabulary constraints. Ishigaki-IDS-Bench supports comparative evaluation, failure analysis, and the development of constrained structured generation methods that conform to domain standards. We release the evaluation scripts and benchmark data under the CC BY 4.0 license on GitHub 1 1 1 https://github.com/onestruction/Ishigaki-IDS-Bench.git and Hugging Face 2 2 2 https://huggingface.co/datasets/ONESTRUCTION/Ishigaki-IDS-Bench . 1. Introduction Large language models (LLMs) are widely used to generate structured outputs such as JSON, SQL, and code ( 28 ; 1 ) . In practical domains, however, structured outputs are not sufficient merely by being syntactically valid. They must simultaneously conform to industry-standard data formats, domain-specific vocabularies, version constraints, and audit results from external validation tools. Evaluation of such domain-standard structured generation remains less mature than evaluation for general-purpose JSON or SQL generation. This work studies this problem through Information Delivery Specification (IDS), a machine-readable format for describing information requirements in the architecture, engineering, and construction domain. Building Information Modeling (BIM) is an information foundation for treating buildings and infrastructure as digital information models that include not only geometry, but also element types, materials, performance information, and management information ( 6 ) . Industry Foundation Classes (IFC) is an international standard data format for sharing BIM models across different software systems ( 11 ) . In IFC, building elements such as walls, columns, doors, and pipes are represented with standardized vocabularies, and information such as names, dimensions, materials, and fire ratings can be attached to those elements. IDS is an XML-based standard specification for describing which kinds of elements in such IFC models should have which information under which conditions ( 3 ) . In other words, whereas IFC provides a shared vocabulary for representing BIM models, IDS describes information requirements that those models should satisfy in a checkable form. IDS generation involves multiple layers of constraints beyond ordinary syntax-constrained generation. For example, when generating IDS from the requirement “all walls must have a fire rating, and the value must be one of EI30, EI60, or EI90,” a model must not only output a valid XML structure. It must also map “wall” to the appropriate IFC standard class, express “fire rating” as an appropriate IFC information item, and describe the allowed-value constraint within an IDS checking unit. IDS generation is therefore a constrained structured generation task that simultaneously involves syntax and type constraints, standard conformance, mapping to specialized vocabularies, expression of value constraints, and semantic correspondence with the input document. In practice, information requirements are not always prepared as IDS in advance. They are often written as natural-language specifications, tabular checklists, employer’s information requirements, or meeting records. Creating IDS from such documents requires both expertise in BIM, IFC, and IDS and the interpretive ability to understand the intent of the input document. LLMs could support this conversion process and reduce expert workload. At the same time, public benchmarks remain insufficient for comparably evaluating how accurately LLMs can generate IDS from practical documents and for identifying which input types or constraints cause failures. In particular, an integrated evaluation framework is needed for the formal validity of generated IDS, conformance to the IDS standard, consistency with IFC vocabularies, and content agreement with the input document. This paper proposes Ishigaki-IDS-Bench, a benchmark for evaluating the ability to generate IDS from practical documents. Ishigaki-IDS-Bench consists of 166 examples created by expanding 83 practical scenarios into Japanese and English. Each example is authored and verified by BIM/IDS experts and has a corresponding gold IDS. Each example is also annotated with metadata such as input format, language, turn setting, target IFC version, and construction domain. This enables analysis not only by a single aggregate score, but also by performance differences between natural-language and tabular inputs, Japanese and English, single-turn and multi-turn settings, IFC versions, and construction domains. We further design a two-stage evaluation protocol for IDS generation. In the first stage, IDSAuditTool is used to evaluate whether the generated result can be extracted as auditable IDS, whether it conforms to the IDS schema, and whether it satisfies the IDS standard and IFC vocabulary constraints. In the second stage, content agreement between the generated result and the gold IDS is evaluated, capturing failures where an IDS is formally valid but differs from the input document. This two-stage evaluation enables more detailed analysis of the ability to generate practically useful IDS, not merely XML. In zero-shot evaluation over 10 LLMs, the best model reaches 65.6% macro F1 for content agreement, while only 27.7% of outputs pass the Content audit. This result shows that current LLMs can express part of the information requirements as IDS, but still face substantial challenges in stably generating outputs that satisfy the IDS standard and IFC vocabulary constraints. Ishigaki-IDS-Bench is released under the CC BY 4.0 license. The contributions of this work are as follows. • We construct an IDS generation benchmark based on practical use cases. Ishigaki-IDS-Bench includes 166 expert-authored and verified examples, 83 scenarios, corresponding gold IDS files, and multifaceted metadata. • We provide a two-stage evaluation protocol that combines formal validity evaluation using IDSAuditTool with facet-level content-agreement evaluation against gold IDS files. • We report zero-shot baselines over 10 LLMs and analyze success cases and failure tendencies in IDS generation. 2. Related Work Schema-constrained and grammar-constrained generation have been developed as methods for improving the formal validity of structured outputs generated by LLMs. Representative approaches include incorporating finite-state machines or context-free grammars into decoding ( 28 ; 10 ) , co-designing inference and grammar engines as in XGrammar ( 5 ) , correcting distributional distortion caused by grammar constraints through Grammar-Aligned Decoding ( 23 ) , and using grammar masking for DSL generation ( 20 ) . These studies have mainly targeted outputs with general-purpose schemas or explicit syntactic constraints, such as JSON, SQL, code, and DSLs. By contrast, a domain-specific XML standard such as IDS requires not only syntactic validity, but also consistency

Read the complete text