Project Auto-World: Towards Automated Benchmarking of Neural Relational Reasoners

Anirban Das Joanne Boisson Irtaza Khalid Sumita Garai Steven Schockaert · 2026

Read Project Auto-World: Towards Automated Benchmarking of Neural Relational Reasoners by Anirban Das Joanne Boisson Irtaza Khalid Sumita Garai Steven Schockaert (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-24965.txt.

Description

Anirban Das, Joanne Boisson, and three colleagues automate the benchmarking of neural models. Reasoning over relational structures is hard for neural models, above all when test problems exceed training difficulty, and nobody knows what makes an instance hard. Their Project Auto-World attacks the evaluation problem itself. A world is parametrised by Datalog rules, and an Edge Transformer serves as the model under test. Two search strategies, LLM driven evolutionary search and autonomous agentic search, hunt for samplers that generate hard instances.

The discovered samplers produce hard problem sets. Training the Edge Transformer on this generated data improves it, and the gain generalises to further data changes. The same machinery runs on worlds proposed by language models. Das and colleagues present this as a step toward self directed research in the field. Benchmarks stop being artefacts written by hand; they become opponents that grow harder as the reasoner improves.

Catalog record

CREATORAnirban Das Joanne Boisson Irtaza Khalid Sumita Garai Steven Schockaert
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE114 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.24965

Browse the full library · About the Tower of Babel