Project Auto-World: Towards Automated Benchmarking of Neural Relational Reasoners
Anirban Das Joanne Boisson Irtaza Khalid Sumita Garai Steven Schockaert · 2026
Read Project Auto-World: Towards Automated Benchmarking of Neural Relational Reasoners by Anirban Das Joanne Boisson Irtaza Khalid Sumita Garai Steven Schockaert (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-24965.txt.
Description
Anirban Das, Joanne Boisson, and three colleagues automate the benchmarking of neural models. Reasoning over relational structures is hard for neural models, above all when test problems exceed training difficulty, and nobody knows what makes an instance hard. Their Project Auto-World attacks the evaluation problem itself. A world is parametrised by Datalog rules, and an Edge Transformer serves as the model under test. Two search strategies, LLM driven evolutionary search and autonomous agentic search, hunt for samplers that generate hard instances.
The discovered samplers produce hard problem sets. Training the Edge Transformer on this generated data improves it, and the gain generalises to further data changes. The same machinery runs on worlds proposed by language models. Das and colleagues present this as a step toward self directed research in the field. Benchmarks stop being artefacts written by hand; they become opponents that grow harder as the reasoner improves.
Catalog record
| CREATOR | Anirban Das Joanne Boisson Irtaza Khalid Sumita Garai Steven Schockaert |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 114 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.24965