An Extensive Evaluation of PDDL Capabilities in off-the-shelf LLMs

Kaustubh Vyas, Damien Graux, Sébastien Montella, Pavlos Vougiouklis, Ruofei Lai, Keshuang Li, Yang Ren, & Jeff Z. Pan · 2025

Read An Extensive Evaluation of PDDL Capabilities in off-the-shelf LLMs by Kaustubh Vyas, Damien Graux, Sébastien Montella, Pavlos Vougiouklis, Ruofei Lai, Keshuang Li, Yang Ren, & Jeff Z. Pan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-20175.txt.

Description

Kaustubh Vyas and co-authors run a broad test of how well today's language models handle a formal planning language in this 2025 paper. The language, PDDL, is the standard way to state planning problems for AI systems: what actions exist, what they need, and what they change. Skill with it would open the door to machine-built plans that can be checked.

The team tested twenty models from seven families, both commercial and open, with no task-specific training. They probed whether the models could read PDDL, write it, and reason over it. Some models handled the language with real skill. Others broke down on harder cases that call for deeper planning knowledge. The authors map where off-the-shelf models stand today and where work is needed before they can be trusted with formal planning.

Catalog record

CREATORKaustubh Vyas, Damien Graux, Sébastien Montella, Pavlos Vougiouklis, Ruofei Lai, Keshuang Li, Yang Ren, & Jeff Z. Pan
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE32 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-20175

Browse the full library · About the Tower of Babel