Evaluating Spatiotemporal Consistency in Automatically Generated Sewing Instructions

Luisa Geiger, Mareike Hartmann, Michael Sullivan, Alexander Koller · 2025

Read Evaluating Spatiotemporal Consistency in Automatically Generated Sewing Instructions by Luisa Geiger, Mareike Hartmann, Michael Sullivan, Alexander Koller (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-24792.txt.

Introduction

Luisa Geiger, Mareike Hartmann, Michael Sullivan, and Alexander Koller propose a new way, in a 2025 preprint, to grade instructions written by language models. The test case is sewing: step-by-step guidance where order and placement matter. Common scoring tools compare texts by wording or by rough meaning. A swapped step can score well under those tools while ruining the garment. The authors' metric instead builds a tree of the build process. Steps are judged by where they sit in space and time within that tree.

Against hand-counted errors, the new metric matched human judgment better than the old scores. It also tracked human quality ratings more closely. Fake examples crafted to fool wording-based tools left it unmoved. The authors argue that assembly-style writing needs structural grading. Their tree approach could extend to furniture, cooking, or repair tasks, where steps out of order break the result.

Catalog record

CREATORLuisa Geiger, Mareike Hartmann, Michael Sullivan, Alexander Koller
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE41 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-24792

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Evaluating Spatiotemporal Consistency in Automatically Generated Sewing Instructions Luisa Geiger, Mareike Hartmann, Michael Sullivan, Alexander Koller Abstract In this paper, we propose a novel, automatic tree-based evaluation metric for LLM-generated step-by-step assembly instructions, that more accurately reflects spatiotemporal aspects of construction than traditional metrics such as BLEU and BERT similarity scores. We apply our proposed metric to the domain of sewing instructions, and show that our metric better correlates with manually-annotated error counts as well as human quality ratings, demonstrating our metric’s superiority for evaluating the spatiotemporal soundness of sewing instructions. Further experiments show that our metric is more robust than traditional approaches against artificially-constructed counterfactual examples that are specifically constructed to confound metrics that rely on textual similarity. 1 Introduction Figure 1: Illustration of our tree-based evaluation metric. Instructions are automatically converted a tree that encodes the order and components of each step. These automatically-extracted trees are then compared to gold trees to yield a score. Creating consistent, high quality instructions in any domain can be a difficult process. Much research has investigated the ability of Large Language Models (LLMs) to generate such instructions, most commonly in the domain of cooking recipes ( Li et al. 2024 ; Salvador et al. 2019 , e.g.) . High quality instructions make it as easy as possible for the reader to connect the textual instructions to the required actions in the physical world. For the instructions to make sense with regard to the physical world, the LLM that is generating them needs an understanding of the current world state. This necessitates spatial (where objects are located) and temporal (when objects change their location) awareness: although LLMs have continuously improved on a wide variety of benchmarks, they still have difficulties with consistency in the area of spatiotemporal reasoning Aghzal et al. 2025 . In particular, the generation of textual sewing instructions is a multimodal reasoning problem that necessitates spatiotemporal awareness of multiple objects and their surroundings, as well as knowledge of the current world state: specifically, the current assembly state of the garment at each step of the instructions. This task is similar to other instruction generation tasks such as cooking recipe generation and Minecraft instruction generation ( Narayan-Chen et al. 2019 , e.g.) , in that the generation task consists of generating procedural textual instructions, including explicit or implicit representations/descriptions of the current world state. However, generating sewing instructions differs from these more widely-researched tasks, due to the more complex operations that need to be performed. The complexity of this process lends to sensitivity to even slight errors: incorrect use of terminology or ambiguous explanations of assembly operations can lead to catastrophic misunderstandings between writer and the reader. While this can also be the case for other instruction generation tasks, the increased complexity involved in the physical manipulation of fabric pieces leads to a far more spatiotemporally sensitive process than (for example) stacking blocks. Since the quality of sewing instructions heavily depends on their ability to capture the correct way to assemble the pieces, evaluating generated sewing instructions in meaningful way is not a trivial task. Common evaluation metrics such as BLEU Papineni et al. 2002 and BERT-Score Zhang et al. 2020 only focus on superficial similarities between the generated instructions and a given set of gold reference instructions (see Sections 2 and 5 ). In this paper, we introduce an automatic, tree-based evaluation metric (Section 4 ), that mitigates the shortcomings of traditional evaluation metrics with respect to the task of instruction generation. We compare our evaluation metric to BLEU, ROUGE-L Lin 2004 , and BERT-Score on sewing instructions generated via a range of prompting strategies, demonstrating the versatility and robustness of our approach with respect to varying methods of instruction generation (Section 5 ). We show that our metric better correlates with manually-annotated error counts in the model-generated instructions, indicating that our approach better reflects spatiotemporal correctness than competing metrics. To further highlight the sensitivity of our approach to correctness, we construct an artificial dataset in which the steps of the model-generated instructions are randomly permuted: while our metric is highly sensitive to these nonsensical instructions, the traditional metrics fail to meaningfully reflect any difference between the original and permuted instructions. Additionally, we find that our metric is weakly positively correlated with subjective human ratings, while all other, traditional similarity measures are negatively correlated. All prompts, generated instructions, input images, code files, and data required to replicate these experiments are available on GitHub 1 1 1 https://github.com/coli-saar/generatingsewinginstructions . 2 Related Work Multimodal Reasoning. Liu et al. 2024b evaluate a range of Multimodal LLMs (MLLMs) across a variety of web tasks, and find that many models do not exceed random-chance performance on action prediction and grounding tasks, speaking to these models’ limited reasoning and grounding abilities—both critical skills for sewing instruction generation (as discussed in Section 1 ). Although GPT-4V Achiam et al. 2023 and Claude Anthropic 2024 outperform their open-source counterparts by a notable margin, GPT-4V—the best-performing model they evaluate—only achieves an average score of 64.6 out of 100, indicating that there is still much room for improvement, even among closed-source MLLMs. Ji et al. 2022 investigate human and LLM abtract visual reasoning abilities through the use of tangram puzzles Hawkins et al. 2020 : both models that they evaluate—CLIP Radford et al. 2021 and ViLT Kim et al. 2021 —demonstrate limited abstract reasoning abilities in comparison to human participants. However, Ji et al. 2022 find that explicit descriptions and color-coded highlights of various tangram components improve performance for both humans and MLLMs, suggesting that similar knowledge-augmentation can be leveraged to improve model-generated sewing instructions. Textual Spatiotemporal Reasoning and Planning. Although the findings of Lyu et al. 2020 indicate that fine-tuning and careful data-engineering can improve the common-sense reasoning abilities of then-SoTA LMs—i.e. BERT Devlin et al. 2019 , XLNet Yang et al. 2019 , RoBERTa Liu et al. 2019 , and GPT-2 Radford et al. 2019 —to near-human performance, Aghzal et al. 2025 suggest that the capabilities of LLMs are limited when it comes to long-term planning and spatial reasoning. Specifically, Aghzal et al. 2025 find that in-context learning (ICL) does not help GPT-4 avoid obstacles—although ICL does improve the model’s ability to reach the goal—in textual grid-navigation environments. While chain-of-thought ( Wei et al. 2022 , CoT;) prompting does improve obstacle-avoidance abilities, these abilities still degrade as obstacle number and distance to the goal increase. Of particular concern to the topic at hand are Wu et al. 2024 ’s ( Wu et al. 2024 ) findings that LLM’s reasoning performance degrades substantially when they are presented with counterfactual instances: reasoning problems in which well-known rules are altered, such as chess puzzles in which the piece types’ valid moves are modified. This indicates that LLMs may struggle to accomplish a task such as sewing-instruction generation—examples of which are unlikely to occur frequently in the models’ training data. Cooking Recipe Generation. Given the similarities between the two tasks, it stands to reason that many findings in the area of cooking-recipe generation—a far more well-researched task—should translate to sewing-instruction generation. Salvador et al. 2019 propose an approach for generating cooking recipes under which the model first predicts ingredients before generating a

Read the complete text