SpatialBench-UC: Uncertainty-Aware Evaluation of Spatial Prompt Following in Text-to-Image Generation

Amine Rostane · 2026

Read SpatialBench-UC: Uncertainty-Aware Evaluation of Spatial Prompt Following in Text-to-Image Generation by Amine Rostane (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-13462.txt.

Introduction

Amine Rostane grades if image models place objects where a prompt says they must sit. A prompt may say a cat left of a chair. A checker must say if the pic obeys. Auto checks are hard, though. Detectors can miss a mark or give more than one apt hit. Plain shape tests blur in close calls.

Rostane treats the check as a pick with a right to pass. A checker may pass when proof is weak. It must also state how sure it is. Scores then read as a risk to coverage trade. For image teams, a checker that can pass beats one that must guess on weak pics.

Catalog record

CREATORAmine Rostane
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-13462

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SpatialBench-UC: Uncertainty-Aware Evaluation of Spatial Prompt Following in Text-to-Image Generation Amine Rostane Abstract Evaluating whether text to image models follow explicit spatial instructions is difficult to automate. Object detectors may miss targets or return multiple plausible detections, and simple geometric tests can become ambiguous in borderline cases. Spatial evaluation is naturally a selective prediction problem, a checker should be allowed to abstain when evidence is weak and should report confidence so results can be interpreted as a risk–coverage trade-off rather than a single score. We introduce SpatialBench-UC , a small, reproducible benchmark for pairwise spatial relations. The benchmark contains 200 prompts (50 object pairs × \times 4 relations) grouped into 100 counterfactual pairs obtained by swapping object roles (e.g., A left of B ↔ \leftrightarrow B right of A ). We release a benchmark package, versioned prompts, pinned configs, per-sample checker outputs, and report tables, enabling reproducible and auditable comparisons across models. We also include a lightweight human audit (N=200) used to calibrate the checker’s abstention margin and confidence threshold. On three baselines, Stable Diffusion 1.5 (prompt-only), SD 1.5+BoxDiff, and SD 1.4+GLIGEN, the checker reports (PASS rate / coverage): 11.8% / 23.8%, 40.4% / 42.5%, and 51.6% / 52.0%, with conditional PASS rates of 49.5%, 95.0%, and 99.3% on decided samples. PASS denotes the checker’s judgment , not human-verified accuracy unless validated by audit labels. 1 Introduction Text-to-image diffusion models can produce visually compelling images, yet they often violate explicit spatial constraints stated in text (e.g., “a dog to the left of a chair” ). Scoring spatial prompt following reliably is harder than it looks, automated evaluation typically depends on intermediate perception (object detection), which introduces common uncertainty sources, missed objects, multiple plausible instances, and ambiguous geometry when objects overlap or lie near relation boundaries. Collapsing these uncertainties into a single scalar score can make comparisons hard to interpret and hard to reproduce. Spatial evaluation is naturally a selective prediction problem, an evaluator should output PASS/FAIL when evidence is sufficient, abstain otherwise, and attach confidence so users can trade coverage for lower risk. Throughout, PASS/FAIL/UNDECIDABLE are checker verdicts (PASS is not ground-truth accuracy unless validated by human labels). Contributions. • Uncertainty-aware evaluation framework. We formalize spatial evaluation with explicit abstention (PASS/FAIL/UNDECIDABLE) and interpretable confidence scores. • Reproducible benchmark package. We release versioned prompts with hashes, pinned configurations, per-sample evaluation outputs, and structured metadata to support independent replication and auditing. • Counterfactual prompts pairing. The benchmark comprises 200 prompts (50 object pairs \times 4 axis-aligned relations) organized into 100 counterfactual pairs via role swapping (e.g., A left of B \leftrightarrow B right of A ). • Human audit and calibration protocol. A lightweight audit (N=200) grounds risk–coverage analysis and calibrates the abstention margin and confidence threshold. • Empirical comparison of generation strategies. We evaluate SD 1.5 (prompt-only), SD 1.5 with BoxDiff, and GLIGEN (SD 1.4) on a fixed set of generated images. See Sections 6 – 7 for the audit protocol and results; Figure 3 and Table 6 summarize the selective prediction and abstention behavior. (a) PASS. Prompt: potted plant right of vase (b) FAIL. Prompt: bus below laptop (c) UNDECIDABLE (near-boundary). Prompt: chair left of dog Figure 1 : Qualitative examples of checker outcomes. Each panel shows a generated image with detector boxes (red=A, blue=B) and the checker verdict. We include one clear PASS, one clear FAIL, and one abstention example where geometry is near the decision boundary. 2 Evaluation under uncertainty Uncertainty is the central obstacle in spatial evaluation. Rather than forcing every sample into PASS or FAIL, we allow the evaluator to abstain (UNDECIDABLE) when the available evidence is insufficient. This section characterizes the sources of ambiguity that motivate abstention and clarifies the scope of what SpatialBench-UC measures. 2.1 Sources of evaluation uncertainty Spatial prompt following is easy to state but difficult to verify automatically. Even when restricting attention to axis-aligned relations between two objects, several sources of uncertainty arise. First, missing detections , object detectors may fail to locate one or both targets, either because the objects are absent, too small, or simply missed by the model. Second, ambiguous referents , when multiple instances of the same object class appear in an image, identifying which pair corresponds to the prompt becomes unclear. Third, overlaps and near-boundary cases , geometric relations can become ill-defined when objects overlap substantially or when their bounding-box centers lie near the decision boundary separating “left” from “right” or “above” from “below.” Finally, perturbation sensitivity , small image changes such as blur, brightness shifts, or resizing can alter detection outputs or flip the geometric verdict, revealing instability in the evaluation pipeline itself. These ambiguities are not rare edge cases. In our fixed evaluation set, UNDECIDABLE outcomes are common, and the dominant causes are directly measurable (Table 6 ). 2.2 Evaluator semantics and scope SpatialBench-UC reports checker verdicts (PASS, FAIL, or UNDECIDABLE) together with a confidence score for each sample. A PASS verdict indicates that the detector–geometry checker judged the spatial relation to be satisfied; it should not be interpreted as ground-truth accuracy unless corroborated by human labels (Section 6 ). The benchmark measures compliance with axis-aligned spatial relations as judged by the evaluator , given pretrained object detectors and explicit abstention rules. Consequently, reported results reflect both the generator’s spatial prompt-following ability and the detectability of the target objects under the chosen detectors. The benchmark does not establish ground-truth correctness for every image, does not cover relations beyond the four axis-aligned cases, and does not claim that the confidence score constitutes a calibrated probability. 2.3 Uncertainty sources handled by the evaluator Table 1 summarizes the main uncertainty sources and how they map to checker behavior. This table appears before pipeline details to make abstention feel like an explicit, inspectable design choice rather than a black box. Uncertainty source How the evaluator handles it (decision + implication) Missing detections If either object is not detected after thresholding/size filtering, the checker abstains ( UNDECIDABLE ; reason missing ) and sets confidence to 0. Multiple instances / ambiguous referents If multiple instances of the target label have similar scores (within ambiguity_delta ), the checker abstains ( UNDECIDABLE ; reason ambiguous ) and sets confidence to 0. Strong overlap between boxes (horizontal relations) For left_of / right_of , if IoU exceeds max_overlap_iou , the checker abstains ( UNDECIDABLE ; reason high_overlap ) because center ordering is unreliable under heavy overlap. Near-threshold geometry If the normalized center delta lies within a margin band ( |d|\leq m ), the checker abstains ( UNDECIDABLE ; reason near_boundary ); geometry confidence goes to (near) zero by construction. Sensitivity to perturbations If a decided verdict (PASS/FAIL) is unstable under lightweight perturbations, the checker abstains ( UNDECIDABLE ; reason unstable ) and the stability component penalizes confidence. Under the current perturbations/threshold, this mechanism did not activate in our runs. Table 1 : Uncertainty sources explicitly handled by the evaluator and their implications for PASS/FAIL/UNDECIDABLE and confidence. Note on stability in the current runs. Stability is implemented and contributes to confidence; with our current perturbation set and threshold, it did not cause additional abstentions in these runs (Table 6 ). We keep it as a configurable safeguard for settings where detector/geometry sensitivity is higher. 3

Read the complete text