TACIT Benchmark: A Programmatic Visual Reasoning Benchmark for Generative and Discriminative Models
Daniel Nobrega Medeiros · 2026
Read TACIT Benchmark: A Programmatic Visual Reasoning Benchmark for Generative and Discriminative Models by Daniel Nobrega Medeiros (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-00206.txt.
Introduction
Medeiros introduces TACIT, a programmatic visual reasoning benchmark with 10 tasks across 6 reasoning domains, from spatial navigation and pattern completion to causal simulation, logic constraints, graph theory and topology. It offers two tracks: a generative track where models produce solution images checked by vision pipelines, and a discriminative track with five-way multiple choice using near-miss distractors. Each distractor breaks exactly one structural constraint, forcing fine-grained reasoning over surface cues. The first release ships 6,000 puzzles with 108,000 PNG images across three resolutions, under the Apache 2 licence.
Catalog record
| CREATOR | Daniel Nobrega Medeiros |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 32901 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.00206
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
TACIT Benchmark: A Programmatic Visual Reasoning Benchmark for Generative and Discriminative Models Daniel Nobrega Medeiros Abstract Existing visual reasoning benchmarks predominantly rely on natural language prompts, evaluate narrow reasoning modalities, or depend on subjective scoring procedures such as LLM-as-judge. We introduce the TACIT Benchmark, a programmatic visual reasoning benchmark comprising 10 tasks across 6 reasoning domains—spatial navigation, abstract pattern completion, causal simulation, logical constraint satisfaction, graph theory, and topology. The benchmark provides dual-track evaluation: a generative track in which models must produce solution images verified through deterministic computer-vision pipelines, and a discriminative track offering five-way multiple choice with structurally plausible near-miss distractors. Each distractor violates exactly one structural constraint, requiring models to reason about fine-grained visual differences rather than exploit superficial cues. Version 0.1.0 distributes 6,000 puzzles (108,000 PNG images across three resolutions) with fully deterministic seeded generation and reproducible verification. The dataset, generation code, and evaluation harness are released under the Apache 2.0 license on HuggingFace (DOI: 10.57967/hf/7904 ). 1 Introduction The rapid advancement of multimodal foundation models [ 17 , 9 , 2 ] has created an urgent need for evaluation instruments that probe genuinely visual reasoning capabilities rather than language-mediated pattern matching. Current benchmarks fall into two broad camps: those that embed reasoning challenges in natural language with visual accompaniment [ 22 , 13 ] , and those that focus on a single reasoning modality such as abstract analogy [ 23 ] or general intelligence [ 6 ] . The former conflate linguistic competence with visual reasoning, while the latter lack the breadth needed to characterize a model’s visual cognition profile across diverse domains. A further limitation of existing benchmarks is their reliance on evaluation methods that are either narrowly discriminative (multiple choice only) or subjective (human annotators, LLM-as-judge). Discriminative-only evaluation cannot distinguish genuine solution construction from educated guessing among options, and subjective scoring introduces variability that undermines reproducibility. For visual reasoning tasks where the correct answer is a structured image—a solved maze, a colored graph, a projected silhouette—there is no principled reason why verification must involve human judgment. The answer is either structurally correct or it is not. The TACIT Benchmark addresses these gaps through three design commitments. First, language minimality : all task instructions are encoded visually, with no natural language clues beyond axis labels and legends, ensuring that reasoning performance reflects visual cognition rather than language comprehension. Second, dual-track evaluation : every puzzle supports both a generative track (the model produces a solution image) and a discriminative track (the model selects from five candidates), enabling direct measurement of the gap between constructive and selective reasoning on identical stimuli. Third, deterministic verification : every generative response is verified through a task-specific computer-vision pipeline—BFS path validation for mazes, pixel-level grid comparison for cellular automata, structural similarity for pattern matrices—eliminating evaluator subjectivity entirely. The benchmark spans six reasoning domains instantiated in ten tasks: multi-layer maze navigation (spatial), Raven’s progressive matrices (abstract pattern), forward and inverse cellular automata (causal simulation), visual logic grids (constraint satisfaction), graph k -coloring and isomorphism detection (graph theory), unknot detection (topology), and orthographic projection paired with isometric reconstruction (geometric projection). Each task features parameterized difficulty axes and near-miss distractors that violate exactly one structural constraint, preventing shortcut strategies based on superficial visual features. Version 0.1.0 of the benchmark distributes 6,000 puzzles, producing 108,000 PNG images at three resolutions (512, 1024, and 2048 pixels), generated deterministically from a single seed. The contributions of this paper are: • A benchmark of 10 visual reasoning tasks across 6 domains with parameterized difficulty. • A dual-track evaluation framework with deterministic CV-based generative verification. • A single-constraint distractor system ensuring structurally plausible wrong answers. • An open-source generation and evaluation pipeline enabling reproducible, extensible research. 2 Related Work Visual reasoning benchmarks. Raven’s Progressive Matrices [ 18 ] have long served as a gold standard for abstract reasoning assessment. Zhang et al. [ 23 ] introduced RAVEN, a procedurally generated dataset of matrix reasoning problems, enabling scalable evaluation but restricted to a single reasoning modality. The Abstraction and Reasoning Corpus (ARC) [ 6 , 5 ] takes a complementary approach, emphasizing generalization from few examples across diverse grid-based tasks, though its evaluation remains discriminative-only. Bongard-LOGO [ 16 ] targets concept learning through visual analogy but operates on simple binary classification. These benchmarks share a common limitation: they evaluate a single facet of visual reasoning and lack generative evaluation tracks. Multimodal evaluation suites. MMMU [ 22 ] and MathVista [ 13 ] evaluate multimodal models on expert-level tasks spanning dozens of disciplines. While comprehensive in scope, these benchmarks present reasoning challenges primarily through natural language, making it difficult to isolate visual reasoning from linguistic competence. CLEVR [ 10 ] pioneered programmatic scene generation for visual question answering but is limited to spatial relationship queries about simple 3D scenes. Programmatic and procedural benchmarks. Procedural generation offers several advantages for benchmark construction: unlimited scale, parameterized difficulty, deterministic reproducibility, and known ground truth. Our work builds on this tradition while extending it to encompass multiple reasoning domains, dual-track evaluation, and computer-vision-based verification of generative outputs. Positioning of TACIT. The TACIT Benchmark distinguishes itself along four axes: (i) language minimality —tasks are specified visually, removing linguistic confounds; (ii) breadth —six reasoning domains spanning spatial, causal, logical, graph-theoretic, topological, and geometric reasoning; (iii) dual-track evaluation —both generative and discriminative assessment on identical puzzles; and (iv) deterministic verification —all scoring is performed by CV pipelines, eliminating evaluator subjectivity. Table 1 summarizes how TACIT compares to prior work. Table 1 : Comparison of TACIT Benchmark with related visual reasoning benchmarks. Lang-min : language-minimal task specification. Gen : generative evaluation track. Disc : discriminative evaluation track. Det-V : deterministic verification (no human/LLM judge). Benchmark Domains Tasks Lang-min Gen Disc Det-V RAVEN 1 1 ✓ × ✓ ✓ ARC 1 vary ✓ × ✓ ✓ Bongard-LOGO 1 1 ✓ × ✓ × MMMU many 900+ × × ✓ × MathVista many 6k+ × × ✓ × CLEVR 1 1 × × ✓ ✓ TACIT 6 10 ✓ ✓ ✓ ✓ 3 Benchmark Design 3.1 Design Principles The TACIT Benchmark is organized around five design principles drawn from the theoretical framework on implicit computation in neural networks [ 15 ] : 1. Language minimality. All task instructions are encoded through visual layout, color coding, and geometric conventions. The only text present consists of axis labels (“Layer 1”, “State T”), numeric indices, and legend entries, ensuring that a model’s performance reflects visual reasoning rather than language understanding. 2. Dual-track evaluation. Every puzzle instance supports both generative (produce a solution image) and discriminative (select from five candidates) evaluation, enabling researchers to quantify the gap between constructive and selective reasoning. 3. Deterministic verification. All generative responses are verified by task-specific computer-vision pipelines. No human judgment or LLM-as-judge is involved at any point in scoring. 4. Parameterized difficulty. Each task exposes independent difficulty axes (grid size, rule complexity, node count, crossing number, etc.) that control problem hardness along interpretable dimensions. 5. Near-miss distractors. Each distractor violates exactly one structural constraint of the correct solution, creating plausible wrong answers that cannot be eliminated by superficial feature matching. 3.2 Dual-Track Evaluation Architecture Figure 1 illustrates the two evaluation tracks. Track 1: Generative evaluation. The model receives