Binary Verification for Zero-Shot Vision

Jeffrey Liu, Rongbin Hu · 2025

Read Binary Verification for Zero-Shot Vision by Jeffrey Liu, Rongbin Hu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-10983.txt.

Introduction

Rongbin Hu and Jeffrey Liu propose a training-free workflow that makes picture models answer with more care. Models that work from words alone often face open-ended visual questions with many plausible replies. The workflow first turns such a query into a short choice list with clear options, so the task has a fixed shape.

It then asks one true-or-false question for each option and applies a plain rule to pick the winner. If just one option passes, that one is chosen; if not, a choice round follows among those left. Tests on grounding, spatial tasks, and puzzles showed large gains, and the workflow was built into a real video editing system.

Catalog record

CREATORJeffrey Liu, Rongbin Hu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-10983

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Binary Verification for Zero-Shot Vision Jeffrey Liu, Rongbin Hu Abstract We propose a training-free, binary verification workflow for zero-shot vision with off-the-shelf VLMs. It comprises two steps: ( i i ) quantization , which turns the open-ended query into a multiple-choice question (MCQ) with a small, explicit list of unambiguous candidates; and ( i ​ i ii ) binarization , which asks one True/False question per candidate and resolves deterministically: if exactly one is True , select it; otherwise, revert to an MCQ over the remaining plausible candidates. We evaluate the workflow on referring expression grounding (REC), spatial reasoning (Spatial-Map, Spatial-Grid, Spatial-Maze), and BLINK-Jigsaw. Relative to answering open-ended queries directly, quantization to MCQ yields large gains, and True/False binarization provides a consistent additional boost. Across all tasks, the same workflow produces significant improvements, indicating generality. Our theory formalizes how open-ended vision queries can be quantized to MCQs and further binarized into True/False verifications, establishing a hardness ladder ( T/F ≤ \leq MCQ ≤ \leq K K -way). A simple analysis explains why Boolean resolution boosts accuracy. Together, these components yield a simple and unified workflow that emphasizes inference-time design over task-specific training. It offers a practical, drop-in path to stronger zero-shot vision with today’s VLMs. 1 Introduction Zero-shot vision frome2013devise ; xian2019zsl is the ability of a model to perform a new visual task without any task-specific training or fine-tuning. It is increasingly critical for science and deployment. Compared with supervised learning, which requires task-specific annotations that are often expensive and time-consuming to build, zero-shot vision enables rapid and cost-effective adaptation to new tasks and domains. Moreover, proprietary vision language models (VLMs) often deliver stronger out-of-the-box performance and ship with mature APIs and infrastructure that simplify integration, but they typically restrict task-specific training beyond prompting, making inference-time methods especially valuable. Modern VLMs openai2024gpt4o ; liu2023llava ; wang2024cogvlm ; bai2023qwenvl ; bao2022beit ; dai2023instructblip ; he2022mae ; hong2024cogagent ; liu2024llavaNext show strong zero-shot ability on tasks such as image captioning, broad scene recognition, and short-form visual QA. In typical zero-shot use, the model answers an open-ended prompt, for example, “ Describe this image ”. However, for more demanding tasks such as precise grounding, spatial reasoning, or fine-grained recognition, performance is often unreliable without task-specific fine-tuning. This unreliability stems from the interaction of several factors: free-form outputs make small numeric and referential errors both frequent and difficult to detect; model confidences are poorly calibrated, undermining decision thresholds; and more. A fundamental idea in computing is that any number can be represented in a binary format. Quantizing and binarizing values to bits simplify arithmetic for machines and enable scalable computation. We adopt an analogous view and instantiate the idea of quantization and binarization for zero-shot vision tasks with a simple, training-free workflow. First, we quantize the task by generating candidate units, e.g. detector boxes, grid cells, or entity pairs, so an open query becomes a multiple choice question (MCQ) with a small set of explicit hypotheses. Next, we binarize the decision by asking, for each candidate h_{i} , a single yes/no claim that checks whether h_{i} matches the description, and the VLM must answer either True or False . Finally, we resolve outcomes deterministically: if exactly one candidate is True , we return it; if multiple answers are True , we restrict to that subset and issue a compact MCQ; if all are False , we fall back to the original MCQ. We evaluate the unified workflow across three task families using the same VLMs without any task-specific fine-tuning, and observe significant improvements on every task. • Referring Expression Comprehension (REC). REC qiao2020rec_survey is the task of locating the specific object described by a natural-language expression in a given image. On REC (RefCOCO/+/g) yu2016refcoco ; mao2016refcocog , our binary verification workflow with GPT-4o reaches [email protected] of 71.7%, 67.1%, and 64.8%, respectively, substantially above zero-shot GroundingDINO liu2023groundingdino (50.4%, 51.4%, 60.4%). Note that the original GPT-4o only has [email protected] of 8.4%, 7.3%, 9.1%, respectively. • Spatial reasoning (Map, Grid, Maze). For spatial reasoning wang2024spatial , we evaluate three tasks, namely, Spatial-Map, Spatial-Grid, and Spatial-Maze, which probe directional relations, grid area recognition, and path finding, respectively. We take the original task interface as the baseline and then apply our two-step binary verification workflow. On Spatial-Map , the baseline is already MCQ. Binarization improves over MCQ from 78.3% to 84.5% and from 25.5% to 31.3% with GPT-4o openai2024gpt4o and CogVLM wang2024cogvlm , respectively. On Spatial-Grid , spatial quantization, i.e., overlaying explicit grid on the original image, boosts the classification accuracy from 47.2% to 95.0% and from 23.2% to 58.8% with GPT-4o and CogVLM, respectively. On Spatial-Maze , spatial quantization by overlaying an explicit grid is again essential, which improves the accuracy of finding the right path from 16.5% to 68.7%. An embedding local T/F corridor checks further improves the accuracy to 75.7%. • BLINK-Jigsaw. BLINK fu2024blinkmultimodallargelanguage is a multiple-choice benchmark that reformats classic vision tasks into image prompts to probe core perceptual skills that humans solve “in a blink,” but that current VLMs often miss. To test whether our approach helps on these harder perception cases, we evaluate a modified and harder BLINK-Jigsaw setting. Our method improves accuracy from 51.6% (MCQ) to 56.8%, demonstrating that binary verification still delivers measurable gains even when the underlying task is challenging for state-of-the-art VLMs. Our approach is supported by a compact theory that mirrors the empirical trends. We show that any open-ended vision query can be quantized to a finite hypothesis set (an MCQ) and then binarized into per-option True/False checks, creating a hardness ladder under 0 – 1 loss: \;R_{K}^{\star}\!\geq\!R_{m}^{\star}\!\geq\!R_{2}^{\star}\, . An information-theoretic perspective yields the same ordering, and when per-option verifiers are calibrated to posteriors, aggregating T/F answers recovers the Bayes K -way rule. A capability model further explains the gains: simpler prompts expose more task-relevant signal, tightening Fano bounds cover2006elements on Bayes error for MCQ and especially for T/F. Finally, a two-hypothesis calculation shows that one-round binary verification typically exceeds MCQ. This paper makes the following contributions: • Method. We introduce a simple and unified workflow that quantizes an open-ended query to a small hypothesis set for a MCQ and then binarizes them into per-candidate True/False checks with deterministic resolution. • Experiment. Using the same/similar workflow, we show broad gains across five tasks in three families, where quantization improves over open-ended prompting and binarization provides a further consistent boost. • Theory. We develop a compact theory to justify the approach, clarify when and why it works, and support its generality across tasks. 2 Related Work Inference-time workflows. Modern LLM/VLM usage increasingly treats generation as a procedure rather than a single shot. Chain-of-thought wei2022chain elicits intermediate reasoning, ReAct yao2022react interleaves tool use with reasoning, and modular frameworks (e.g., DSPy khattab2024dspy ) compose these behaviors declaratively. Our work adopts this workflow view for vision, but focuses on a verification-centric interface. Verification vs. selection. A large body of methods improves answers by selection over multiple candidates, including self-consistency/majority voting wang2023selfconsistency , process-of-elimination prompting zhou2023poe , LLM-as-a-judge liu2023geval ; zheng2023scoring . These approaches remain MCQ-style, typically relying on sampling or confidence margins. In contrast, we convert each option into an independent True / False claim and apply a deterministic, codeword-style resolution

Read the complete text