PuzzleGPT: Emulating Human Puzzle-Solving Ability for Time and Location Prediction

Hammad Ayyubi, Xuande Feng, Junzhang Liu, Xudong Lin Zhecan Wang Shih-Fu Chang · 2025

Read PuzzleGPT: Emulating Human Puzzle-Solving Ability for Time and Location Prediction by Hammad Ayyubi, Xuande Feng, Junzhang Liu, Xudong Lin Zhecan Wang Shih-Fu Chang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-14210.txt.

Introduction

Hammad Ayyubi and co-authors present PuzzleGPT in 2025, a system that guesses when and where a photo was taken. People solve such puzzles by spotting clues, like signs, plants, or building styles, and chaining them together. The team split that skill into parts and built a module for each.

One module spots visual clues, one reasons over them, one merges the chains, one searches the web when stuck, and one filters noise. The system needs no training on the task and explains its steps. On two test sets it beat big vision-language models, including GPT-4V, by wide margins. It even matched models trained for the job.

Catalog record

CREATORHammad Ayyubi, Xuande Feng, Junzhang Liu, Xudong Lin Zhecan Wang Shih-Fu Chang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-14210

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

PuzzleGPT: Emulating Human Puzzle-Solving Ability for Time and Location Prediction Hammad Ayyubi, Xuande Feng, Junzhang Liu, Xudong Lin Zhecan Wang Shih-Fu Chang Abstract The task of predicting time and location from images is challenging and requires complex human-like puzzle-solving ability over different clues. In this work, we formalize this ability into core skills and implement them using different modules in an expert pipeline called PuzzleGPT. PuzzleGPT consists of a perceiver to identify visual clues, a reasoner to deduce prediction candidates, a combiner to combinatorially combine information from different clues, a web retriever to get external knowledge if the task can’t be solved locally, and a noise filter for robustness. This results in a zero-shot, interpretable, and robust approach that records state-of-the-art performance on two datasets – TARA and WikiTilo. PuzzleGPT outperforms large VLMs such as BLIP-2, InstructBLIP, LLaVA, and even GPT-4o, as well as automatically generated reasoning pipelines like VisProg Gupta and Kembhavi (2022) , by at least 32% and 38%, respectively. It even rivals or surpasses finetuned models. Figure 1: An illustration of different skills (Puzzle-like) required for solving Time and Reasoning prediction tasks. 1 Introduction Recent advances in Vision-Language (VL) research have led to models that perform impressively Zhu et al. (2023) ; Li et al. (2022) ; Lu et al. (2022a) ; Alayrac et al. (2022) ; Team et al. (2024) on a variety of tasks such as GQA Hudson and Manning (2019) , VQA v2 Antol et al. (2015) , VCR Zellers et al. (2019) , OK-VQA Marino et al. (2019) , Science-QA Lu et al. (2022b) , visual entailment Xie et al. (2019) . Chain-of-thought reasoning Lu et al. (2024) ; Lu et al. (2022b) . These tasks primarily assess one of, or at most a combination of, perception, reasoning, and outside knowledge retrieval abilities. For example, OK-VQA requires perception and outside knowledge retrieval, and GQA and VCR require perception and commonsense reasoning. However, humans seamlessly integrate a variety of skills – perception, reasoning, knowledge retrieval, and common sense – to solve complex, multi-step problems. Tasks and benchmarks that test these diverse skills are crucial for developing models that approach human-level reasoning. The task of time and place reasoning from images proposed by TARA Fu et al. (2022) takes a step closer to this goal. It demands a mix of perception, reasoning, combinatorial, and outside knowledge retrieval abilities over multiple steps. It is like solving a puzzle. For example, in Figure 1, it is required to detect entities such as Times Square and visual text, “Justice for George Floyd”. Then, a reasoner needs to deduce possible location (New York/United States) and time candidates (post-2000, 2020-2021) from these clues. Next, these candidates need to be combined in various ways to find a candidate at the intersection of all candidates (post 2000 \cap 2020-2021 = 2020-2021). Finally, if the answer is still unclear (2020-2021), a web search is required using the deduced information. The practical applications of the task stem from its focus on images depicting events that occurred at a specific location and time. This has incredibly useful applications, such as timeline construction, and stitching together news stories from online pictures and social media posts. Existing works take a direct approach to this nuanced problem. TARA tries to supervise a model directly to predict location and time directly. The hope is that the model will learn to identify time and location discriminative clues implicitly, given appropriate supervisory signals. While the approach might have worked for a limited time and location candidates, the real scenario of hundreds of locations/time with fine differences makes this approach unscalable, and thus impractical. QR-CLIP Shi et al. (2023) additionally tries to incorporate external knowledge in the learning process. However, it oversimplifies the problem and assumes mere retrieval can accomplish the task without relying on specific clues and their combinatorial intersections. We argue that a complex puzzle-like problem like TARA, requires an equally well thought-out solution. To this end, we propose PuzzleGPT. PuzzleGPT abstracts the skills required to solve it into five core abstract ideas: perceiver, reasoner, combiner, noise filter, and knowledge retriever. It represents each with specific modules that perform specific tasks. The perceiver processes visual signals and identifies different entities such as people, buildings, cultural signals, and OCR text. For each of these entities, the reasoner tries to deduce their co-relations with a location and time candidate. Integrating clues from multiple entities is crucial for accurate prediction. However, simply combining all clues can introduce noise from irrelevant information, while relying on individual clues might provide insufficient context. To address this challenge, we propose a confidence-based hierarchical combination approach. This approach analyzes clues at increasing levels of granularity: first, individual entities; then, pairs; followed by triplets, and so on, tracking candidate predictions. The process stops once a candidate reaches a threshold vote, efficiently combining entities while minimizing noise. Apart from being zero-shot, our design choices lend PuzzleGPT desirable properties. Reasoner makes the approach interpretable and thus trustworthy. A hierarchical combination approach makes it not only combinatorial but also noise-resistant. Web retriever infuses the approach with the ability to incorporate world knowledge into the reasoning process. The noise filter adds further robustness. PuzzleGPT scores state-of-the-art (SOTA) zero-shot performance on TARA, coming close to or surpassing even fine-tuned approaches. We demonstrate that our method outperforms existing SOTA VL models like Instruct BLIP Dai et al. (2023) , BLIP-2 Li et al. (2023) , LLaVA Liu et al. (2023a) , by a margin of at least 32% (standardized location accuracy). It even beats the popular proprietary GPT-4o OpenAI (2024) . This highlights current VLMs’ inability to simultaneously employ multiple skills to solve a task. We also report superior performance to automatically generated modular pipelines like VisProg, indicating generating an automatic pipeline for this complex task perhaps exceeds their current capabilities. Furthermore, we show that our method generalizes and scores SOTA on another location and time dataset, WikiTilo Zhang et al. (2024) . We make the following contributions: • We propose a novel method, PuzzleGPT, to emulate human puzzle-solving ability for predicting time and location from images. • Our design choices make our approach interpretable, robust, combinatorial, and retrieval augmented. • PuzzleGPT scores SOTA performance on TARA and WikiTilo. Figure 2: PuzzleGPT overview depicting each of the components – Perceiver, Reasoner, Combiner, Noise Filter, and the Online Retriever. The modular approach makes PuzzleGPT interpretable and robust. All VLMs/LLMs are pretrained and frozen. 2 Related Work Vision-Language Models. Recently, VLMs Radford et al. (2021) ; Alayrac et al. (2022) ; Li et al. (2023) ; Liu et al. (2023b) have demonstrated remarkable multimodal capabilities through large-scale vision-language training. One family of VLMs such as CLIP Radford et al. (2021) typically trains a visual encoder and a text encoder to map visual and text input into a common embedding space. The resulting visual encoders are widely adopted to extract visual features that are fed to LLMs in the other family of work Alayrac et al. (2022) ; Li et al. (2023) ; Liu et al. (2023b) ; Lin et al. (2023) . For example, LLaVA takes CLIP’s visual feature as input and is trained to generate target text. These VLMs with text-generation abilities have achieved superior performance

Read the complete text