Visual Chronicles: Using Multimodal LLMs to Analyze Massive Collections of Images
Boyang Deng Songyou Peng Kyle Genova, Gordon Wetzstein Noah Snavely Leonidas Guibas Thomas Funkhouser · 2025
Read Visual Chronicles: Using Multimodal LLMs to Analyze Massive Collections of Images by Boyang Deng Songyou Peng Kyle Genova, Gordon Wetzstein Noah Snavely Leonidas Guibas Thomas Funkhouser (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-08727.txt.
Introduction
Boyang Deng and co-authors built a system that mines tens of millions of city photos for signs of how places change over time. Their goal is to answer open questions, such as what kinds of changes happen most often in a city, without pre-set categories or labeled training data. The photo sets are far too large for a multimodal language model to read at once, by a factor of about ten thousand.
The system breaks the job into small steps, each handled by the model: comparing matched photos, describing local changes, then merging the findings into shared trends. In large-city experiments it beat baseline methods and surfaced believable trends, such as outdoor dining appearing and an overpass being painted blue. The authors see the approach as a new lens on urban change.
Catalog record
| CREATOR | Boyang Deng Songyou Peng Kyle Genova, Gordon Wetzstein Noah Snavely Leonidas Guibas Thomas Funkhouser |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 79 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-08727
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Visual Chronicles: Using Multimodal LLMs to Analyze Massive Collections of Images Boyang Deng Songyou Peng Kyle Genova, Gordon Wetzstein Noah Snavely Leonidas Guibas Thomas Funkhouser Abstract We present a system using Multimodal LLMs (MLLMs) to analyze a large database with tens of millions of images captured at different times, with the aim of discovering patterns in temporal changes. Specifically, we aim to capture frequent co-occurring changes (“trends”) across a city over a certain period. Unlike previous visual analyses, our analysis answers open-ended queries (e.g., “what are the frequent types of changes in the city?”) without any predetermined target subjects or training labels. These properties cast prior learning-based or unsupervised visual analysis tools unsuitable. We identify MLLMs as a novel tool for their open-ended semantic understanding capabilities. Yet, our datasets are four orders of magnitude too large for an MLLM to injest as context. So we introduce a bottom-up procedure that decomposes the massive visual analysis problem into more tractable sub-problems. We carefully design MLLM-based solutions to each sub-problem. During experiments and ablation studies with our system, we find it significantly outperforms baselines and is able to discover interesting trends from images captured in large cities (e.g., “addition of outdoor dining,”, “overpass was painted blue,” etc.). See more results and interactive demos at our project page . † † * Equal contributions. Ordered decided by a random number generator. Figure 1 : Visual Trend Discovery from Massive-Scale Collections of Images . We introduce Visual Chronicles , a novel approach that uses Multimodal LLMs (MLLMs) to make discoveries from massive collections of images over time. We analyze millions of images collected over more than 10 years by Google Street View. As seen above, our method discovers both expected and surprising visual trends in San Francisco—e.g. outdoor dining areas were added to many storefronts, and miles of overpass was painted blue, both with visual evidence. 1 Introduction Searching for semantic patterns and correlations in image databases is a long-standing problem in computer vision with applications in urban planning, visual analytics, and other fields. Classical research on this problem includes tracking changes in images within specific domains over time ( e.g . , “A Century of Portraits” [ 15 ] ) and finding salient patterns of objects in images ( e.g . , estimating demographics from vehicle sightings [ 14 ] ). However, these methods have only been demonstrated for predetermined specific target subjects ( e.g . , faces [ 15 ] or cars [ 14 ] ); and often require labeled training data ( e.g . , labeled car attributes [ 14 ] ). In this paper, we study how to make open-ended discoveries about frequently occurring changes (“trends”) across a city within a given time period, using visual analysis of a massive collection of images without required labeled training data. We provide a study using Google Street View as a data source [ 3 ] . Street View provides tens of millions of posed and timestamped images covering large cities with repeated captures spanning a decade. Our goal is to answer open-vocabulary queries about how a city has changed over time. In contrast to prior work, our queries are open-ended, and not limited to a predetermined list of subject categories. Additionally, the data doesn’t come with labeled answers to our queries, and manual labeling at scale is infeasible because the nature of the queries are unknown a priori . The new properties of our analysis make it a difficult problem. Any pretrained subject-specific recognition model, e.g . a car detector, is insufficient for open-ended queries. Meanwhile, the lack of labels prevents us from training a tailored learning-based model as in prior work [ 14 ] . Training-free image analysis techniques, from image features [ 11 ] to unsupervised remote sensing [ 40 ] , have also been used in prior work. However, our goal is to find semantically interesting insights. This demands a holistic semantic understanding of images, e.g . , understanding that a shop is closed from visual evidence, which these prior techniques fail to offer. Therefore, we develop a new, scalable, training-free solution to open-ended visual analysis, leveraging the considerable scene understanding capabilities of Multimodal Large Language Models (MLLMs). Our solution is motivated by a key observation—readymade MLLMs are very effective at open-ended semantic analyses of a small collection of images. They can detect semantically salient temporal changes with high accuracy in 20-40 images. However, a straightforward application of MLLMs does not solve our problem due to the massive scale of the data. For instance, simply asking an MLLM to “name the trends” without using images would fail, because we empirically find they mainly produce abstract answers such as economical growth. Alternatively, we could feed all images as context to an MLLM, and ask it to “find the trends”. However, this does not work due to the inability of MLLMs to handle massive context. Thus, we design a novel bottom-up hierarchical system to decompose the problem into smaller subtasks: detecting and summarizing local visual changes followed by finding global trends among these local changes. For each subtask, we propose an effective MLLM-based solution. To the best of our knowledge, our system marks the first use of MLLMs for the analysis of a database of more than a million images. Experiments with our proposed system reveal several interesting insights. According to results of tests with manually labeled data produced for this paper, we find that our system discovers visual trends more accurately than prior solutions or trivial use of MLLMs. In experiments with 20 million Street View images from each of two major cities (NYC and SF), it finds several interesting trends that would be difficult to discover without detailed visual analyses (see Fig. 1 ). Overall, our research contributions are as follows: • We study a novel open-ended visual analysis problem on a massive-scale dataset without labels . • We identify MLLMs as the critical tool for such unprecedented analyses, and design a novel system for massive-scale analyses using MLLMs, bypassing the context and efficiency limitations. • We present experiments using our system in two major cities yielding unexpected and informative findings. These contributions together signal a new paradigm of computer vision studies—advancing from previous task-specific supervised training or supervised fine-tuning, to carefully designed use of MLLMs in a system like ours, for complicated large-scale tasks without labels. 2 Related Work 2.1 Large-Scale Image Analysis General Analysis. Analyzing large-scale visual data–such as datasets composed of millions of images–can unveil important characteristics of the data that cannot be observed from just a few examples. Prior work has applied such analytical study to various visual data collections, leading to interesting discoveries, including distinctive visual styles [ 11 , 47 , 21 ] , recurring visual patterns in art [ 46 , 17 ] , dataset biases and irregularities [ 51 , 4 , 24 ] , and socio-economic insights such as economic outcomes [ 19 ] , urban neighborhood perceptions [ 12 ] , and demographic compositions [ 14 ] . Among these works, imagery linked to GIS [ 3 ] has been a popular data source [ 11 , 38 , 19 , 12 , 14 ] , closely related to