PhotoBench: Beyond Visual Matching Towards Personalized Intent-Driven Photo Retrieval
Tianyi Xu, Rong Shan, Junjie Wu, Jiadeng Huang, Teng Wang, Jiachen Zhu, Wenteng Chen, Minxin Tu, Quantao Dou, Zhaoxiang Wang, Changwang Zhang, Weinan Zhang, Jun Wang, Jianghao Lin · 2026
Read PhotoBench: Beyond Visual Matching Towards Personalized Intent-Driven Photo Retrieval by Tianyi Xu, Rong Shan, Junjie Wu, Jiadeng Huang, Teng Wang, Jiachen Zhu, Wenteng Chen, Minxin Tu, Quantao Dou, Zhaoxiang Wang, Changwang Zhang, Weinan Zhang, Jun Wang, Jianghao Lin (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-01493.txt.
Introduction
Tianyi Xu, Rong Shan, Junjie Wu, and colleagues built a test set for finding photos in personal albums in this 2026 work. Real albums hold years of images tied to places, people, and life events. Most past tests use lone web images, so they miss the rich links that shape real user requests.
Their set, PhotoBench, draws on real personal albums and builds a profile for each image. Profiles join what the image shows with time, place, people, and event facts. Queries built from life paths then ask for multi-step thought, not just visual match. Tests expose two flaws: fused models fail on non-visual limits, and agent systems juggle search tools poorly. Both flaws point to what photo search must fix next.
Catalog record
| CREATOR | Tianyi Xu, Rong Shan, Junjie Wu, Jiadeng Huang, Teng Wang, Jiachen Zhu, Wenteng Chen, Minxin Tu, Quantao Dou, Zhaoxiang Wang, Changwang Zhang, Weinan Zhang, Jun Wang, Jianghao Lin |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 70619 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.01493
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
PhotoBench: Beyond Visual Matching Towards Personalized Intent-Driven Photo Retrieval Tianyi Xu, Rong Shan, Junjie Wu, Jiadeng Huang, Teng Wang, Jiachen Zhu, Wenteng Chen, Minxin Tu, Quantao Dou, Zhaoxiang Wang, Changwang Zhang, Weinan Zhang, Jun Wang, Jianghao Lin Abstract . Personal photo albums are not merely collections of static images but living, ecological archives defined by temporal continuity, social entanglement, and rich metadata, which makes the personalized photo retrieval non-trivial. However, existing retrieval benchmarks rely heavily on context-isolated web snapshots, failing to capture the multi-source reasoning required to resolve authentic, intent-driven user queries. To bridge this gap, we introduce PhotoBench , the first benchmark constructed from authentic, personal albums. It is designed to shift the paradigm from visual matching to personalized multi-source intent-driven reasoning. Based on a rigorous multi-source profiling framework, which integrates visual semantics, spatial-temporal metadata, social identity, and temporal events for each image, we synthesize complex intent-driven queries rooted in users’ life trajectories. Extensive evaluation on PhotoBench exposes two critical limitations: the modality gap , where unified embedding models collapse on non-visual constraints, and the source fusion paradox , where agentic systems perform poor tool orchestration. These findings indicate that the next frontier in personal multimodal retrieval lies beyond unified embeddings, necessitating robust agentic reasoning systems capable of precise constraint satisfaction and multi-source fusion. Our PhotoBench is available 1 1 1 https://github.com/LaVieEnRose365/PhotoBench . 1. Introduction Table 1 . We compare PhotoBench with existing vision-language or multimodal retrieval datasets, highlighting its unique characteristics for real-world personalized photo retrieval tasks. Quality Var. : Image Quality Variance ( e.g. , blur, noise); Near-dup. : Near-duplicate burst shots. Dataset Query Dimensions Image Dimensions One-to-Many Unmatched Narrative Personalized Multi-Source Reasoning Volume Personalized Metadata Temporal Quality Var. Near-dup. Candidates MSCOCO_t2i ( 19 ) ✗ ✗ ✗ ✗ ✗ ✗ 500K ✗ ✗ ✗ ✗ ✗ 123K Flickr30k ( 29 ) ✗ ✗ ✗ ✗ ✗ ✗ 16K ✗ ✗ ✗ ✗ ✗ 31K Winoground ( 36 ) ✗ ✗ ✗ ✗ ✗ ✓ 800 ✗ ✗ ✗ ✗ ✗ 2 INQUIRE ( 39 ) ✓ ✗ ✗ ✗ ✗ ✓ 250 ✗ ✓ ✓ ✓ ✓ 5M VisDial ( 7 ) ✗ ✗ ✓ ✗ ✓ ✓ 1.2M ✗ ✗ ✗ ✗ ✗ 100 VisualNews ( 20 ) ✗ ✗ ✓ ✗ ✓ ✓ 1.1M ✗ ✓ ✗ ✓ ✓ 1M Wiki-SS-NQ ( 24 ) ✓ ✓ ✗ ✗ ✓ ✓ 10K ✗ ✓ ✗ ✓ ✗ 1M PhotoBench ✓ ✓ ✓ ✓ ✓ ✓ 1.1K+ ✓ ✓ ✓ ✓ ✓ 1K Personal photo galleries have evolved from static storage into the primary repository of human memory. Unlike curated web-scale datasets ( e.g. , LAION ( 31 ) or MSCOCO ( 19 ) ) where images are isolated snapshots of visual content, a personal album is a living, ecological archive . It is temporally continuous, socially entangled, and deeply personalized. In this context, user queries are not merely simple visual descriptions ( e.g. , a black dog ), but intent-driven requests anchored in heterogeneous signals, such as specific events, social relationships, or spatial-temporal constraints ( e.g. , the dinner with my parents before the flight ). Consequently, effective retrieval requires not merely visual matching, but multi-source reasoning to fuse visual perception with user-specific context. Despite the rapid progress in multimodal retrieval, existing benchmarks are generally non-personalized and fail to capture this ecological complexity, as summarized in Table 1 . We identify two critical limitations in current research: • Lack of Ecological Fidelity (Image Gap) . Benchmarks like MSCOCO ( 19 ) or Flickr30k ( 29 ) focus on web-scraped, context-isolated images. They lack the temporal continuity and rich metadata (timestamps, GPS, identity graphs) inherent to personal albums, rendering them unsuitable for testing temporal- or social-based complex reasoning. • Shallow User Intent (Query Gap) . Datasets such as INQUIRE ( 39 ) or VisualNews ( 20 ) often rely on descriptive captions that map directly to visual content, which is usually sparse, incomplete one-to-one mapping. They fail to capture the multi-source entanglement and evolving user intent of real-world queries, where visual signals must be fused with non-visual constraints ( e.g. , specific time or social role) to resolve the ambiguity. To this end, we introduce PhotoBench , a diagnostic benchmark explicitly designed to shift the field of multimodal retrieval from visual matching to personalized intent-driven reasoning . Unlike prior efforts, PhotoBench is constructed from authentic, personal albums, preserving the natural noise, burstiness, and metadata headers of real-world photography. We employ a rigorous multi-source profiling to model each photo not merely as pixels, but as an information union of visual semantics \mathcal{V} , spatial-temporal metadata \mathcal{M} , social identity \mathcal{F} , and temporal events \mathcal{E} . Furthermore, we conduct intent-driven query synthesis by conditioning query generation on the multi-source information and the user’s life trajectory. In this way, we reconstruct the latent motivation behind the visual photos. Finally, through exhaustive ground truth mining and verification , we produce complex queries associated with a comprehensive ground truth set, which necessitates cross-modal reasoning in heterogeneous, personalized contexts. Moreover, we also synthesize zero-ground-truth queries to evaluate a retrieval system’s rejection capability to resist the user’s “false memory”. Evaluating on SOTA retrieval models and systems, PhotoBench exposes critical architectural failures that remain hidden in standard benchmarks. Our experiments reveal two critical phenomena: • Modality Gap . Unified embedding models ( e.g. , VLM2Vec ( 25 ) collapse when queries require precise non-visual constraints (Metadata or Face), revealing that they function primarily as visual similarity calculators rather than holistic multi-source reasoners. • Source Fusion Paradox . While agentic retrieval systems equipped with external tools outperform embedding models, they exhibit a non-linear performance degradation as query complexity increases. We find that strong single-source capabilities do not automatically translate to reliable multi-source fusion, highlighting a fundamental bottleneck in tool orchestration and constraint satisfaction for complex personalized photo retrieval. These findings indicate that the next frontier in personal multimodal retrieval, especially for photo album scenarios, lies not only in establishing stronger unified embedding models, but also in developing robust and lightweight agentic reasoning systems capable of traversing the modality gap and resolving the source fusion paradox. We believe that our PhotoBench could serve as the key testbed for this evolution. In summary, our main contributions are: • We introduce PhotoBench , the first multimodal retrieval benchmark derived from authentic, metadata-rich personal albums. Through multi-source profiling for each image, it provides the dense context necessary to evaluate complex reasoning on multi-source personalized information beyond visual matching. • We propose intent-driven query synthesis , which is a generalized methodology for personalized multimodal retrieval query generation. It synthesizes narrative yet complex queries rooted in users’ life trajectories, followed by exhaustive ground truth mining for comprehensive evaluation. Moreover, zero-ground-truth queries are also introduced to evaluate the system reliability. • Experiments on PhotoBench demonstrate that current retrieval models and systems struggle to fulfill the personalized multi-source photo retrieval task. By identifying the modality gap and source fusion paradox , we point out a critical direction for personalized multimodal retrieval, especially for photo album scenarios, i.e. , shifting from unified embedding-centric paradigms to robust and lightweight agentic retrieval systems. 2. Related Work Multimodal Retrieval Benchmarks. Multimodal retrieval has moved beyond object