ARGOS: Who, Where, and When in Agentic Multi-Camera Person Search
Myungchul Kim, Kwanyong Park, Junmo Kim, In So Kweon · 2026
Read ARGOS: Who, Where, and When in Agentic Multi-Camera Person Search by Myungchul Kim, Kwanyong Park, Junmo Kim, In So Kweon (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-12762.txt.
Introduction
Myungchul Kim and three colleagues turn multi-camera person search into reasoning from witness dialogue and camera topology. Witness accounts are vague and partial. An agent plans, questions, and eliminates under information asymmetry. A Spatio-Temporal Topology Graph encodes camera links and transition times. The benchmark holds 2,691 tasks in 14 scenarios. Turn-Weighted Success scores correctness and efficiency. The best agent reaches 0.383 on track two and 0.590 on track three in tests. Removing tools drops accuracy by about 50 points, and removing reasoning halves success.
Catalog record
| CREATOR | Myungchul Kim, Kwanyong Park, Junmo Kim, In So Kweon |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 87731 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.12762
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
ARGOS: Who, Where, and When in Agentic Multi-Camera Person Search Myungchul Kim, Kwanyong Park, Junmo Kim, In So Kweon Abstract We introduce ARGOS, the first benchmark and framework that reformulates multi-camera person search as an interactive reasoning problem requiring an agent to plan, question, and eliminate candidates under information asymmetry. An ARGOS agent receives a vague witness statement and must decide what to ask, when to invoke spatial or temporal tools, and how to interpret ambiguous responses, all within a limited turn budget. Reasoning is grounded in a Spatio-Temporal Topology Graph (STTG) encoding camera connectivity and empirically validated transition times. The benchmark comprises 2,691 tasks across 14 real-world scenarios in three progressive tracks: semantic perception ( Who ), spatial reasoning ( Where ), and temporal reasoning ( When ). Experiments with four LLM backbones show the benchmark is far from solved (best TWS: 0.383 on Track 2, 0.590 on Track 3), and ablations confirm that removing domain-specific tools drops accuracy by up to 49.6 percentage points. 1 Introduction Identifying a target person across a multi-camera network is a fundamental requirement in surveillance, yet existing approaches remain limited. Traditional person re-identification 26 ; 25 ; 20 relies on clear visual queries, while text-based 23 ; 18 and interactive methods 13 ; 4 ; 11 ; 10 ; 15 ; 8 use appearance descriptions alone. None of these formulations equip the system with the ability to actively plan questions or leverage spatial and temporal cues— information that witnesses routinely provide in real-world scenarios (e.g., “I saw them in the warehouse, then near the lobby a few minutes later”). Recent spatial reasoning benchmarks 5 ; 22 ; 7 and agent evaluation frameworks 12 ; 24 remain limited to single-image or general-purpose settings, leaving interactive spatial-temporal reasoning over camera networks unaddressed. We introduce ARGOS (Agentic Retrieval with Grounded Observational Search), a benchmark and agent framework that formulates person search as an active spatio-temporal reasoning problem. The agent conducts multi-turn dialogue with a witness, invokes tools grounded in a physically validated Spatio-Temporal Topology Graph (STTG), and eliminates candidates whose movements are infeasible—combining multimodal interaction, spatial grounding, and temporal reasoning in a single evaluation protocol. Figure 1 : Overview. Left: An ARGOS agent interacts with an ambiguous witness through multi-turn dialogue, combining appearance, spatial, and temporal queries. Right: The four-module agent architecture (Analyst \to Planner \to Interviewer \to Interpreter) forms an observe-think-act loop over the evaluation environment. Our contributions are: • We define interactive multi-camera person search , a task at the intersection of multi-turn witness interaction and spatio-temporal reasoning over camera networks. • We construct the ARGOS benchmark (2,691 tasks, 14 scenarios, 3 progressive tracks) with Turn-Weighted Success (TWS) as the primary metric, jointly measuring correctness and turn efficiency. • We introduce the STTG , a directed weighted graph encoding camera connectivity and transition-time statistics that serves as both the structural backbone for task construction and the agent’s grounding tool for spatial-temporal reasoning. • We provide a strong baseline agent with a four-module pipeline and eight tools, demonstrating that tool-augmented agentic interaction substantially outperforms direct LLM inference. 2 The ARGOS Benchmark Problem formulation. An agent \pi receives an initial witness statement about a target g^{*} in a gallery \mathcal{G} and engages in multi-turn dialogue with a witness simulator \mathcal{W} . At each turn the agent selects an action— 1) asking about visual attributes, 2) querying spatial location, or 3) invoking temporal reasoning—and receives a natural-language response. The agent accesses an STTG \mathcal{T} encoding camera connectivity and transition-time statistics. A task is solved when the agent identifies the unique target within a limited turn budget. Figure 2 : Left: STTG for a 16-camera factory environment. Nodes are grouped into zones by OVERLAP connectivity. Edge types: OVERLAP (blue), SOFT_ADJ (orange), TRAVEL (gray); labels show median transition time. Right: 3D camera layout with sample imagery. Spatio-Temporal Topology Graph. The STTG is a directed weighted graph \mathcal{T}=(V,E) where each node represents a camera with a zone label and sub-area description. Each edge e_{ij} carries a type \lambda_{ij}\in\{\texttt{OVERLAP},\texttt{SOFT\_ADJ},\texttt{TRAVEL}\} and transition-time statistics (t_{\min},t_{\mathrm{med}},t_{\max},n) computed from observed movements. OVERLAP edges connect cameras with shared fields of view; their connected components define zones . TRAVEL edges cover longer transitions with empirically measured times. The STTG serves two roles: the benchmark derives ground-truth tasks from it (Tracks 2 and 3), and agents receive it as an environment representation for spatio-temporal reasoning (Fig. 2 ). Three tracks. ARGOS consists of 2,691 tasks across 14 scenarios, organized into three progressive tracks sharing the same gallery of 1,273 persons observed by 16 synchronized cameras in two environments (factory and campus). Each person is annotated with 24 visual attributes extracted via vision-language models 16 . Track 1: Who (989 tasks) tests semantic perception. The agent receives a completed multi-turn dialogue and must extract attributes to filter the gallery. Track 2: Where (550 tasks) tests spatial reasoning. The witness reports a sighting in a multi-camera zone; the agent resolves the specific sub-area through spatial questions and attribute queries (oracle avg. 2.02 turns). Track 3: When (1,152 tasks) tests temporal reasoning. The witness reports two sightings at different times and locations; the agent eliminates candidates whose transitions violate STTG constraints (oracle avg. 1.89 turns). Witness simulator. ARGOS evaluation uses a deterministic witness simulator \mathcal{W} . For three observable attributes (visual gender, upper clothing color, and lower clothing color), selected for high class variation and perceptual salience 14 , \mathcal{W} returns the ground-truth value in varied natural-language templates. For other attributes it responds with uncertainty; for spatial and temporal queries it returns pre-computed responses from the ground-truth path. This design ensures reproducibility (identical witness behavior across runs) and diagnostic clarity (agent failures trace to parsing, reasoning, or strategy, rather than witness noise). Dataset construction. All three tracks share a two-stage pipeline. Stage 1 generates deterministic ground truth through algorithmic computation: information-theoretic clue selection for Track 1, zone-based spatial disambiguation for Track 2, and STTG-based temporal feasibility classification for Track 3. No language model is involved; task validity and optimal turn sequences are computed from the attribute database, camera topology, and trajectory records. Stage 2 wraps the structured ground truth in natural-language dialogues via an LLM, preventing agents from bypassing comprehension by directly matching structured labels. 3 ARGOS Agent The ARGOS Agent is a tool-augmented LLM agent that processes each turn through four modules (Fig. 1 , right): (1) Analyst queries the gallery and computes attribute elimination power over the current candidate set; (2) Planner decides the next action (attribute question, spatial query, or temporal check); (3) Interviewer executes the action via the appropriate tool; (4) Interpreter parses the witness’s response and applies filters. The agent leverages recent multimodal LLMs 1 ; 9 ; 6 as backbone reasoning engines. The agent accesses eight tools: gallery queries (T1–T2), zone structure retrieval (T3), witness interaction (T4), temporal feasibility checking via the STTG (T5), and filtering/prediction actions (T6–T8). The same tool set serves all three tracks, but the Planner must make qualitatively different strategic decisions in each. A critical design property is the information boundary: the agent does not know which attributes the witness can answer (only 3 of 21 are observable), forcing strategic decisions under uncertainty. 4 Experiments Metrics. For Track 1 the primary