CuriosAI Submission to the CASTLE Challenge at EgoVis 2026

Yuto Kanda, Hayato Tanoue, Takayuki Hori · 2026

Read CuriosAI Submission to the CASTLE Challenge at EgoVis 2026 by Yuto Kanda, Hayato Tanoue, Takayuki Hori (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-27800.txt.

Introduction

Yuto Kanda, Hayato Tanoue, and Takayuki Hori enter the CASTLE 2026 challenge of 185 multiple-choice questions over 600-plus hours of multi-view egocentric video. Their Search-Verify-Answer pipeline narrows hierarchically to a primary window, checks sub-windows under four anti-invention rules, then blends evidence with a judge. Search-Verify-Answer reaches 50 percent accuracy as the final submission, against 35 percent for the knowledge-graph approach.

Catalog record

CREATORYuto Kanda, Hayato Tanoue, Takayuki Hori
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE10170
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.27800

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

CuriosAI Submission to the CASTLE Challenge at EgoVis 2026 Yuto Kanda, Hayato Tanoue, Takayuki Hori Abstract CASTLE 2026 asks 185 185 multiple-choice questions over 600 + 600+ hours of synchronised multi-view egocentric video. We explore two approaches on top of a shared multimodal preprocessing layer (per-person timelines, speaker-resolved transcripts, multi-VLM caption ensembles, etc.). Approach A (SVA: Search–Verify–Answer) is a three-stage pipeline that hierarchically narrows to a primary window, verifies sub-windows with a VLM under four anti-confabulation rules, and fuses evidence with an LLM judge under an evidence-priority hierarchy. Approach B (TMKG: Temporal–Multimodal–Knowledge–Graph) is the contrast: it builds a temporal multimodal knowledge graph, locates a primary cell via graph search, and produces the final answer with a single grounded VLM. SVA reaches a leaderboard accuracy of 0.50 0.50 and is our final challenge submission; TMKG reaches 0.35 0.35 . 1 Introduction In this paper, we present our submission to the CASTLE Challenge at EgoVis 2026. The challenge asks 185 multiple-choice questions over 600+ hours of synchronised multi-view egocentric ( ego ) and exocentric ( exo ) footage from the CASTLE 2024 dataset 12 , coupling long-form retrieval with fine-grained multimodal verification — infeasible for a single VLM at this scale. A second obstacle is that an unconstrained video-audio specialist frequently confabulates : it re-quotes prompt context, picks an option on silent or test-pattern clips, and asserts counts without spatial grounding. These failure modes motivate the prompt-level discipline central to our design. Building on these observations, we pursue two complementary directions on top of a shared preprocessing layer. SVA (§ 3.2 ) frames retrieval as cell-indexed search and isolates the final judgement behind an explicit anti-confabulation discipline. TMKG (§ 3.3 ) is the contrast: it tests whether knowledge-graph retrieval paired with a single grounded VLM consuming the selected cell’s multi-camera video frames and bundled evidence (captions, transcripts, tags) can replace cell-indexed search and the decoupled verify–judge stack. On the leaderboard, SVA 0.50 vs TMKG 0.35 indicates that verifier discipline contributes more than altering the retrieval structure at the CASTLE 2026 scale. SVA is our final submission. 2 Task The footage covers four days of shared-living recordings from 12 participants, with 10 head-mounted ego cameras (worn per day) and five fixed exo cameras yielding 15 synchronised 4 K video streams with aligned audio. The 185 questions are four-choice; the official metric is the accuracy averaged over all questions. 3 Method Our two approaches share a common multimodal preprocessing layer (§ 3.1 ). On top of it, SVA decouples retrieval, verification, and judgement into three explicit stages, while TMKG packages observations into a temporal knowledge graph and produces the final answer from a single grounded VLM (Fig. 2 ). 3.1 Shared Preprocessed Multimodal Databases Figure 1 : Shared preprocessing substrate. Five offline lanes over the CASTLE 2024 corpus ( 15 cameras, \approx 600 h of 4 K video and audio) aligned on a common temporal axis. Both approaches consume this layer read-only; their approach-specific aggregations are described in § 3.2 (SVA) and § 3.3 (TMKG). Five preprocessing lanes are built offline from the publicly released CASTLE 2024 footage and consumed read-only by both approaches through approach-specific aggregation (Fig. 1 ). We describe the lanes in the order shown in Fig. 1 : person identity, visual captions, detected objects, audio transcripts, and action timeline. Person identity. YOLO11x detects bodies; ArcFace 6 face embeddings are matched against twelve participant centroids built from frontal reference images, with cross-camera propagation gated by OSNet 16 body re-identification requiring body–body similarity and face–centroid agreement. Visual captions. We run four captioner models in five settings: Qwen3-VL-30B-A3B-Instruct at 300 s windows (scene) and 1800 s windows (temporal narrative); InternVL3-78B 17 with a verb-focused prompt for action descriptions; Qwen3-Omni-30B 15 for joint video+audio captioning; and Qwen3-VL-235B-A22B-Thinking-AWQ 1 for reasoning-heavy captions. Detected objects. An open-vocabulary detection lane is produced by SAM 3 3 . CASTLE 2026 questions frequently reference objects outside generic ImageNet/COCO vocabularies (e.g., specific board games, kitchen equipment, holiday decorations), so open-vocabulary detection is required. Audio transcripts. WhisperX 2 (large-v3 10 ) transcribes, pyannote 8 diarises, and WeSpeaker 13 resolves speakers. On fixed exo cameras a fixed-camera consensus rule restricts speaker candidates to those corroborated by both a concurrent transcript on another camera and the identity lane. Action timeline. A per-person action timeline is summarised by Qwen3.5-35B-A3B 9 from the captions and identity lane as (time-span, verb, co-actors) tuples. These five lanes form the shared layer; their approach-specific aggregation is described in § 3.2 (SVA) and § 3.3 (TMKG). 3.2 Approach A: SVA (Search–Verify–Answer) SVA is a three-stage pipeline (Fig. 2 , top row): Search narrows the candidate cells of § 3.1 to one primary 15 -minute window; Verify extracts evidence under anti-confabulation rules; Answer fuses it with a single LLM judge. Search. SVA aggregates preprocessing into 50 (day, hour) cells, summarises each with Qwen3.5-35B-A3B, and indexes via BM25 11 + e5-large-v2 14 hybrid retrieval. A GPT-5-mini 7 reranker and 15 -min bucket scorer narrow the candidates; a final GPT-5 reasoning call picks the primary window, cross-camera supporting windows, and a tentative answer carried into Answer as a prior. Verify. The Search anchor is expanded into \approx 24 adjacent 5 -minute sub-windows across cameras, each verified by Qwen3-Omni-30B-A3B-Instruct. Unconstrained, the verifier exhibits four characteristic confabulations (re-quoting prompt context, asserting choices on silent or test-pattern clips, counting without spatial grounding, ungrounded high confidence), which we suppress with four anti-confabulation rules in the system prompt: no echo , abstain , localise , and ground . Answer. A single GPT-5 judge fuses the question, choices, ranked per-window evidence, the Search-stage tentative answer, and a Gemini-2.5-Pro 4 reasoning trace on the narrowed clip as an external prior. The judge follows an explicit evidence-priority hierarchy (OCR \succ audio quote \succ visual \succ context), deduplicates echoed quotes, and rejects counting answers that lack spatial localisation. Figure 2 : Walk-through of both pipelines on a sample question (q0168), reading the shared DB (Fig. 1 ). SVA (top): narrow \sim 50 cells, verify \approx 24 sub-windows under anti-confabulation rules, fuse with a GPT-5 judge. TMKG (bottom): build and search a temporal knowledge graph, answer with a single grounded Omni VLM. 3.3 Approach B: TMKG (Temporal–Multimodal–Knowledge–Graph) TMKG (Fig. 2 , bottom row) consumes the preprocessing lanes (§ 3.1 ) at a 5 -minute per-camera granularity and operates in three layers: Graph Construction builds and persists a temporal knowledge graph; Retrieval narrows the evidence cells via hybrid indexes and graph constraints; and Answer produces the final choice with a single grounded VLM. Graph Construction. For each (camera, 5 -minute cell) we bundle the concurrent transcript, captions, actions, objects, and summary tags into a single Observation node, and aggregate multi-view Observations that share time, location, people, actions, and visual features into a single GlobalEvent . Typed edges link Observation \to Event (evidence), Person \to Event (participation), Event \to Place/Object (location/involvement), and consecutive Events to each other ( PRECEDES ). Each (date, time) segment is persisted independently. Retrieval. A query parser extracts date, time, person, place, object, action, and intent from the question and choices. BM25 and e5-large-v2 are run over Qwen3-VL-30B-A3B-Instruct summaries of the Observations and fused via reciprocal-rank fusion 5 . Graph predicates ( PARTICIPATES_IN , LOCATED_AT , INVOLVES ) rerank relational

Read the complete text