FUSE: Failure-aware Usage of Subagent Evidence for MultiModal Search and Recommendation

Tushar Vatsa, Vibha Belavadi, Priya Shanmugasundaram, Suhas Suresha, Dewang SultaniaAdobe Inc.345 Park Avenue, San Jose, CA 95110 · 2026

Read FUSE: Failure-aware Usage of Subagent Evidence for MultiModal Search and Recommendation by Tushar Vatsa, Vibha Belavadi, Priya Shanmugasundaram, Suhas Suresha, Dewang SultaniaAdobe Inc.345 Park Avenue, San Jose, CA 95110 (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-02365.txt.

Introduction

Tushar Vatsa and four co authors at Adobe improve how a multimodal creative assistant searches for and recommends content. The assistant splits a user goal among subagents for layout, style, retrieval, and generation, and retrieval can fail at several points, from misreading intent to poor ranking. Sending raw images through every step is also expensive.

Their system, FUSE, replaces most raw image prompting with a compact grounded design representation, a structured JSON record of canvas elements, layout, styles, colors, and the user's selection. Seven strategies for budgeting context are compared, from full prompting to minimal zero shot use. A pipeline attribution layer turns subagent signals into a diagnosis of where a failure began, so weak stages can be fixed directly.

Catalog record

CREATORTushar Vatsa, Vibha Belavadi, Priya Shanmugasundaram, Suhas Suresha, Dewang SultaniaAdobe Inc.345 Park Avenue, San Jose, CA 95110
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-02365

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

FUSE: Failure-aware Usage of Subagent Evidence for MultiModal Search and Recommendation Tushar Vatsa, Vibha Belavadi, Priya Shanmugasundaram, Suhas Suresha, Dewang SultaniaAdobe Inc.345 Park Avenue, San Jose, CA 95110 Abstract Multimodal creative assistants decompose user goals and route tasks to subagents for layout, styling, retrieval, and generation. Retrieval quality is pivotal, yet failures can arise at several stages: understanding user intent, choosing content types, finding candidates (recall), or ranking results. Meanwhile, sending and processing images is costly, making naive multimodal approaches impractical. We present FUSE: Failure-aware Usage of Subagent Evidence for multimodal search and recommendation. FUSE replaces most raw-image prompting with a compact Grounded Design Representation (GDR): a selection-aware JSON of canvas elements (image, text, shape, icon, video, logo), structure, styles, salient colors, and user selection provided by the Planner team. FUSE implements seven context budgeting strategies: comprehensive baseline prompting, context compression, chain-of-thought reasoning, mini-shot optimization, retrieval-augmented context, two-stage processing, and zero-shot minimalism. Finally, a pipeline attribution layer monitors system performance by converting subagent signals into simple checks: intent alignment, content-type/routing sanity, recall health (e.g., zero-hit and top-match strength), and ranking displacement analysis. We evaluate the seven context budgeting variants across 788 evaluation queries from diverse users and design templates (refer Figure 3 ). Our systematic evaluation reveals that Context Compression achieves optimal performance across all pipeline stages , with 93.3% intent accuracy, 86.8% routing success(with fallbacks), 99.4% recall, and 88.5% NDCG@5. This approach demonstrates that strategic context summarization outperforms both comprehensive and minimal contextualization strategies. The narrow intent performance variance (89.5-93.3%) across variants validates that all context budgeting approaches provide substantial benefits over zero-shot baselines. We also measure p50/p95/p99 latency and normalized token cost: Context Compression delivers the best latency–cost trade-off (45% lower p95 vs. Baseline; 8 times fewer input tokens vs. Chain-of-Thought) while Chain-of-Thought provides maximal reasoning at the highest cost. Fig. 1: Multi‑stage search system flow with FUSE integration. The diagram shows three operation modes: (a) raw query, (b) superagent‑enriched prompt, and (c) superagent prompt + compact GDR. The contextual subagent performs intent analysis, content‑type routing, recall, ranking, and semantic tie‑breaking. The evaluation and attribution layer extracts stage‑wise metrics, applies heuristic and LLM‑as‑judge evaluation, localizes failures via blame distribution, and drives targeted fixes and configuration updates. I Introduction Traditional multimodal search and recommendation in creative platforms is largely non‑assistive: users condense intent into broad keywords to fetch assets, skim through presented results to select the asset appropriate for their intent, decide the modality or tool to be used, all while understanding different platform specific features. This burdens both novices and experts often leading to missed retrievals despite the platform possessing assets that might have been able to satisfy the user intent if the user experiences were more assistive. The advent of large language models has enabled customers to converse in natural language with various systems on the web, increasing their expectations from traditional search and recommendation systems to scale to natural language inputs and improve ease of use through real-time discovery assistants. Modern multimodal assistants address this by adopting agentic orchestration: a neural planner decomposes user queries and routes tasks to specialized subagents for layout, styling, retrieval, and generation [ 17 , 9 , 5 ] . Within this architecture, the search subagent is a linchpin; downstream actions cannot redeem poor retrieval. Failures can originate anywhere in the pipeline (intent, routing, recall, ranking), and naïvely sending rich multimodal context—especially images not only degrades the output quality but also significantly increases latency and token cost [ 12 , 11 ] . Vision-language models present significant overhead in terms of latency and cost, as each image must be tiled into patches (e.g., 512\times 512 ) and contributes hundreds of tokens in addition to a fixed per-image overhead [ 13 ] . Commercial APIs price vision input at roughly $10 per million input tokens and $40 per million output tokens, with high-quality image generation reaching $0.17 per image [ 8 ] . Latency is also substantial: benchmarks of GPT-4o (Vision) report over 5 seconds per 800\times 800 image, compared to hundreds of milliseconds for task-specific detectors [ 1 ] . Optimized vision-language models like FastVLM reduce but do not eliminate this gap, achieving up to 85\times faster time-to-first-token than LLaVA-OneVision while retaining accuracy [ 3 ] . These drawbacks make naive raw-image prompting impractical for production grade pipelines that must meet strict real-time SLAs and cost budgets. FUSE: Failure‑aware Usage of Subagent Evidence. We propose FUSE, a production grade recipe for multimodal search and recommendation that unifies (i) a compact Graphical Design Representation (GDR) from planner for conditioning, (ii) budgeted context policies that right‑size prompts under cost caps, and (iii) a pipeline attribution layer that detects failure modes and selects/escalates policies accordingly. We experimented with three practical modes that reflect how real users engage the system (shown in Figure 1 ): 1. Raw query \rightarrow direct search with no additional context. 2. Planner‑enriched prompt \rightarrow the neural planner expands and clarifies the user request. 3. GDR + planner-enriched prompt \rightarrow the planner’s prompt is augmented with GDR. Over these modes, FUSE applies lightweight prompting variants chosen to meet latency and token constraints: adjustable examples (none, brief, or retrieved), GDR compression for large contexts, selective reasoning triggered only by ambiguity, and two-stage processing. Policies are chosen on demand rather than using fixed configurations. A stage‑wise attribution layer converts subagent evidence into operational checks: (1) intent alignment (semantic proximity between the planner subprompt and expected intent), (2) routing sanity (content‑type confidence with fallbacks), (3) recall health (zero‑hit and top‑match strength), and (4) ranking stability . These checks serve two purposes: they enable real-time policy adjustments within budget constraints and inform offline system tuning (such as fallback mappings and re-ranking thresholds). Fig. 2: Dataset created based out of large scale human evaluation. The left image shows query length distribution with the average length of seven words per query(after stemming). The right image shows the most frequently occurring words(after stemming) in the query, those being add , background , replace , shape and image . II Related Work Within the image-rich project creation like brochures (refer Figure 3 ), search and recommendation extend beyond generic multimodal matching to incorporate template structure, visual style, and layout cues. Design-aware retrieval systems highlight the role of structured conditioning for template retrieval, while user-interactive creative assistants increasingly incorporate context signals such as selection history and semantic grouping. Despite these advances, existing systems primarily emphasize improving embedding quality or context fusion rather than systematically diagnosing where failures occur within multi-stage pipelines. Our approach addresses this gap by combining a compact Graphical Design Representation (GDR) with stage-wise failure attribution. This enables both efficient multimodal representation and systematic error diagnosis in production grade creative workflows. II-A Agent-Orchestrated Search Systems Agent-orchestrated architectures have evolved from monolithic systems toward modular, specialized components coordinated by a Planner. Early frameworks such as ReAct [ 17 ] and Toolformer [ 9 ] showed how large language models (LLMs) can interleave reasoning with external actions or teach themselves to invoke tools. Cascades [ 5 ] extended orchestration by recursively summarizing long contexts, while surveys of augmented language models highlight general strategies for tool use and memory integration. More recent production grade frameworks such as EHR-Agent [ 12 ]

Read the complete text