Prominence-Stratified Failure Modes in Retrieval-Augmented Commercial Recommendation: A 37,000-Run Audit
Will Jack, Noah Lehman, Keller Maloney, Sarah Xu · 2026
Read Prominence-Stratified Failure Modes in Retrieval-Augmented Commercial Recommendation: A 37,000-Run Audit by Will Jack, Noah Lehman, Keller Maloney, Sarah Xu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-27439.txt.
Introduction
Will Jack, Noah Lehman, Keller Maloney, and Sarah Xu audit tens of thousands of AI brand recommendation runs across five prominence tiers in 19 sectors. Top brands appear almost everywhere but win only a quarter to two fifths of slots reached, while mid-market brands mark the inflection point. The smallest brands face catastrophic invisibility, with about half never surfacing at all.
Catalog record
| CREATOR | Will Jack, Noah Lehman, Keller Maloney, Sarah Xu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 57130 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.27439
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Prominence-Stratified Failure Modes in Retrieval-Augmented Commercial Recommendation: A 37,000-Run Audit Will Jack, Noah Lehman, Keller Maloney, Sarah Xu Abstract AI assistants like ChatGPT and Claude are recommendation engines, not search engines: they answer commercial queries by directly nominating brands rather than returning a list of links. Marketing to AI is therefore a broader problem than “show up in search” — positioning, content, and product fit matter as much as discoverability. We audit ≈ \approx 37,000 production runs across four model configurations and 215 commercially-framed prompts spanning 19 sectors, evaluated against a 533-brand reference catalog stratified into five prominence tiers (L1 category leaders → \to L5 regional players) sourced from external authority lists. The ladder proxies a brand’s awareness footprint within its sector, not revenue or market share. The failure mode differs sharply by tier. L1 brands appear in nearly every relevant retrieval but win only 25–41% of the recommendation slots they reach — the leverage is differentiation, not visibility. L2 challengers carry the highest conversion rates of any tier (37–52%) but lose to persona-mediated substitution on the Anthropic models. L3 mid-market brands are the inflection level: aggregate coverage drops to 88%, conversion to 34–40%, and persona effects peak. L4 specialists and L5 regional players face catastrophic invisibility — 48–52% never surface in any of the 37,000 runs. No uniform optimization recipe wins; the right marketing investment depends on where the brand sits on the prominence ladder. 1 Introduction When a buyer asks ChatGPT or Claude for the best CRM, the best observability tool, or the best payroll software for a UK SMB, the system issues web searches, retrieves a small set of authoritative-looking pages, and synthesizes a recommendation. From the buyer’s perspective the interaction looks like a list of brand names. From the brand’s perspective it is a three-stage funnel: the model must (1) retrieve a page or domain referencing the brand, (2) carry the brand from the retrieval pool into the answer text it generates, and (3) specifically endorse the brand in the final recommendation set rather than mentioning it in passing as a foil or alternative not picked. Any of these stages can fail; each represents a different bottleneck, requiring different work to fix. A growing practitioner literature on answer-engine and generative-engine optimization (AEO / GEO) [ 4 , 7 ] characterizes the aggregate response of LLM commercial recommendation to content-side interventions, often framing the brand-side problem as the AI analog of search-engine optimization — a discoverability problem. Aggarwal et al. 2024 report that statistics, citations, and quoted sources can lift visibility by up to 40% in a generic prompt corpus. Chen et al. 2025 document a systematic bias in AI Search toward “earned” (third-party) over brand-owned content. Both are headline findings that have shaped the AEO/GEO industry; both implicitly treat brand-side optimization as a single discoverability target. That target is real, but it is one of several. Frontier reasoning models read broadly across the open web, weigh candidates against each other, and synthesize a specific recommendation on the buyer’s behalf. Once a brand reaches the retrieval pool, whether it survives into the final answer depends on how the model compares it against the other candidates it found, and on how well its positioning matches the specific buyer the model thinks it is talking to — a positioning, content, and product-fit problem layered on top of discoverability. Our audit asks: does the optimal intervention depend on where the brand sits on the prominence ladder? The answer is yes, and the magnitudes are large. 1.1 Contributions This paper makes three contributions. 1. Prominence-stratified discoverability map. Across 37,000+ runs spanning four production model cells, three retrieval conditions (native web_search , neural retrieval via Exa, conventional keyword search via Brave), and 215 commercially-framed prompts, we measure aggregate and per-query brand surface rates at five prominence levels. The five-level structure was chosen to expose long-tail and regional cells that a 4-quadrant industry taxonomy (e.g., Gartner’s Magic Quadrant) cannot resolve. 2. Five-mode taxonomy of recommendation failure. Each prominence level corresponds to a distinct dominant funnel stage: L1 brands drop at S2–S3 (in retrieval or in completion, but not recommended); L2 at S2–S3, with persona-mediated substitution as the candidate failure mechanism but cleanly present only on the Anthropic cell; L3 across S1–S3 simultaneously; L4 at S1 (not retrieved and not mentioned); L5 at S1 plus geographic gating. We interpret S1 as a discoverability failure, S2 as a compellingness failure, and S3 as a positioning failure — those interpretations are defended in the Discussion. Practitioner advice that does not condition on prominence misallocates effort. 3. Per-prominence prescription framework. Drawing on the discoverability map and failure-mode taxonomy, we derive tier-specific prescriptions (defensive positioning for L1; segment-targeted positioning for L2; hybrid investment for L3; authority-list seeding for L4; region-specific authority for L5). The framework directly counters the implicit one-size-fits-all framing common in current AEO/GEO practice — and the implicit narrowing of “marketing to AI” to discoverability work alone. The audit produces conservative numbers: every reported headline proportion is bounded with a Wilson 95% confidence interval, and the few cells with insufficient sample size are reported as undersampled rather than as zero-effect. 2 Background This paper sits at the intersection of three lines of work. Generative engine optimization. Aggarwal et al. 2024 formalized GEO as an optimization paradigm, introducing GEO-bench and showing content modifications can raise LLM-mediated visibility by \sim 40%. Chen et al. 2025 extended the analysis cross-provider and documented systematic AI-Search bias toward earned media over brand-owned content. Both are aggregate findings; neither stratifies the response by where the brand sits on a prominence ladder. Long-tail and popularity bias in recommendation. That recommender systems systematically over-expose head items and under-expose long-tail content is one of the most replicated findings in the field [ 6 , 13 , 17 ] . The phenomenon predates LLMs by two decades. Mallen et al. 2023 showed that LLMs answer popular entities from parametric memory but require retrieval for long-tail; Andre et al. 2025 showed cold-start LLM recommenders amplify demographic and cultural stereotypes. Lichtenberg et al. 2024 report the counterpoint that LLM-based recommenders exhibit less popularity bias than traditional recommender baselines on standard benchmarks, suggesting the LLM-era pattern may not replicate the classical one in aggregate; our prominence-stratified data clarifies that the LLM-era picture is bimodal — L1–L2 reachability is high, L4–L5 reachability is catastrophically low — rather than a smooth popularity gradient. Our work documents the brand-side analog in retrieval-augmented commercial chat. LLM brand-recommendation auditing. The closest prior is Rienecker et al. 2026 (ChoiceEval), which audits \sim 2,000 questions across Gemini, GPT, and DeepSeek and finds provider-of-origin bias in entity preference. Practitioner studies from Fishkin 2026 and Blake and Scharf 2025 document consideration-set instability across reruns and Bing-rank correlation with ChatGPT citations respectively. Our audit operates at different scale (37,000+ runs across four model cells) and a different design dimension (prominence-stratified rather than provider-of-origin). We use the funnel-stage framing from the consumer-decision-journey literature [ 8 , 11 ] and the brand-equity / brand-salience construct from Keller 1993 , adapted from a human consumer journey to an algorithmic one where the LLM now constructs the initial consideration set the consumer used to. 3 Method 3.1 Reference brand catalog We construct a