Internal APIs Are All You Need Shadow APIs, Shared Discovery, and the Case Against Browser-First Agent Architectures

Lewis Tham, Nicholas Mac Gregor Garcia, Jungpil Hahn · 2026

Read Internal APIs Are All You Need Shadow APIs, Shared Discovery, and the Case Against Browser-First Agent Architectures by Lewis Tham, Nicholas Mac Gregor Garcia, Jungpil Hahn (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-00694.txt.

Introduction

Tham and colleagues argue that internal APIs, not browser-first design, should power the agentic web. Most sites are built for human browsers, not agents. Agents browse pages, inspect document models, and decode routes, a slow, brittle process done over and over. Every modern site exposes internal APIs behind its interface, first-party endpoints powering its functions. Unbrowse, a shared route graph, turns route discovery by browsers into an index of callable first-party interfaces. It learns routes passively from web traffic and serves saved routes by API calls.

Catalog record

CREATORLewis Tham, Nicholas Mac Gregor Garcia, Jungpil Hahn
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE53972
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.00694

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Internal APIs Are All You Need Shadow APIs, Shared Discovery, and the Case Against Browser-First Agent Architectures Lewis Tham, Nicholas Mac Gregor Garcia, Jungpil Hahn Abstract Autonomous agents increasingly interact with the web, yet most websites remain designed for human browsers — a fundamental mismatch that the emerging “Agentic Web” must resolve. Agents must repeatedly browse pages, inspect DOMs, and reverse-engineer callable routes — a process that is slow, brittle, and redundantly repeated across agents. We observe that every modern website already exposes internal APIs (sometimes called shadow APIs ) behind its user interface — first-party endpoints that power the site’s own functionality. We present Unbrowse, a shared route graph that transforms browser-based route discovery into a collectively maintained index of these callable first-party interfaces. The system passively learns routes from real browsing traffic and serves cached routes via direct API calls. In a single-host live-web benchmark of equivalent information-retrieval tasks across 94 domains, fully warmed cached execution averaged 950 ms versus 3,404 ms for Playwright browser automation (3.6 × \times mean speedup, 5.4 × \times median), with well-cached routes completing in under 100 ms. A three-path execution model — local cache, shared graph, or browser fallback — ensures the system is voluntary and self-correcting. A three-tier micropayment model via the x402 protocol charges per-query search fees for graph lookups (Tier 3), a one-time install fee for discovery documentation (Tier 1), and optional per-execution fees for site owners who opt in (Tier 2). All tiers are grounded in a necessary condition for rational adoption: an agent uses the shared graph only when the total fee is lower than the expected cost of browser rediscovery. Keywords: Internal APIs, Shadow APIs, First-party APIs, Agentic Web, Autonomous agents, Web agents, Shared route graph, API discovery, Discovery cost amortisation, Route-level economics, Micropayments, x402, Agent economies 1. Introduction The web was built for humans to navigate pages. Agents do not want pages — they want actions. Today, a web-capable agent repeatedly pays a discovery tax to interact with any website: it must open the site, inspect the DOM, click through the UI, infer requests, retry when the workflow breaks, and spend LLM tokens reasoning about page state. This process is not only expensive but redundantly so, as different agents re-derive the same workflows on the same sites, each paying the full discovery cost independently. Current approaches to web interaction fall into two categories, each with significant limitations: 1. Browser automation: Systems like WebVoyager [ 1 ] , AutoWebGLM [ 2 ] , and BrowserAgent [ 3 ] use LLMs to navigate and interact with web interfaces through browser actions. While effective for human-designed workflows, these approaches are inherently slower than direct API access, brittle (breaking when the UI changes), and computationally expensive; Song et al. [ 7 ] show that agents with API access outperform browsing-only agents on realistic web benchmarks. 2. Official APIs: Where available, APIs provide clean machine interfaces. However, coverage is limited — most websites lack public APIs — and access often requires registration, approval, and compliance with rate limits. We propose a third path: Unbrowse, a shared route graph that turns repeated browser discovery into shared memory. Rather than treating every interaction as a fresh reverse-engineering exercise, Unbrowse passively learns callable interfaces from real usage and stores them in a shared index. Agents remain free to rediscover routes themselves — the shared graph competes directly against self-production and wins only when it delivers genuine surplus (Section 7 ). In this sense, Unbrowse implements a concrete layer of what Yang et al. term the Agentic Web [ 22 ] — an infrastructure where autonomous agents interact with digital services directly rather than through human-designed interfaces. This paper makes the following contributions: 1. A shared route graph architecture that passively learns callable web interfaces from real browsing traffic, transforming redundant private discovery into a collectively maintained commons. 2. A conceptual economic model grounded in a necessary adoption condition — f_{\text{route}}<c_{\text{rediscovery}} — creating a market-disciplined system with a built-in outside option. 3. A novel delta-based attribution mechanism for route-level micropayments via the x402 protocol [ 20 ] , compensating contributors in proportion to their marginal contribution to route quality. 4. Empirical evaluation across 94 domains demonstrating significant speedups over browser automation on equivalent information-retrieval tasks, with analysis of both warmed-cache and cold-start performance. We situate Unbrowse within four areas of prior work: web agents and browser automation, tool learning and API discovery, API aggregation platforms, and agent economies and communication protocols. 2. Related Work 2.1. Web Agents and Browser Automation The field of web agents has evolved from early text-based systems to modern multimodal approaches. WebGPT [ 4 ] demonstrated browser-assisted question answering, while ReAct [ 5 ] introduced a more general reasoning-and-action framework that influenced many later interactive agent designs. Contemporary systems such as WebVoyager [ 1 ] , together with benchmarks such as Mind2Web [ 6 ] , show rapid progress in web agents while also highlighting the fundamental efficiency constraints of GUI-based interaction. Song et al. [ 7 ] directly addressed this limitation in “Beyond Browsing: API-Based Web Agents,” demonstrating that hybrid agents with API access outperform pure browsing agents by over 24% on WebArena benchmarks [ 23 ] . This provides empirical validation for our core thesis: direct API-style access is superior to browser automation when available. Our contribution extends this insight by asking what happens when such access can be collectively discovered and shared, rather than requiring pre-documented APIs. 2.2. Tool Learning and API Discovery The integration of external tools into LLM workflows has emerged as a critical capability. Toolformer [ 8 ] pioneered self-supervised tool learning, while subsequent work has explored hierarchical tool retrieval [ 9 ] , natural-language tool calling interfaces [ 10 ] , and automatic tool generation [ 11 ] . Gorilla [ 29 ] demonstrated that LLMs fine-tuned on API documentation can generate accurate API calls, and ToolBench [ 30 ] introduced a large-scale benchmark for tool-augmented LLMs spanning over 16,000 real-world APIs. Prior work in network traffic analysis — tools such as mitmproxy [ 12 ] and HAR-to-OpenAPI converters built on the W3C HAR specification [ 13 ] — has long enabled developers to inspect and document internal APIs. Our approach extends both lines of work. The novelty lies not in traffic capture per se, but in the synthesis of passive private observation into a shared index: where prior tools produce developer artefacts for human use, Unbrowse produces a collectively maintained index of callable interfaces for autonomous agent consumption. 2.3. API Aggregation Platforms Commercial API marketplaces such as RapidAPI and Postman’s public workspace have demonstrated the value of centralised API discovery. RapidAPI aggregates over 40,000 official APIs behind a unified gateway with standardised authentication and billing. Postman’s public API network provides searchable collections of documented endpoints. However, these platforms are limited to official , publicly documented APIs — they cannot address the long tail of websites that expose no public API. Our approach differs in two respects: routes are discovered automatically from browser traffic rather than manually submitted by API providers, and the graph covers internal APIs that were never intended for public consumption but are nonetheless callable. In parallel to shared discovery

Read the complete text