Outcome-Aware Tool Selection for Semantic Routers: Latency-Constrained Learning Without LLM Inference
Huamin Chen, Xunzhuo Liu, Junchen Jiang, Bowei He, Xue Liu, [6pt] vLLM Semantic Router Project · 2026
Read Outcome-Aware Tool Selection for Semantic Routers: Latency-Constrained Learning Without LLM Inference by Huamin Chen, Xunzhuo Liu, Junchen Jiang, Bowei He, Xue Liu, [6pt] vLLM Semantic Router Project (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-13426.txt.
Introduction
Huamin Chen and co-authors propose Outcome-Aware Tool Selection for semantic routers in LLM gateways. Routers pick tools in the request path, where each added millisecond compounds over millions of requests. OATS shifts tool embeddings toward the centroid of queries where each tool has historically succeeded. That offline step adds no parameters, latency, or GPU cost at serving time. It raised NDCG scores on MetaTool and ToolBench test sets. Two learned extensions helped only when outcome data were dense enough, they find.
Catalog record
| CREATOR | Huamin Chen, Xunzhuo Liu, Junchen Jiang, Bowei He, Xue Liu, [6pt] vLLM Semantic Router Project |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 48402 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.13426
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Outcome-Aware Tool Selection for Semantic Routers: Latency-Constrained Learning Without LLM Inference Huamin Chen, Xunzhuo Liu, Junchen Jiang, Bowei He, Xue Liu, [6pt] vLLM Semantic Router Project Abstract Semantic routers in LLM inference gateways select tools in the critical request path, where every millisecond of added latency compounds across millions of requests. We propose Outcome-Aware Tool Selection (OATS) , which interpolates tool embeddings toward the centroid of queries where they historically succeed—an offline process that adds no parameters, latency, or GPU cost at serving time. On MetaTool (199 tools, 4,287 queries), this improves NDCG@5 from 0.869 to 0.940; on ToolBench (2,413 APIs), from 0.834 to 0.848. We also evaluate two learned extensions: a 2,625-parameter MLP re-ranker and a 197K-parameter contrastive adapter. The MLP re-ranker hurts or matches baseline when outcome data is sparse relative to the tool set; the contrastive adapter provides comparable gains on MetaTool (NDCG@5: 0.931). All methods are evaluated on the same held-out 30% test split. The practical takeaway is to start with the zero-cost refinement and add learned components only when data density warrants it. All mechanisms run within single-digit millisecond CPU budgets. 1 Introduction LLM serving infrastructure increasingly uses semantic routers —lightweight proxies that inspect requests and make routing decisions before forwarding them to backend model pools. These routers sit in the critical path: added latency compounds across millions of daily requests. Among their functions—model selection, prompt-length-based pool assignment, rate limiting—one of the most consequential for agentic applications is tool selection : choosing which tools to attach to a request before it reaches the LLM. 1.1 The Latency–Accuracy Tradeoff Tool selection in a router must satisfy a hard constraint: Complete within single-digit milliseconds, without GPU resources or LLM inference. At 10,000 requests/second with a 5 ms budget, tool selection alone consumes 50 CPU-seconds per second. Any method requiring LLM inference—even on a small model—exceeds this budget by orders of magnitude. Table 1 shows that all OATS methods stay within this budget on CPU. Table 1 : Cost of tool selection mechanisms. Latency is per-request p50 on 2,413 tools (ToolBench). All methods run on CPU. Method Latency Parameters GPU Viable at (ms) Required 10K rps? BM25 (lexical) \sim 7 0 No ✓ Static embedding \sim 5 22M † No ∗ ✓ OATS-S1 (offline) \sim 4 22M † No ∗ ✓ OATS-S2 (+ MLP) \sim 5 22M + 2.6K No ∗ ✓ OATS-S3 (+ adapter) \sim 5 22M + 197K No ∗ ✓ † Embedding model (all-MiniLM-L6-v2); runs on CPU at batch throughput. ∗ CPU inference; GPU optional for higher throughput but not required. Figure 1 illustrates the architecture difference. In the LLM-based approach, a dedicated orchestrator runs full inference on every request—requiring GPU resources and adding hundreds of milliseconds. In the semantic router approach, selection is a lightweight CPU operation, but it relies on static similarity with no learning. OATS preserves the fast path while incorporating outcome feedback through offline loops. (a) LLM-based tool selection User Router (proxy) LLM Orchestrator 8B–70B params LLM Serving Response + tools 500–2000 ms, GPU 200–2000 ms 700–10,000 ms (b) OATS semantic router (ours) User Semantic Router (CPU) embed \to sim \to re-rank \to top- K LLM Serving Response query + tools 3–7 ms, CPU only 200–2000 ms Offline Learning Stages 1–3 Outcome Logs outcomes updates Figure 1 : (a) LLM-based tool selection requires a GPU-bound orchestrator in the request path (500–2,000 ms). (b) OATS performs selection on CPU in the router (3–7 ms), with all learning offline. LLM serving latency is the same in both cases. 1.2 Limitations of Static Embedding Similarity The standard approach in production routers is static embedding similarity : embed each tool’s description once, compute dot products against the query embedding at request time, and return the top- K . This satisfies the latency constraint but has clear accuracy limitations: 1. Description quality bottleneck. Tool descriptions are written by developers once and never updated. A poorly worded description permanently under-selects an otherwise useful tool. 2. No outcome feedback. The system cannot learn that Tool A consistently produces better outcomes than Tool B for a given query class, even after processing millions of requests. 3. Single-point representation. Each tool is a single embedding vector, which cannot capture tools that serve multiple distinct use cases in different regions of the embedding space. 4. Lexical bias. Embedding similarity rewards surface-level textual similarity between prompts and descriptions rather than functional relevance. Our experiments confirm this: on ToolBench, BM25 (pure lexical) achieves NDCG@5 of 0.853, exceeding dense embedding retrieval at 0.834. These limitations leave room to improve accuracy within the existing latency budget. RL-based tool orchestration ( Jin et al. 2025 ; Li et al. 2025 ; Qian et al. 2025 ; Wang et al. 2025 ; Su et al. 2025 ) has shown that outcome-aware selection outperforms static methods—but it relies on LLM inference and multi-turn rollouts, which are incompatible with router-level latency constraints. 1.3 Our Approach We propose Outcome-Aware Tool Selection (OATS) . The core idea is straightforward: interpolate tool embeddings offline toward the centroid of queries where they succeed . This takes the same idea as RL-based orchestration—learning from outcomes—but implements it as an embedding transformation that adds no parameters, latency, or LLM inference at serving time. Core mechanism: Embedding-Space Outcome Refinement. For each tool, OATS collects queries where it was selected and succeeded vs. failed, then moves the tool’s embedding toward the positive-outcome centroid and away from the negative-outcome centroid. The refined embeddings replace the originals in the tool database as a periodic batch job. At serving time, the path is unchanged: embed the query, compute similarities, return top- K . On MetaTool (199 tools), this improves NDCG@5 from 0.869 to 0.940 with no added serving cost. Appendix A walks through real examples where the refinement corrects specific selection errors. Ablations: learned components. We also evaluate two extensions to understand when added complexity pays off: 1. Learned Re-Ranking (2,625 parameters). A small MLP re-scores candidates using outcome-derived features. Adds < 0.5 ms. 2. Contrastive Embedding Adaptation (197K parameters). Reshapes the embedding space via hard-negative contrastive learning. Deployed as a drop-in model replacement. The MLP re-ranker can actually hurt performance when outcome data is sparse—a result that has direct implications for deployment (Section 6 ). 1.4 Contributions • An outcome feedback mechanism (Section 4.1 ) that improves tool selection without adding parameters, latency, or GPU cost at serving time. • A cost-aware framework (Section 3 ) that separates the learning and serving concerns, formalizing the latency–accuracy tradeoff. • Experiments on ToolBench (2,413 APIs) and MetaTool (Section 6 ) showing when refinement suffices and when learned components add value, including a data-to-tool ratio threshold below which learned re-ranking hurts. 2 Related Work Tool-Use in LLMs. Tool learning has become a standard way to extend LLM capabilities. Toolformer ( Schick et al. 2023 ) showed that models can learn tool use in a self-supervised manner. ToolLLM ( Qin et al. 2024 ) scaled to 16,000+ real-world APIs and introduced the ToolBench benchmark. MetaTool ( Huang et al. 2024 ) benchmarks the selection decision—whether to use tools and which to use—and finds that most LLMs still struggle with it. Tool Retrieval and Reranking. As tool sets grow, retrieval becomes necessary before selection.