Retrieval-Augmented Generation for Service Discovery: Chunking Strategies and Benchmarking

Robin D. Pesl, Jerin G. Mathew, Massimo Mecella, Marco Aiello · 2025

Read Retrieval-Augmented Generation for Service Discovery: Chunking Strategies and Benchmarking by Robin D. Pesl, Jerin G. Mathew, Massimo Mecella, Marco Aiello (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-19310.txt.

Introduction

Robin Pesl, Jerin Mathew, Massimo Mecella, and Marco Aiello study how to feed software service manuals to language models that must find the right service fast. Their paper appeared in two thousand twenty-five. Modern systems are stitched from many small services listed in a registry. Full manuals are too long for a model's input window.

The team tests retrieval-backed search over chopped-up manuals. They vary how the text is cut and sized, and grade how well the right endpoint is found. A new bench spans many domains to keep the test fair. A helper agent then asks only for summaries first, pulling details on demand. That two-step flow shrinks input length and lifts retrieval accuracy. Tests on both the new bench and a real-world one back the design. The work guides teams that wire services together with AI help.

Catalog record

CREATORRobin D. Pesl, Jerin G. Mathew, Massimo Mecella, Marco Aiello
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE68 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-19310

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Retrieval-Augmented Generation for Service Discovery: Chunking Strategies and Benchmarking Robin D. Pesl, Jerin G. Mathew, Massimo Mecella, Marco Aiello Abstract Integrating multiple (sub-)systems is essential to create advanced Information Systems. Difficulties mainly arise when integrating dynamic environments, e.g., the integration at design time of not yet existing services. This has been traditionally addressed using a registry that provides the API documentation of the endpoints. Large Language Models (LLMs)have shown to be capable of automatically creating system integrations (e.g., as service composition) based on this documentation but require concise input due to input token limitations, especially regarding comprehensive API descriptions. Currently, it is unknown how best to preprocess these API descriptions. In the present work, we (i) analyze the usage of Retrieval Augmented Generation (RAG)for endpoint discovery and the chunking, i.e., preprocessing, of state-of-practice OpenAPIs to reduce the input token length while preserving the most relevant information. To further reduce the input token length for the composition prompt and improve endpoint retrieval, we propose (ii) a Discovery Agent that only receives a summary of the most relevant endpoints and retrieves specification details on demand. We evaluate RAGfor endpoint discovery using (iii) a proposed novel service discovery benchmark SOCBench-D representing a general setting across numerous domains and the real-world RestBench benchmark, first, for the different chunking possibilities and parameters measuring the endpoint retrieval accuracy. Then, we assess the Discovery Agent using the same test data set. The prototype shows how to successfully employ RAGfor endpoint discovery to reduce the token count. Our experiments show that endpoint-based approaches outperform naïve chunking methods for preprocessing. Relying on an agent significantly improves precision while being prone to decrease recall, disclosing the need for further reasoning capabilities. I Introduction The specification language OpenAPI is the state-of-practice for describing interfaces for integrating systems. It contains formal elements like paths and natural language constituents such as descriptions. For integrating these systems automatically, automated service composition using Large Language Models (LLMs)has been recently proposed [ 1 , 2 , 3 ] . These approaches exploit the capabilities of LLMsto process formal and natural language input, combining them with the inherent nature of automated service composition of decoupling and independent lifecycle management. While saving on manual modeling efforts by relying on already broadly available OpenAPIs, the approaches face the challenge of limited input token length [ 3 ] . This bounds the quantity and extent of the input service description. Even for proprietary models with a large input token context, e.g., OpenAI’s GPT4 with a context size of 128,000 tokens [ 4 ] , an economic constraint emerges as these use of the models is paid in relation to the input and output token count. Therefore, a smaller prompt length is beneficial both for inserting further service documentation and reducing proprietary models’ usage costs. To address these challenges, Retrieval Augmented Generation (RAG) [ 5 ] has emerged as a promising technique. In such an approach, the external information is collected in a database, typically structured as a set of documents or document chunks. The primary goal is retrieving only a small subset of the most relevant documents or document chunks, which is then inserted into the prompt [ 5 ] . How to optimally apply RAGfor endpoint discovery is open to investigation, leading to the following research questions that we address in this paper: RQ1. How to benchmark service discovery with natural language queries across the most relevant domains? RQ2. How best to preprocess, i.e., chunk, OpenAPIs for RAGendpoint discovery? RQ3. Can LLMagents be employed to reduce token count further and improve retrieval performance? For answering RQ1 and extending our previous work [ 6 ] , we propose the novel service discovery benchmark SOCBench-D comprising natural language query q , expected endpoints e_{\text{expected}} pairs to evaluate RAGfor OpenAPI endpoint discovery thoroughly. We rely on the Global Industry Classification Standard (GICS) [ 7 ] as a leading standard for classifying industries into sectors, i.e., domains, to ensure generalizability across various domains. It provides the following domains: energy, materials, industrials, consumer discretionary, consumer staples, health care, financials, information technology, communication services, utilities, and real estate. Similar to the ToolBench approach [ 8 ] , which employs ChatGPT to create training data, we use an LLMto construct for each of the domains five services with ten endpoints each as OpenAPIs. We validate the OpenAPIs syntactically using an OpenAPI validator and semantically using another LLM. Using the services for each domain, we let an LLMcreate ten queries for a random subset e_{\text{expected}} of the endpoints, i.e., e_{\text{expected}} is the solution to q . We recheck the solution correctness using an LLMand preclude ambiguity between the queries related to the same domain by defining a similarity threshold relying on an embedding model. To reduce the influence of randomness, we create five instances of the benchmark, resulting in 5\text{ (benchmark instances)}\cdot 11\text{ (domains)}\cdot 10\text{ (queries)}=550 queries in total. Based on SOCBench-D, we compute the accuracy metrics across all domains to determine the accuracy and generalizability of the approach. To answer RQ2, we develop an OpenAPI RAG system that takes as input service descriptions. We apply various token-based and LLM-based chunking strategies to split the documentation and evaluate them based on retrieval quality. The token-based strategies process the document using a classical parser and then split the parts into equal-sized chunks. The LLM-based strategies let an LLMcreate a description, i.e., a summary or a question, for each endpoint and then use these descriptions for similarity matching. We employ mainstream open-source and proprietary embedding models for similarity matching, which can create an embedding vector for an input. The similarity between two inputs can then be determined by comparing their embedding vectors using, e.g., the cosine similarity. We evaluate the OpenAPI RAG and the chunking strategies by relying on our novel SOCBench-D benchmark and the already available RestBench benchmark for LLMsagents [ 9 ] , measuring recall and precision for each chunking strategy. We employ SOCBench-D to retrieve generalizable results across multiple domains and the RestBench benchmark consisting of the Spotify and TMDB OpenAPI descriptions and corresponding queries, each with a set of endpoints as the sample solution, for real-world applicability. To address RQ3, we propose an LLMagent called Discovery Agent . As LLMagents allow the usage of external tools, we investigate using one tool that filters and enters the LLMendpoint summaries to the prompt using RAG, while the second tool allows the retrieval of the endpoint details on demand. We resort to the same benchmarks for evaluation and measure recall and precision. As the chunking strategy, we rely on the LLM-based summary strategy with OpenAI’s text-embedding-3-large embedding model [ 10 ] . The remainder of the paper is structured as follows. First, we provide an overview of related works regarding service discovery and LLMsin Section II . Then, we present how to use RAGfor endpoint discovery and the OpenAPI chunking strategies in Section III . We introduce SOCBench-D and evaluate and discuss the OpenAPI RAG and the different chunking strategies in Section IV . Final considerations are presented in Section V . II State of the Art Service discovery has been actively investigated in the fields of networking and information systems. Next, we provide a brief review of

Read the complete text