Agent-centric Information Access
Evangelos Kanoulas, Panagiotis Eustratiadis, Yongkang Li, Yougang Lyu, Vaishali Pal, Gabrielle Poerwawinata, Jingfen Qiao, Zihan Wang · 2025
Read Agent-centric Information Access by Evangelos Kanoulas, Panagiotis Eustratiadis, Yongkang Li, Yougang Lyu, Vaishali Pal, Gabrielle Poerwawinata, Jingfen Qiao, Zihan Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-19298.txt.
Introduction
Search engines fetch documents, but a future web may hold millions of expert language models, each trained on private data and strong in one niche. The authors sketch a framework for this agent-centric access. A query would be routed to a small set of models chosen for their proven expertise, their answers gathered and merged. Unlike document search, expertise must be inferred on the fly, since no static catalog of skills can be trusted.
The hard parts include picking experts cheaply, querying them without waste, merging their replies, and resisting manipulation. To study these problems at scale, the authors propose an evaluation setup that uses retrieval-augmented generation and clustering to build and grade thousands of specialist models. They frame it as groundwork for systems that could one day coordinate millions of knowledge agents.
Catalog record
| CREATOR | Evangelos Kanoulas, Panagiotis Eustratiadis, Yongkang Li, Yougang Lyu, Vaishali Pal, Gabrielle Poerwawinata, Jingfen Qiao, Zihan Wang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 53 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-19298
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Agent-centric Information Access Evangelos Kanoulas, Panagiotis Eustratiadis, Yongkang Li, Yougang Lyu, Vaishali Pal, Gabrielle Poerwawinata, Jingfen Qiao, Zihan Wang Abstract . As large language models (LLMs) become more specialized, we envision a future where millions of expert LLMs exist, each trained on proprietary data and excelling in specific domains. In such a system, answering a query requires selecting a small subset of relevant models, querying them efficiently, and synthesizing their responses. This paper introduces a framework for agent-centric information access, where LLMs function as knowledge agents that are dynamically ranked and queried based on their demonstrated expertise. Unlike traditional document retrieval, this approach requires inferring expertise on the fly, rather than relying on static metadata or predefined model descriptions. This shift introduces several challenges, including efficient expert selection, cost-effective querying, response aggregation across multiple models, and robustness against adversarial manipulation. To address these issues, we propose a scalable evaluation framework that leverages retrieval-augmented generation and clustering techniques to construct and assess thousands of specialized models, with the potential to scale toward millions. DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06 1. Introduction Figure 1 . An illustration of a framework for querying both domain-specialized LLMs (knowledge agents) and user-specific LLMs (user agents). Users interact with the system through personalized user agents, which not only track past queries and responses to refine retrieval but also share specialized user knowledge with others. A belief model on expertise determines which knowledge agents ( K_{1},K_{2},\dots,K_{M} ) and user agents are experts on specific topics and should be queried to generate the most relevant answers. The system optimizes for cost and latency, ensuring efficient and accurate responses while minimizing unnecessary queries. This architecture enables the dynamic orchestration of expert LLMs, moving beyond document retrieval toward a multi-expert synthesis of knowledge. The rapid advancement of generative artificial intelligence has significantly reshaped the field of information retrieval (IR) ( Zhu et al., 2023 ; Allan et al., 2024 ; White and Shah, 2025 ) . Traditionally, a user-provided query prompts an IR system to align the query’s intent with a pre-existing corpus of textual resources, returning a ranked list of documents from which the user then extracts relevant information. In this paper, we envision a future beyond traditional information access, in which information is served to users through large language models (LLMs) ( OpenAI, 2022 ; Wei et al., 2022 ) . As generative AI models increasingly act as primary sources of information, this shift necessitates a new paradigm. This paradigm must address the challenges of selecting the most suitable LLMs to query and synthesizing knowledge across multiple expert models. This future is unfolding at a fast pace ( Caramancion, 2024 ) 1 1 1 https://kafkai.com/en/blog/ai-impact-on-search-engines/ , driven by the proliferation of domain-specialized LLMs tailored to increasingly narrow fields of expertise. We are moving toward a world where millions of such expert LLMs will exist, each trained on distinct knowledge domains, some overlapping, others highly specialized. This shift is already underway, with companies like BloombergGPT ( Wu et al., 2023b ) , Thomson Reuters 2 2 2 https://legalsolutions.thomsonreuters.co.uk/en/c/practical-law/now-with-generative-ai.html , Elsevier 3 3 3 https://www.elsevier.com/products/scopus/scopus-ai developing LLMs on proprietary material. This trend is set to accelerate as more content providers, institutions, and organizations build models fine-tuned to their unique datasets. As this paradigm evolves, a user’s information needs will increasingly be met not by a single system, but by querying and synthesizing responses from multiple expert LLMs, requiring careful orchestration to extract the most accurate and relevant insights. Beyond expert LLMs, we envision a future where users themselves become “experts” through their search behavior. Personalized digital assistant LLMs will continuously read, process, and synthesize everything a user has searched for and engaged with, leading to the creation of millions of individualized user-LLMs, each evolving to reflect a specific user’s knowledge, expertise, and interests. In this evolving digital ecosystem, an agent will be necessary to orchestrate queries and synthesize responses from domain-expert LLMs. However, querying all available LLMs is prohibitively expensive and inefficient. Thus, a new type of search engine is required that, given a user’s question, intelligently retrieves only the most relevant expert LLMs. This agent must dynamically determine which expert models to consult, balancing the need for accuracy with cost efficiency. Upon receiving a user’s query, it should strategically decide which LLMs to engage, ensuring the response is both comprehensive and cost-effective while minimizing redundant computations. This new paradigm presents several fundamental challenges. First, selecting relevant LLM experts by querying all available models is computationally expensive and inefficient, requiring intelligent mechanisms to identify the most relevant models based on expertise, context, and cost. Second, multi-LLM answer aggregation introduces new complexities, as responses from different expert LLMs must be synthesized, reconciled, and ranked to provide coherent and contextually accurate answers. Third, bias, trust, and adversarial manipulation in multi-LLM retrieval become pressing concerns, as LLM-generated responses may reflect underlying biases, selectively omit critical information, or be manipulated by adversarial actors. Finally, these challenges require a novel evaluation framework and the infrastructure necessary to study. Addressing these issues will be crucial in shaping the next generation of model-centric 4 4 4 In this work we use model-centric and LLM-centric interchangeably. IR. To address these challenges, we propose a framework for orchestrating domain-specialized LLMs (knowledge agents) and personalized user LLMs (user agents). As illustrated in Figure 1 , this framework structures IR as a dynamic interaction between users and a distributed network of expert models. Users interact with the system through personalized user agents, which refine retrieval strategies based on past queries, responses, and feedback. These agents maintain a belief model on expertise, dynamically assessing which knowledge agents (domain-specialized LLMs) and user agents (peers with relevant expertise) are most suitable for responding to a given query. This belief model is continuously updated based on user interactions, ensuring that information synthesis prioritizes the most reliable and contextually relevant sources. Each knowledge agent specializes in a distinct domain, accessing structured knowledge bases (KBs) or proprietary datasets. Some knowledge agents overlap in expertise, while others are highly specialized, creating a network of interoperable expert models that can be selectively queried based on the belief model’s assessment. Additionally, user agents share specialized user knowledge, allowing insights from personalized experiences to propagate through the system. To optimize cost and latency, the framework incorporates an adaptive query mechanism, reducing redundant queries and minimizing computational overhead while maintaining response quality. Instead of blindly querying all potential sources, the system selectively engages only the most relevant knowledge and user agents, balancing efficiency with accuracy. To ground our discussion, we first examine previous and current IR paradigms relevant to the multi-LLM retrieval paradigm. We then explore each of the key challenges in detail, analyzing the complexities of selecting relevant LLM experts, aggregating responses across multiple models, ensuring trust and robustness. Finally, we introduce a simplified evaluation framework for ranking relevant LLM experts as a starting point. 2. Related Work In this section we focus on two topics that bear similarities to the proposed model-centric information access paradigm, (i) previous retrieval paradigms in which multiple collections/models were combined to produce the final ranking of resources upon