Supercharging Federated Intelligence Retrieval
Dimitris Stripelis, Patrick Foley, Mohammad Naseri, William Lindskog-Münzing · 2026
Read Supercharging Federated Intelligence Retrieval by Dimitris Stripelis, Patrick Foley, Mohammad Naseri, William Lindskog-Münzing (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-25374.txt.
Introduction
Stripelis and co-authors federate document search so private stores never pool their text. Standard retrieval-augmented generation assumes one central pile of documents. Their Flower-based system searches each silo on site. Merging and answer writing run inside an attested, sealed compute space. That keeps remote model use private even if the server is nosy or breached. A cascade trick can also call a public third-party model for extra context. It does so without leaking the private material.
Catalog record
| CREATOR | Dimitris Stripelis, Patrick Foley, Mohammad Naseri, William Lindskog-Münzing |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 11777 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.25374
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Supercharging Federated Intelligence Retrieval Dimitris Stripelis, Patrick Foley, Mohammad Naseri, William Lindskog-Münzing Abstract RAG typically assumes centralized access to documents, which breaks down when knowledge is distributed across private data silos. We propose a secure Federated RAG system built using Flower that performs local silo retrieval, while server-side aggregation and text generation run inside an attested, confidential compute environment, enabling confidential remote LLM inference even in the presence of honest-but-curious or compromised servers. We also propose a cascading inference approach that incorporates a non-confidential third-party model (e.g., Amazon Nova) as auxiliary context without weakening confidentiality. \workshoptitle Presented at the Amazon Trusted AI Symposium 2026, New York, NY, USA 1 Introduction Retrieval-Augmented Generation (RAG) improves factual grounding and domain adaptability of LLMs by conditioning generation on external knowledge. However, RAG assumes centralized document access - a loose assumption that fails in practice, since organizational data is distributed across data silos and cannot be aggregated centrally due to regulatory, proprietary, or privacy constraints. Federated RAG (FedRAG) mitigates this limitation by performing retrieval locally within each silo and aggregating the results on the server to construct a unified generation context 4 . This approach enables cross-silo knowledge access while preserving data locality, making it well-suited for sensitive domains such as healthcare and finance. However, most existing FedRAG pipelines offer limited security guarantees, as retrieved documents are exchanged in plaintext and can be exposed to honest-but-curious or compromised servers. In this work, we present a secure FedRAG architecture built on Flower 3 that uses confidential computing to protect federated aggregation and generation within a trusted execution environment (TEE), with silo participation enforced via attestation and authorization 1 . Unlike prior FedRAG approaches based on fine-tuning 9 , secure inference 1 , or encrypted retrieval and communication 1 ; 20 ; 11 , we enable confidential remote LLM inference via Flower Confidential Remote Compute (CRC) 7 , a service running LLMs remotely with confidential computing. We also introduce a cascading inference approach that integrates a non-confidential third-party model (AWS Nova) as an auxiliary probabilistic knowledge source without compromising end-to-end confidentiality. In summary, our contributions are: • a secure Federated RAG system for cross-silo retrieval without data centralization • a confidential server-side aggregation and text generation using TEEs • a mixed-trust cascading inference mechanism integrating third-party models • an end-to-end protected remote LLM inference pipeline using Flower CRC 2 Related Work & Background Federated Learning. Federated Learning (FL) enables collaborative model training across decentralized data silos without requiring raw data sharing 12 . While FL has been widely studied for model training and fine-tuning, its application to inference-time retrieval and generation has received comparatively less attention. Flower 3 provides a flexible framework for federated workloads beyond training, including secure aggregation and orchestration across heterogeneous clients. Confidential Compute. Confidential computing leverages hardware-based Trusted Execution Environments (TEEs) to protect code and data during execution, even from privileged system software 6 . Recent systems integrate TEEs into distributed ML pipelines to protect training 2 , inference 1 , and aggregation. However, many approaches focus on securing individual stages rather than end-to-end retrieval and generation pipelines. Flower CRC 7 extends confidential computing to remote LLM inference, enabling large-scale models to be executed with strong confidentiality guarantees. Federated RAG. Federated RAG (FedRAG) decentralizes retrieval by executing document search locally at each silo and aggregating retrieved results at a central server 4 . Prior work explores federated fine-tuning 9 , encrypted retrieval and communication 20 ; 11 , and secure aggregation 1 . However, most existing systems either expose retrieved documents in plaintext at the server or do not protect the generation step itself. In contrast, our approach combines federated retrieval with confidential aggregation and generation, while additionally supporting mixed-trust cascading inference with external LLM providers. Threat Model. We assume an honest-but-curious or compromised server operator and infrastructure provider, while trusting hardware-enforced TEEs and remote attestation mechanisms 6 ; 15 . Clients (data silos) do not trust each other and never share raw documents, following standard federated learning assumptions 12 ; 10 . Network adversaries may observe, replay, or tamper with communication but cannot break standard cryptographic protections such as TLS 14 . We follow prior confidential computing work in assuming the TEE implementation is correct and that side-channel attacks are out of scope 18 ; 5 . 3 Methods For every query, the server, operating inside a TEE, broadcasts the query to a set of verified clients/silos. Each client retrieves the top- k relevant documents from its local private document store and returns them to the server along with their retrieval scores. The server aggregates the results, merges and re-ranks the retrieved documents, and constructs an augmented context that is passed to the LLM. On the client side, retrieval is performed using a local FAISS index built over each client’s document corpus. The server merges the results into a single ranked list by sorting based on the retrieval score, computed as 1/(k\-{RRF}+doc_{i}+1) , where k\-{RRF} represents the value of the Reciprocal Rank Fusion and doc_{i} the document index. Figure 1 illustrates the overall FedRAG pipeline. Figure 1: FedRAG with confidential server-side aggregation and various inference options. In our setting, we evaluate three inference modes (server always operates inside a TEE): • Standalone Inference: The augmented query is processed by a server-hosted LLM, which generates a response using the retrieved documents as context. • Cascading Inference: The server-hosted LLM generates the final response using both the retrieved documents and an auxiliary answer obtained from a non-confidential third-party provider, which is treated as an additional contextual source. • Confidential Inference: The augmented query is processed by a large-scale LLM running inside Flower CRC, ensuring that both prompts and context remain protected during inference. 4 Evaluation Experimental Settings. We use the SmolLM 1.7B Instruct model as the server-hosted LLM. In the case of the cascading mode, the external 3rd party model was AWS Nova Micro, and in Flower CRC, we use the Qwen3 235B model. The entire server-side workload is CPU-bound; no GPU is employed to deploy SmolLM. In all experiments, the top- k value is set to 8 and the reciprocal rank fusion parameter k\-{RRF} to 60. The federation consists of four clients, each maintaining an independent document corpus: PubMed 13 (23.9M snippets), StatPearls 16 (301k snippets), Textbooks 8 (126k snippets), and Wikipedia 17 (29.9M snippets); sourced from the MedRAG Toolkit 19 . Evaluation is performed on the MIRAGE benchmark 19 : PubMedQA (501 Y/N/maybe questions), BioASQ (619 Y/N questions), and MedQA (1272 A/B/C/D selection questions). Table 1 reports performance and mean end-to-end query time. Mode Dataset PubMedQA BioASQ MedQA Time Accuracy Time Accuracy Time Accuracy Standalone Inference 43s 0.32 44s 0.81 43s 0.39 Cascaded Inference 48s 0.45 45s 0.83 46s 0.57 Confidential Inference 25s 0.47 26s 0.82 25s 0.78 Table 1: Mean query execution time and accuracy per benchmark dataset for each inference mode. Results Analysis. Compared to the standalone inference mode, cascading inference is especially compelling: even with the lightweight Amazon Nova Micro model, the additional context provided to SmolLM yields a \sim 40% and \sim 46% improvement on PubMedQA and MedQA, respectively, indicating that more powerful Nova Pro/Premier models are a promising direction for further evaluation. This result also highlights