RAGnaroX: A Secure, Local-Hosted ChatOps Assistant Using Small Language Models
Benedikt Dornauer, Mircea-Cristian Racasan · 2026
Read RAGnaroX: A Secure, Local-Hosted ChatOps Assistant Using Small Language Models by Benedikt Dornauer, Mircea-Cristian Racasan (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-03291.txt.
Introduction
Dornauer and Racasan introduce RAGnaroX, a ChatOps helper that runs fully on cheap hardware with a fully open, on-premise Rust stack. Its modular intake, hybrid retrieval, and function calling support flexible yet secure use without outside firms. Tests show strong accuracy with low resource use, including 0.90 context precision on single-hop questions at about two and a half seconds per request.
Catalog record
| CREATOR | Benedikt Dornauer, Mircea-Cristian Racasan |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 15583 |
| LICENSE | http://creativecommons.org/licenses/by-sa/4.0/ |
| RIGHTS | CC BY SA |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.03291
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
RAGnaroX: A Secure, Local-Hosted ChatOps Assistant Using Small Language Models Thanks: This work was carried out within the ITEA4 GENIUS project, funded by FFG (grant 921318). Benedikt Dornauer, Mircea-Cristian Racasan Abstract This paper introduces RAGnaroX , a resource-efficient ChatOps assistant that operates entirely on commodity hardware. Unlike existing solutions that often rely on external providers such as Azure or OpenAI, RAGnaroX offers a fully auditable, on-premise stack implemented in Rust. Its architecture integrates modular data ingestion, hybrid retrieval, and function calling, enabling flexible yet secure deployment. Our evaluation focuses on the RAG pipeline, with benchmarks conducted on the SQuAD (single-hop QA), MultiHopRAG (multi-hop QA), and MLQA (cross-lingual QA) datasets. Results show that RAGnaroX achieves competitive accuracy while maintaining strong resource efficiency, for example, reaching 0.90 context precision on single-hop questions with an average response time of 2.5 seconds per request. A replication package containing the tool, the demonstration video ( https://www.youtube.com/watch?v=cDxfuEbcoM4 ), and all supporting materials are available at https://github.com/genius-itea/RAGnaroX.git . I Introduction By 2025, more than two-thirds of companies had integrated AI into their business operations across a variety of use cases [ 16 ] . Thereby, a commonly seen technique is the enhancement of LLMs through knowledge integration, commonly referred to as Retrieval-Augmented Generation (RAG), which is increasingly combined with function-calling mechanisms. Consequently, as dependence on generative AI grows, so does the risk of vendor lock-in, particularly with major U.S.-based technology firms whose proprietary ecosystems dominate the market [ 7 ] . At the same time, China is rapidly expanding its AI capabilities and investments, positioning itself as a significant competitor to U.S. dominance [ 1 ] . Overall, the tech industry is investing heavily in frontier LLMs, with compute demand projected to increase 2.25‑fold over the next two years [ 9 ] . While these advancements are likely to enhance model quality, they are also expected to drive up usage costs. The combination of increasing external dependency and escalating costs poses a significant strategic risk to organizations. Additionally, in regulated fields such as medicine and finance, dependence on external AI providers also raises compliance concerns. External (proprietary) hosted models are often difficult to control or audit fully, exposing organizations to legal and reputational risks [ 18 ] . Given the aforementioned challenges with external, commercial providers, we developed RAGnaroX, an on-premise, auditable RAG stack that combines the advantages of one of the most secure and performant programming languages, Rust [ 2 ] , and llama.cpp [ 5 ] for quantized local inference of large and small language models (SLMs). We further integrated ChatOps, conversational agents embedded in operational workflows to directly execute system actions [ 13 ] , into RAGnaroX via the Model Context Protocol (MCP) [ 8 ] , thereby rendering retrieved knowledge operational rather than merely informative. II Conception of RAGnaroX Fig. 1: Conceptual overview of the RAGnaroX data integration and retrieval–generation pipeline for ChatOps. To reduce integration and operational costs, the hardware requirements for running RAGnaroX were kept to a minimum. Therefore, the target configuration was a commodity computer equipped with 64 GB of RAM and a 24GB VRAM graphics card NVIDIA RTX 4090. However, we expect the requirements to continue declining as the quality of SLMs improves (e.g., RTX 4060). The interoperability with existing infrastructures and the adaptability to new requirements, both in software and hardware, were factors that led to the initial decision to adopt a Rust microservice architecture that utilizes HTTP and JSON for interprocess communication. As shown in Figure 1 , RAGnaroX’s conception is organized around two main components: Data Integration Component The modular adaptability enables the integration of various data sources (e.g., GitLab, Redmine) and data types (e.g., emails, files, wiki pages, issues). In the first stage of the processing pipeline, the raw artifacts are converted to Markdown (e.g., from Textile), a format chosen because it seems to be highly LM interpretable, providing also simple syntactic markers and a rich toolset to transform to (e.g., PDF \rightarrow Markdown) [ 3 ] . During the conversion phase, redundant repeating sequences (e.g., white spaces and dashes in table headers) are removed, thereby reducing the overall document size. This, in turn, decreases the storage footprint and optimizes the chunking process, enabling faster/efficient loading. Each Markdown document is then divided by headings, preserving the document hierarchy. Each heading’s contents are separated into paragraphs, tables, lists, and code snippets, an approach already positively supported by findings by Nguyen et al. [ 12 ] . The resulting text blocks are paired with their headings and then tokenized to ensure that the text fits the context size of the embedding model (e.g., multilingual-e5-large-instruct). If the text is too large, it is divided by retaining the headings and headers for the tables, ensuring that each part of the information is placed in context, which is crucial for semantic search. Once the chunks are generated, their source and the timestamp of their generation are attached, and their dense and sparse vectors (BM25) are calculated and stored in Parquet data format. Retrieval and Generation Component Similar to the backend microservices, the RAGnaroX frontend is built in Rust. This enables the two to exchange data structures and logic. The frontend transmits the chat history to the backend, and the backend uses HTTP to reach all registered RAG source microservices and request all pertinent chunks. Every RAG source uses BM25 and semantic search (cosine similarity) to look for chunks in its embedded database. The hybrid approach to chunk retrieval is predicated on the complementary nature of both algorithms as well as their acknowledged advantages and disadvantages [ 15 ] . Duplicate entries are eliminated after every RAG source has returned its chunks. Next, a reranker SLM (e.g., bge-reranker-v2-m3) is used to reorder the remaining chunks and eliminate irrelevant information [ 15 ] . Due to the finite context size of the reranker, the chunks are grouped by their size in tokens and iteratively reranked in batches until the desired number of chunks remains. The next step, which is the foundation of ChatOps, involves using another specialized SLM (e.g., Qwen3-4B-Thinking-2507) to call registered functions via MCP. Due to constraints in the model context size, the prompt is generated iteratively. With each iteration, older chat history messages are removed until all chunks, the description of the available function calls, and the most recent messages in the chat history fit in the context. This procedure adapts to the different model context sizes and also preserves as much pertinent data as possible. Once the model decides which functions to call, the registered MCP endpoints (e.g., GitLab) are invoked, and their responses are added to the RAG chunks. The final prompt to generate the user’s answer is once again put together in the same manner as for function calling. Once the answer has been streamed token by token, the list of chunks is also transmitted to allow the user to verify the answer. II-A Methodology for RAG Evaluation The evaluation of RAGnaroX centers on the information retrieval task, with three benchmark datasets selected to highlight different challenges commonly encountered in practical deployments: (i) SQuAD v1.1 [ 14 ] : a single-hop factoid QA benchmark requiring answer extraction from a single document , (ii) MultiHopRAG [ 19