From Prompt to Service: An SLM-Based Agent Orchestration Gateway for AI-Driven Virtual Worlds

Louis Nisiotis, Aimilios Hadjiliasi · 2026

Read From Prompt to Service: An SLM-Based Agent Orchestration Gateway for AI-Driven Virtual Worlds by Louis Nisiotis, Aimilios Hadjiliasi (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-03557.txt.

Introduction

Nisiotis and Hadjiliasi build a gateway that routes virtual world requests to the right AI service. Generative AI users in virtual worlds send mixed requests. Each request may need a different backend model and different compute. Embedding every capability in the client hurts extensibility and maintenance. It also complicates coordination across edge and cloud machines.

Their gateway uses a small language model at the edge to classify each prompt's intent. A service registry checks and resolves the routing choice. The chosen backend is then called without changing the client. Tests run in the InterwovenXR virtual museum testbed. Compact models prove reliable as intent routers on edge hardware. Fine-tuning turns sub-billion-parameter models into low-latency routers. Pairing a fine-tuned router with a larger model for replies beats using one model for both jobs. The design lets new AI services join a virtual world cleanly.

Catalog record

CREATORLouis Nisiotis, Aimilios Hadjiliasi
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE44 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.03557

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

From Prompt to Service: An SLM-Based Agent Orchestration Gateway for AI-Driven Virtual Worlds Louis Nisiotis, Aimilios Hadjiliasi Abstract As generative AI capabilities expand, AI-driven virtual worlds face a growing architectural challenge. Users interact through in-world interfaces in multimodal ways, yet their requests demand fundamentally different AI backend models and computational resources. Embedding these capabilities directly into virtual world systems reduces extensibility, complicates maintenance, and limits the ability to coordinate services distributed across edge and cloud infrastructure. This paper presents an SLM-based Agent Orchestration Gateway , a lightweight runtime coordination mechanism that decouples a virtual world client from heterogeneous AI backends through intent-driven service routing. An edge-deployed SLM classifies the semantic intent of each user prompt, a configurable service registry validates and resolves the routing decision, and the selected backend is invoked transparently, enabling new AI capabilities to be introduced in the virtual world without modifying the client application. The gateway is implemented and evaluated within the InterwovenXR virtual museum testbed. The evaluation shows that compact SLMs can serve as reliable intent routers on edge hardware, and that task-specific fine-tuning can transform sub-billion-parameter models into practical, low-latency routers. A layered configuration pairing a fine-tuned sub-billion-parameter model as router with a larger SLM for conversational response generation is shown to be deployable on mid-range edge hardware and more efficient than delegating both responsibilities to a single model. The findings show that SLMs can support practical AI service orchestration in virtual worlds and the work contributes an evaluated architecture for scalable, extensible, and edge-supported AI interaction, enabling virtual agents become access points to distributed generative AI services. I Introduction The advances in eXtended Reality (XR) technologies, real-time rendering, distributed infrastructures, and interoperability standards have expanded the technical foundations of virtual worlds, enabling immersive interaction, persistent spaces, cloud-edge execution, and cross-platform deployment for enriched interactive experiences [ 16 , 35 ] . Within this context, there is a lot of recent research in S/LLM-powered virtual agents and generative AI systems being explored as runtime components of immersive environments. Examples include conversational avatars that support natural dialogue, user assistance, and adaptive interaction [ 32 , 22 ] , LLMs for prompting interactive worlds, guiding non-player characters, generating 3D objects, supporting procedural creation of virtual elements from natural language prompts and other [ 7 , 29 , 13 ] . In fact, the integration of generative AI into XR environments to support immersive experiences is changing the technical design of virtual worlds. Emerging AI-driven virtual environments need to coordinate several specialised services at runtime where user requests may require lightweight conversational response generation, translation, accessibility-oriented language adaptation, retrieval from a domain knowledge base, expert interpretation from a fine-tuned model, or 3D asset generation from a text-to-3D backend. These capabilities are increasingly relevant for XR environments, cultural heritage spaces, educational simulations, collaborative workspaces, and other interactive virtual world applications [ 23 , 5 , 3 , 32 ] . This shift creates a runtime orchestration problem where a virtual-world client should not need to be directly coupled to every AI service, endpoint, payload format, deployment location, and response structure. Direct integration reduces extensibility, complicates service maintenance, and makes it harder to control routing behaviour, latency, fallback handling, safety validation, and structured response delivery. The problem becomes more important when services are distributed across local machines, edge devices, institutional servers, and cloud-based generative AI systems. In such settings, the virtual world requires a coordination layer that can interpret user intent, select the correct service, and return a response in a format that the environment can process. Prior work has examined generative AI in virtual worlds through conversational agents, procedural content generation, and text-to-3D systems [ 7 , 29 , 13 , 22 ] , but less attention has been given to the runtime software architecture required to coordinate heterogeneous AI services during live interaction. This paper addresses that gap by presenting an SLM-based Agent Orchestration Gateway that operates between a virtual world client and multiple AI backends, using an edge-deployed orchestration mechanism using SLM as an intent router for service selection. The contribution of this paper lies in the design, implementation, and empirical evaluation of a routing-based orchestration architecture for AI-driven virtual worlds. The gateway mechanism is integrated and evaluated within the InterwovenXR virtual museum, where several SLMs are assessed as candidate intent routers on an NVIDIA Jetson Orin NX, examining routing accuracy, invalid-output rate, and latency. The paper further investigates a layered edge configuration driven by fine-tuned sub-billion-parameter model performing low-latency intent recognition, and combined with a larger compact SLM handling conversational response generation, to examine whether this separation offers practical advantages for edge-supported AI-driven virtual worlds. II Background and Context II-A Immersive Virtual Worlds Virtual worlds have evolved from persistent multi-user environments into complex interactive systems featuring avatar-mediated communication, persistent shared spaces, user-generated content, XR interfaces, real-time networking, spatial audio, interactive objects, and intelligent agents among other features [ 8 , 18 , 23 ] . In fact, virtual worlds have found their way into real life contexts beyond socialisation and entertainment, for example in education with immersive simulation, experiential learning, virtual tutoring, and collaborative activities [ 33 , 6 , 9 ] , industrial contexts with digital twins, simulations, remote maintenance, training, and cyber-physical monitoring [ 31 , 34 , 25 ] , professional workspace with meetings and virtual presentations and other, highlighting their relevance and appeal for immersive interactions. The technical maturity of virtual worlds has been strongly influenced by advances in XR hardware, real-time computer graphics, rendering, and GPU acceleration. Modern software development tools and game engines such as Unity and Unreal Engine provide mature tools for user interactions, gameplay logic development, physics, animation, lighting, networking, and cross-platform deployment, enabling the construction of high-fidelity virtual environments with interactive behaviours and support cross platform deployment. The development ecosystem has also improved through interoperability standards such as OpenXR, WebXR, glTF, and OpenUSD supporting device access, asset portability, and 3D scene exchange across heterogeneous platforms, reducing fragmentation in virtual world development, interaction data and platform-specific constraints [ 35 ] . Interaction modalities have also expanded beyond keyboard, mouse, and controller input as were commonly used in interactive 3D environments. Modern virtual worlds integrate gaze tracking, hand tracking, gesture recognition, spatial audio, haptic feedback, motion capture, wearable sensors, and physiological sensing to support in world interactions, improving embodiment and presence, supporting richer visual fidelity, lower development barriers, and more scalable deployment across educational, cultural, industrial, and entertainment scenarios [ 23 , 35 , 14 ] . With the interest in deploying large-scale virtual environments and distributed systems leveraging edge and cloud network infrastructures increasing, there is a shift in reshaping how virtual worlds are deployed and experienced [ 2 , 11 ] . Cloud services support large-scale persistence, shared state, asset streaming, and multi-user coordination, and edge computing brings computation closer to users and devices, reducing latency and limiting dependency on distant cloud infrastructure [ 30 , 26 ] . This is particularly relevant for AI-driven virtual worlds, where conversational agents, adaptive environments, and generated content must respond with minimal delay. Advances in 5G and emerging 6G networks further support these requirements by targeting higher bandwidth, lower latency, denser connectivity, and more reliable communication for distributed immersive

Read the complete text