IIRSim Studio : A Dashboard for User Simulation CCS: Information systems Users and interactive retrieval CCS: Information systems Evaluation of retrieval results CCS: Human-centered computing User models CCS: Computing methodologies Simulation tools
Saber Zerhoudi, Adam Roegiest, Michael Granitzer · 2026
Read IIRSim Studio : A Dashboard for User Simulation CCS: Information systems Users and interactive retrieval CCS: Information systems Evaluation of retrieval results CCS: Human-centered computing User models CCS: Computing methodologies Simulation tools by Saber Zerhoudi, Adam Roegiest, Michael Granitzer (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-23406.txt.
Introduction
Zerhoudi, Roegiest, and Granitzer present IIRSim Studio, a web workbench for user simulation in information search. Existing kits are code-centric libraries needing heavy setup, limiting uptake, while the gap is infrastructure linking design, execution, and sharing. The workbench offers visual pipeline building for all users, plus component versioning and sharing through Git-backed storage. A provenance model of experiment bundles and templates clarifies replication scope, and a shared-task workflow is shown by rerunning a Sim4IA micro-task. It ships hosted and containerised.
Catalog record
| CREATOR | Saber Zerhoudi, Adam Roegiest, Michael Granitzer |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 30905 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.23406
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
IIRSim Studio : A Dashboard for User Simulation CCS: Information systems Users and interactive retrieval CCS: Information systems Evaluation of retrieval results CCS: Human-centered computing User models CCS: Computing methodologies Simulation tools Saber Zerhoudi, Adam Roegiest, Michael Granitzer Abstract . User simulation is a valuable methodology for evaluation in Information Retrieval (IR), enabling low-cost experimentation and counterfactual analysis. However, existing simulation frameworks are primarily code-centric libraries that require substantial setup effort, which limits adoption and hinders reproducibility. The bottleneck is not the simulation engines themselves, but the lack of infrastructure connecting experiment design, execution, and sharing into a single verifiable workflow. This paper introduces IIRSim Studio , 1 1 1 https://iirsim.com a web-based workbench that addresses this gap through four contributions: (1) a visual environment for composing simulation pipelines on top of simulation frameworks, serving both novices learning simulation concepts and experts piloting large-scale experiments; (2) a component lifecycle that supports authoring, versioning, and sharing custom simulation components through Git-backed storage and runtime injection; (3) a provenance model based on experiment bundles and environment templates that makes the scope of replication explicit; and (4) a shared-task workflow, demonstrated through the re-deployment of a Sim4IA micro-task. IIRSim Studio is available as a hosted service and as a portable containerized deployment. † † footnotetext: ©ACM, 2026. This is the author’s version of the work. The definitive version was published in: Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’26), July 20–24, 2026, Melbourne, VIC, Australia . DOI: https://doi.org/10.1145/3805712.3808593 1. Introduction Interactive Information Retrieval (IIR) evaluation aims to reflect how users issue queries, examine results, click, reformulate, and stop. Cranfield-style test collections support repeatable experiments, but they often do not model the feedback loop between users and the system. Human studies capture interaction, yet they require time, resources, and can be hard to replicate. User simulation offers a controlled alternative that models sequences of user actions and supports repeated runs under fixed settings ( 5 ; 4 ; 30 ) . Simulation frameworks in IR have matured ( 14 ; 15 ) , including modular engines such as SimIIR ( 22 ; 28 ; 3 ) and recent work on conversational search simulation ( 20 ; 7 ; 1 ; 8 ; 12 ) . Despite this progress, many simulation studies still rely on local scripts, hand-built environments, and one-off configuration files. The bottleneck is not the engines themselves but the absence of infrastructure that connects experiment design, execution, and sharing into a single verifiable workflow. New users face a steep learning curve that slows adoption, and replication requires reconstructing the environment, the component versions, and other settings using incomplete records. 2 2 2 A task for which Cranfield-style evaluation has also struggled with despite numerous attempts ( 10 ; 18 ; 2 ) . This paper introduces IIRSim Studio , a resource that reduces many of the associated costs for simulation experiments using the SimIIR engine ( 3 ) . IIRSim Studio is not a new simulator model. It is a workbench and orchestration layer that supports composing a simulator pipeline, running it in a controlled container environment, inspecting logs, and exporting a versioned experiment bundle that others can rerun or scale up for more exhaustive experiments. Figure 1 . The IIRSim Studio architecture. The Frontend Workbench provides visual pipeline composition, tutorials, and a playground for prototyping. The Orchestration Backend translates pipelines into versioned experiment bundles that are executed in isolated Docker containers, either through the workbench or at scale through the API wrapper. IIRSim Studio is designed for researchers with little simulation experience that may also be less technically adept, while also aiding experienced practitioners in developing and piloting more extensive experiments. The core design goals are: (1) interactive use through a visual interface for composing simulations (Section 3.2.1 ); (2) extensibility through custom components that can be authored and versioned (Section 3.3 ); and (3) portability across hosted and local deployments (Section 3.2 ). Beginners learn core concepts through a hands-on tutorial and run short experiments in a Playground . More advanced users can add custom components (e.g., novel Generative AI-based user models), manage experiment variants, and run large batches through an API wrapper or local deployment. With support for a wide spectrum of practitioner experience levels, IIRSim Studio also helps to facilitate shared tasks and their organization through a task environment that can be pinned and shared, and with participant submissions represented as rerunnable simulator configurations ( 17 ; 10 ; 18 ; 16 ) . In doing so, IIR-based shared tasks become more accessible to all practitioners in the IR community and result in less effort for organizers who would otherwise need to build much of this infrastructure themselves. To demonstrate the utility of the resource, we re-deployed the Sim4IA Shared Task ( 25 ; 26 ) and this transforms participation in the shared task from a coding challenge into a streamlined process with minimal overhead. Our goal is to support a hosted version of IIRSim Studio for as long as possible (i.e., a minimum of at least 5 years), while simultaneously providing a fully containerized local deployment option for researchers requiring data privacy or offline access. 0 0 footnotemark: 0 , 3 3 3 https://github.com/simint-ai/simiir-studio Moreover, should the community wish to become active developers, we welcome direct contributions to the repository. 2. Related Work Several frameworks have been developed to model user interactions with different levels of control. The SimIIR framework ( 22 ; 28 ; 3 ) provides a modular architecture for simulating query generation, clicking, and stopping behaviors ( 11 ; 13 ; 21 ; 24 ) . More recently, the UserSimCRS ( 7 ; 1 ) and UXSim ( 27 ) frameworks have integrated conversational and generative AI-based user models. While these tools are powerful, they are primarily code-centric libraries. They lack a unified graphical interface for component composition, requiring researchers to understand the system architecture prior to simulation. IIRSim Studio serves as an orchestration layer for different simulation frameworks. Currently, SimIIR 3 ( 3 ) is the only supported framework due to the authors’ familiarity with it and its modularity and established adoption. In the broader IR ecosystem, platforms like TIREx ( 18 ) , repro_eval ( 9 ) , and other container-based evaluation infrastructures ( 10 ) have standardized the execution of retrieval systems through Evaluation-as-a-Service (EaaS) ( 19 ) . IIRSim Studio differs by focusing on the design of the experimental subject itself—the user simulator—rather than the system under test. This distinction matters for interactive IR experiments where the simulator logic is a primary experimental variable. IIRSim Studio complements EaaS infrastructure, with the simulator bundle acting as a rerunnable companion artifact. More broadly, general-purpose experiment management tools (e.g., MLflow, Jupyter-based workflows) support pipeline tracking and reproducibility, but they do not provide a domain-specific interface for composing behavioral models from interacting simulation components, which is the core requirement for IIR simulation research. 3. IIRSim Studio Design IIRSim Studio is designed to streamline the research cycle, from experiment design to analysis and finally to sharing. Its architecture is built to separate the user-facing interface from the underlying simulation engines, enabling flexibility and extensibility beyond the