NeuralOS: Towards Simulating Operating Systems via Neural Generative Models

Luke Rivard, Sun Sun, Hongyu Guo, Wenhu Chen, Yuntian Deng · 2025

Read NeuralOS: Towards Simulating Operating Systems via Neural Generative Models by Luke Rivard, Sun Sun, Hongyu Guo, Wenhu Chen, Yuntian Deng (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-08800.txt.

Introduction

Luke Rivard and four co-authors present NeuralOS, a learned model that acts like a computer screen and replies to mouse and key input. It joins a memory network that tracks the state of the machine with an image maker that draws each new frame.

The model trained on tapes of a real desktop system, some made by chance moves and some by smart agents doing real tasks. It drew lifelike screens, tracked clicks well, and showed apps opening as they should. It could even fake an app it had never had installed, from made up training tapes alone.

Catalog record

CREATORLuke Rivard, Sun Sun, Hongyu Guo, Wenhu Chen, Yuntian Deng
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE48 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-08800

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

NeuralOS: Towards Simulating Operating Systems via Neural Generative Models Luke Rivard, Sun Sun, Hongyu Guo, Wenhu Chen, Yuntian Deng Abstract We introduce NeuralOS, a neural framework that simulates graphical user interfaces (GUIs) of operating systems by directly predicting screen frames in response to user inputs such as mouse movements, clicks, and keyboard events. NeuralOS combines a recurrent neural network (RNN), which tracks computer state, with a diffusion-based neural renderer that generates screen images. The model is trained on a large-scale dataset of Ubuntu XFCE recordings, which include both randomly generated interactions and realistic interactions produced by AI agents. Experiments show that NeuralOS successfully renders realistic GUI sequences, accurately captures mouse interactions, and reliably predicts state transitions like application launches. Although modeling fine-grained keyboard interactions precisely remains challenging, NeuralOS offers a step toward creating fully adaptive, generative neural interfaces for future human-computer interaction systems. 1 Introduction “Chatting” with LLM feels like using an 80s computer terminal. The GUI hasn’t been invented yet, but some properties of it can start to be predicted. — Andrej Karpathy Recent breakthroughs in generative models have transformed human-computer interaction, making it increasingly adaptive, personalized, and intuitive. Historically, computing interfaces were rigid and predefined, such as command-line terminals and static graphical menus ( Engelbart 1968 ) . The emergence of large language models (LLMs) and multimodal AI systems expanded this paradigm by enabling interactions through natural language ( Radford et al. 2019 ; Brown et al. 2020 ) , images ( Ho et al. 2020 ; Lipman et al. 2022 ; Radford et al. 2021 ; Song et al. 2020b ) , and videos ( OpenAI 2024 ) . Recently, generative models have even begun simulating dynamic visual environments ( Ha and Schmidhuber 2018a ; He et al. 2025 ) , notably interactive video games ( Alonso et al. 2024 ; Feng et al. 2024 ; Oh et al. 2015 ; Valevski et al. 2024 ) . These advancements suggest a future where computing interfaces could become fully generative, dynamically adapting in real-time based on user inputs, contexts, and intentions ( Deka et al. 2017 ) . In this paper, we introduce NeuralOS, a first step toward realizing this vision. NeuralOS simulates an operating system’s graphical interface entirely using deep neural networks. By modeling the OS interface as a generative process, it directly predicts graphical frames from user input events, such as mouse movements, clicks, and keyboard interactions, without manually programmed kernels or applications. Figure 1 illustrates an example sequence generated by NeuralOS, demonstrating realistic cursor movements and window interactions predicted solely from user inputs. (a) (b) (c) (d) (e) (f) Figure 1 : Example image sequence predicted by NeuralOS, illustrating the model’s ability to simulate realistic GUI interactions. The sequence shows key frames as a user: (a-b) moves the cursor to the “Home” icon, (c-d) double-clicks to open the “Home” folder, (e) moves the cursor toward the window’s close button, and (f) clicks to close the window. Cursor positions are highlighted with red circles. Frames are generated autoregressively, conditioned on previous frames and user inputs. NeuralOS integrates two complementary neural architectures, analogous to the traditional separation between OS kernels and desktop rendering programs: a recurrent neural network (RNN) ( Hochreiter and Schmidhuber 1997 ) maintains internal computer states (such as open applications, hidden windows, and recent actions), while a diffusion-based convolutional neural renderer generates screen images. We train NeuralOS end-to-end on interaction sequences recorded from Ubuntu XFCE environments, combining randomly generated and realistic AI-generated human-like interactions. Developing NeuralOS posed several challenges. (1) Long-term state tracking was essential due to delayed interface responses (e.g., opening Firefox could take up to 30 frames); we addressed this by using an RNN-based state representation. (2) Precise cursor modeling required explicit positional encodings within our diffusion model. (3) Without pretrained encoders for GUI interactions, we developed a novel pretraining method in which the RNN outputs were pretrained via regression losses and subsequently integrated into the diffusion model via finetuning. (4) Exposure bias during inference was mitigated using scheduled sampling techniques ( Bengio et al. 2015 ; Deng et al. 2023 ; Ranzato et al. 2015 ) . (5) Extensive engineering was necessary for scalable data collection and real-time inference, leveraging parallel Docker environments and AI-generated user interactions. Experiments show that NeuralOS can generate realistic screen sequences, accurately predict mouse interactions, and reliably simulate transitions such as application openings. While computational constraints limit our model’s ability to capture fine-grained keyboard inputs precisely, NeuralOS offers a step toward neural operating systems that adapt interfaces in real time, potentially enabling users to personalize interactions through natural language or gestures instead of fixed menus. More broadly, this work suggests the exciting possibility of blurring boundaries between software applications. For example, in the future, passive media such as movies could be transformed into interactive experiences ( Yu et al. 2025 ) . These results point toward user interfaces that may eventually become fully AI-driven, highly flexible, and tailored to individual needs. Our code, pretrained model, and an interactive demo are available at https://neural-os.com . 2 Generative Modeling of Operating System Interfaces We formalize the task of simulating operating system (OS) graphical interfaces as an autoregressive generative modeling problem. At each discrete timestep t , the model predicts the next graphical frame x_{t} based on the sequence of previously observed frames x_{<t}=x_{0},x_{1},\dots,x_{t-1} and user input events a_{\leq t}=a_{1},a_{2},\dots,a_{t} up to and including the current timestep. Formally, each frame x_{t} is represented as an image tensor x_{t}\in\mathbb{R}^{H\times W\times C} , with H and W denoting image height and width, respectively, and C the number of color or feature channels. The input event a_{t} at timestep t includes cursor coordinates (x,y) , binary indicators for mouse clicks (left or right), and a binary vector indicating which keyboard keys are pressed or released. 1 1 1 We assume user inputs are discretized and associated with each graphical frame, aggregating any events occurring between frames at the subsequent timestep. The joint probability distribution of an OS graphical sequence given user inputs can be expressed as: P(x_{1:T}\mid a_{1:T};\theta)=\prod_{t=1}^{T}P(x_{t}\mid x_{<t},a_{\leq t};\theta), (1) where \theta represents the parameters of our neural generative model. Unlike standard video generation, OS simulation must respond instantly to unpredictable user inputs, often causing abrupt changes in the interface, such as when a new application is launched. This contrasts with the smooth, predictable transitions typical in video generation. As a result, the model must maintain accurate and responsive state tracking. Next, we describe the NeuralOS architecture and training strategies designed for these requirements. Time Step 1 Time Step 2 Time Step t Frame x_{1} Renderer Renderer Context RNN Frame x_{2} Renderer Renderer Context RNN \cdots Frame x_{t} Renderer Renderer Context RNN (a) High-level temporal architecture of NeuralOS. Input Embeddings U_{t-1} Lower LSTM Frame x_{t-1} Attention U_{t} Upper LSTM Cursor Position Map + Renderer Context (b) RNN structure at step t . Figure 2 : NeuralOS Model Architecture. (a) High-level architecture of NeuralOS . At each timestep, an RNN tracks the operating system’s internal state based on user inputs (cursor positions, mouse clicks, keyboard events) and previously generated frames. This state is then passed as context to a diffusion-based renderer (UNet) that generates the next graphical frame. (b) Detailed two-level RNN

Read the complete text