The Latent Bridge: A Continuous Slow–Fast Channel for Real-Time Game Agents

Bojie Li Noah Shi Pine AI University of Washington · 2026

Read The Latent Bridge: A Continuous Slow–Fast Channel for Real-Time Game Agents by Bojie Li Noah Shi Pine AI University of Washington (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-24470.txt.

Introduction

Li and Shi couple a slow, thoughtful model to a fast, reactive one, through a learned channel instead of text. A real time game agent must act within tens of milliseconds, while planning over seconds. Their testbed pairs a deliberative reasoning model, needing over a second per response, with a reactive model answering in milliseconds. Both are frozen; only the channel trains.

The standard coupling makes the slow model write text the fast model reads. Their Latent Bridge instead projects the slow model's internal residuals directly into the fast model's input embedding space. The text round trip is skipped entirely. On seven Atari games and a driving domain, the Latent Bridge matches or beats the text bridge everywhere. Large significant gains land in two games. Safe drop in behaviour covers the rest. Li and Shi's 2026 result suggests model teams should talk in vectors, not sentences.

Catalog record

CREATORBojie Li Noah Shi Pine AI University of Washington
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE82 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.24470

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

The Latent Bridge: A Continuous Slow–Fast Channel for Real-Time Game Agents Bojie Li Noah Shi Pine AI University of Washington Abstract A real-time agent for general computer use—with games as the most demanding case—must act within tens of milliseconds while still planning over seconds. These two regimes sit at opposite ends of the latency–quality tradeoff. A reasoning VLM (Qwen3-VL-8B-Thinking) deliberates effectively but requires ∼ \sim 1.5 s per response—far too slow for a 15 Hz control loop. In contrast, a reactive VLM (MiniCPM-o 4.5) acts in milliseconds but underperforms on planning-heavy tasks. We couple two frozen models of matched scale (9 B reactive, 8 B reasoning), leaving the communication channel as the sole trainable component. The standard coupling is a Text Bridge ( T T ): the slow model writes a suffix the fast model reads. We introduce a learned continuous Latent Bridge ( L L ) that projects the slow model’s residuals into the fast model’s input-embedding space in a LLaVA-style manner, avoiding any text round-trip; both are compared against Fast-Only ( F F ). On 7 Atari games and a driving domain (MetaDrive), tuning the action decoder per channel on held-out seeds, the Latent Bridge matches or beats the Text Bridge in every domain: it significantly improves two games (MsPacman + 57 % +57\% , RoadRunner + 28 % +28\% ) and is a safe drop-in elsewhere. Combining both channels interferes destructively (RoadRunner − 96 % -96\% ), so only one should be used. The benefit is highly predictable: the bridge helps if and only if slow reasoning already beats fast reaction ( T > F T\!>\!F )—the Latent and Text gains over Fast-Only move together at r = 0.93 r=0.93 . MetaDrive is the controlled negative, where the Latent Bridge is demonstrably inert because the Text Bridge adds no value. We release replay recordings and reproducible pipelines. Code: https://github.com/19PINE-AI/latent-bridge-games Website: https://01.me/research/latent-bridge-games 1 Introduction We seek agents for general computer use (GCU): read the screen, issue low-level inputs, close the loop, repeat. Its most demanding form is real-time video games on a phone or desktop, where the agent reacts to raw pixels every few tens of milliseconds while pursuing goals that need planning over seconds—two opposing regimes, fast reaction and slow deliberation, that are hard to get from one model without purpose-built training. Figure 1 : Cross-game scores with each channel at its own best decoder, selected on held-out seeds ( n=12 ; reported action-head variant per game, selected by the rule in § 2 ). The Latent Bridge significantly beats the Text Bridge on MsPacman ( +57\% ) and RoadRunner ( +28\% ) and ties on the other five, never losing. Stars: L -vs- T significance by Mann–Whitney U ( ** p<.01 , *** p<.001 ); * on a game label marks the robust-action-head variant (§ 5 ). Both variants are tabulated in Appendix A . Why couple two models rather than train one. A single model can , in principle, do both—StepFun’s open-weights Step-Audio-R1.1 [ 29 ] interleaves reasoning with speech, answering fast while it keeps thinking—but training one to do both well chases a moving target. Reasoning models [ 18 , 7 ] advance every few months: GPT-5.2 through 5.5 [ 20 ] and Claude Opus 4.5 through 4.8 [ 4 , 27 ] all shipped within about six months, in a race won on coding [ 13 ] and agentic tool use [ 31 ] , not real-time interaction. Building interaction in as its own objective—and creating the data it demands—risks diluting the raw intelligence the race optimizes for, the capability that matters most. Even frontier labs can ill afford that; and at the smaller sizes others can train, no tuning closes the gap to the state of the art. The decoupled alternative. The pragmatic alternative is to leave the best reasoning model untouched and pair it with a separate, custom-trained model for the real-time loop. Recent decoupled systems do exactly this—Thinking Machines Lab’s Interaction Models [ 28 ] couple a live interaction model to an asynchronous background reasoner, as do xAI’s Grok Voice Think Fast [ 30 ] and Pine AI’s voice agent [ 22 ] —running a fast responder in the loop and a slow reasoner in the background, coupled by streamed text or context. These are voice-centric, tightly co-designed, and largely closed. Our setting. We carry this principle to general computer use, and specifically to real-time games. The open, general-purpose models one would actually deploy come with no built-in fast/slow split: a reasoning VLM (Qwen3-VL-8B-Thinking [ 23 ] ) deliberates well but needs \sim 1.5 s per response—tens of frames late for a \sim 15 Hz loop—while a reactive VLM (MiniCPM-o 4.5 [ 21 ] ) answers in milliseconds but plans poorly: acting alone ( Fast-Only , F ) it leaves much on the table—on MsPacman, coupling in a reasoner roughly doubles the score. We therefore couple two frozen, open-source models of matched scale and study the question these systems leave implicit: how should the slow model’s deliberation reach the fast model—can a learned continuous latent channel beat the standard text coupling? Text Bridge vs. Latent Bridge. The standard coupling is the Text Bridge ( T ): the slow model writes its conclusion and the fast model reads it as a prompt suffix. We introduce the learned continuous Latent Bridge ( L ): it projects the slow model’s residual stream directly into the fast model’s input-embedding space (LLaVA-style) and prepends a few latent tokens, with no text round-trip. Both are compared against Fast-Only ( F ). Because both base models are frozen and matched in scale (9 B fast, 8 B slow), the channel —the sole learned component—is the only variable under study. The fast model acts at \sim 15 Hz, the slow model deliberates at \sim 1 Hz. We evaluate on Atari [ 5 ] (100 ms ghost-dodging meets 10 s route-planning) and the MetaDrive driving simulator as a controlled negative. Figure 2 previews all eight evaluated domains. Figure 2 : The eight evaluated domains. Seven Atari games (raw pixels, \sim 15 Hz control), spanning fast hazard-avoidance—ghosts, obstacles, enemy fire—and slower route/strategy planning, plus MetaDrive (top-down driving), our non-Atari controlled negative. Frostbite (excluded) and Pong (reported but uninformative) are not shown. What this paper shows. • Architecture matters. A first attempt (v1, 256-d cross-attention at 2 of 36 layers) converged to KL=0.004 offline yet failed at deployment; the working v2 is the LLaVA pattern—project slow residuals into the fast LLM’s 4096-d input-embedding space and prepend, so all layers attend via standard causal attention (§ 3 ). • Tuned per channel, the Latent Bridge is never significantly worse than the Text Bridge and is significantly better on 2 of 7 games (MsPacman +57\% , RoadRunner +28\% ; Figure 1 , § 4.4 ). A fixed greedy decoder (always taking the highest-probability action) flatters the Latent Bridge to 4 wins, but that edge is greedy-specific —it vanishes once actions are instead sampled (§ 4.3 ). Since the decoder is a tunable deployment hyperparameter, the fair test gives each channel its own best decoder, selected on held-out seeds; the two in fact prefer different decoders. • Using both channels at once hurts —couple

Read the complete text