Frog Soup: Zero-Shot, In-Context, and Sample-Efficient Frogger Agents

Xiang Li, Yiyang Hao, Doug Fulop · 2025

Read Frog Soup: Zero-Shot, In-Context, and Sample-Efficient Frogger Agents by Xiang Li, Yiyang Hao, Doug Fulop (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-03947.txt.

Introduction

Xiang Li, Yiyang Hao, and Doug Fulop present a 2025 paper on game-playing agents for Frogger. Reinforcement learning agents master many Atari games but train slowly per game. The field wants agents that adapt to new tasks fast. The team shows that recent reasoning language models can play Frogger with no task training at all. Their post-training came from other domains entirely.

In-context learning and reasoning effort are then varied to see what lifts scores. Last, language model demonstrations bootstrap a classic RL agent. The seeded agent learns faster and scores higher than one starting cold. Code for the work is shared in public. Agent researchers get a recipe that joins two camps: language models supply instant skill and teaching traces, while RL polishes them cheaply. Frogger's traffic timing makes it a sharp test of quick adaptation.

Catalog record

CREATORXiang Li, Yiyang Hao, Doug Fulop
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE45 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-03947

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Frog Soup: Zero-Shot, In-Context, and Sample-Efficient Frogger Agents Xiang Li, Yiyang Hao, Doug Fulop Abstract One of the primary aspirations in reinforcement learning research is developing general-purpose agents capable of rapidly adapting to and mastering novel tasks. While RL gaming agents have mastered many Atari games, they remain slow and costly to train for each game. In this work, we demonstrate that latest reasoning LLMs with out-of-domain RL post-training can play a challenging Atari game called Frogger under a zero-shot setting. We then investigate the effect of in-context learning and the amount of reasoning effort on LLM performance. Lastly, we demonstrate a way to bootstrap traditional RL method with LLM demonstrations, which significantly improves their performance and sample efficiency. Our implementation is open sourced at https://github.com/AlienKevin/frogger . 1 Introduction Atari games have proven to be valuable testing environments for RL algorithms, leading to advancements such as Deep Q Networks ( Mnih et al. 2013 ) . RL game playing agents are traditionally initialized with zero pre-existing knowledge about a specific game environment and learn to play the game through millions of interactions with the environment. Significant time and compute is often spent exploring states that will not be experienced during high scoring policies. Exploration is particularly challenging in environments that require long horizon action sequences and provide sparse rewards, such as the Atari games and real-world robotics challenges where the state space is too large to effectively sample through free-form exploration. In this paper we will explore whether pretrained general RL agents like reasoning LLMs can play Atari games and investigate ways to leverage pretrained RL agents to reduce the training samples for training smaller agents from scratch. We first explore whether the contextual understanding and reasoning capabilities of multi-modal LLMs extend to successful zero-shot game play. Next, we analyze the performance of latest reasoning LLMs that are underwent RL post-training in domains such as math and code. We explore a variety of methods for leveraging LLMs with zero-shot and in-context RL and investigate whether LLMs can accelerate trainings of RL agents from scratch. We will evaluate whether our approach can reduce the training time required to reach equivalent performance thresholds to traditional DQN self-play. If this approach to prioritizing exploration is successful then it may accelerate the training of RL agents for other discrete game environments, and we can extend this research to explore opportunities in continuous state space game environments (e.g. 3D action games) as well as simulators for real-world robotics applications. 2 Background 2.1 RL for Atari Games Existing RL methods that train from scratch often require massive number of training samples, which makes them resource intensive and limits their application to real-world domains where interaction with the environment is costly. For example, the original Nature DQN ( Mnih et al. 2015 ) trained for 50 million frames to reach human-level performance. Figure 1 : States, actions, and rewards of the Atari 2600 Frogger game environment 2.2 Atari Game Selection Traditional DQN has demonstrated strong performance in Atari games without any prior knowledge, even surpassing human players in some cases. However, certain games pose significant challenges for DQN when they include: 1. Sparse Rewards: Many games only provide rewards at the end of the episode, making it difficult for the agent to learn which actions contribute to success. Key actions often lack immediate feedback but are crucial for long-term success. 2. Exploration Challenges: Large state-action spaces make it inefficient for the agent to learn by random exploration, requiring significant time to discover meaningful trajectories. 3. Long Term Planning: Some games demand strategic foresight and an understanding of game mechanics across multiple phases, which is difficult for a DQN agent to learn without explicit memory or hierarchical decision-making. While games like Montezuma’s Revenge and Pitfall are famous for their sparse rewards and exploration challenges, the pace of those games are significantly slower than games like Pong and Breakout. After some search, we identified Frogger as a great mix of planning and fast-paced game play. Frogger features a multi-phase gameplay where road-crossing and river-crossing follow different rules, as players must avoid objects while crossing the road but must land on objects while crossing the river. Under a strict 30 seconds time limit and faced with fast moving cars and logs at every turn, the player has to make rapid low-level decisions while planning their future routes maticulously to not be hit by a car or fall into water. While moving the frog up gives a 1 point reward, the most significant rewards are sparse and difficult to reach since the player has to move the frog all the way through 12 lanes to reach the home bays. Additionally, the river section requires sophisticated multi-step planning to land the frog in one of the unoccupied home bays. 2.3 Atari 2600 Frogger Game Mechanics The objective of the Frogger game is to hop as many frogs to safety as the player can and to score the most points along the way. The frog needs to cross a road with five lanes to reach the riverbank. Then, it needs to cross a river with five lanes dotted with logs and turtles to reach one of the home bays. When all five home bays are filled with frogs, the player receives a hefty reward of 100 points and 2 points for the number of seconds remaining to encourage fast traversals. The player is given at most 30 seconds to move a frog into the home bay and the frog automatically dies on time out. Then, the player moves on to the next level with more and faster cars. Later levels also have alligators in the river and snakes on the riverbank and logs that can bite the player. There is no limit to the number of levels and the player can theoretically earn infinite points if they don’t make a mistake. Figure 2 : Zero-Shot/In-Context RL with o3-mini on Object-Centric Representation of Frogger 3 LLM Agents Two research questions we have are: ”Can LLMs play Atari games out of the box?” and ”Can LLMs learn the rule of the game through a few trial and errors”. Recent research suggests that this may be possible. Atari-GPT ( Waytowich et al. 2024 ) demonstrates that modern LLMs like GPT-4o play Atari games like Frogger. The paper highlights the LLMs’ ability to understand contextual game information and provide strategic guidance, something traditional RL agents lack. However, it was shown that the LLMs struggle with low-level controls and often performs worse than traditional RL algorithms like DQN. Digging into the cause of low performance, AtariGPT broke down game playing skills into visual, spatial, strategy, and identification subtasks. GPT-4o was shown to excel in all subtasks except spatial reasoning. In particular, Multimodal LLMs were found to have trouble accurately assessing the distance between game objects. 3.1 Spatial Reasoning of Multimodal LLM To further investigate whether latest Multimodal LLMs have sufficient spatial reasoning skills for playing Frogger, we tested two latest vision LLMs: gpt-4o-2024-11-20 and Claude-3.7-sonnet. To make sure that the models can see the frames clearly, we tested two frame size configurations: the original 210 by 160 pixels and an upsampled 1024 by 1024 pixels version. The upsampled

Read the complete text