APF+: Boosting adaptive-potential function reinforcement learning methods with a W-shaped network for high-dimensional games
Yifei Chen · 2025
Read APF+: Boosting adaptive-potential function reinforcement learning methods with a W-shaped network for high-dimensional games by Yifei Chen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-13557.txt.
Introduction
Yifei Chen and Lambert Schomaker extend a reward-shaping method for teaching game-playing AI in this 2025 paper. Reward shaping adds guide signals that speed up learning. Their earlier method, APF, worked when games were described by a few numbers. Raw screen pixels are far richer, so the method needed a way to shrink them first.
The team built a W-shaped network that reads a game frame and outputs two compact codes, one for the scene and one for how the scene stands out. This encoder feeds a dueling deep Q-network trained with the shaping method. Across twenty Atari games, the full system beat the plain network in fourteen games and a rival encoder in thirteen. It nearly matched a version fed with true internal game state, which the authors take as a strong sign.
Catalog record
| CREATOR | Yifei Chen |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 60 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-13557
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
APF+: Boosting adaptive-potential function reinforcement learning methods with a W-shaped network for high-dimensional games Yifei Chen Abstract Abstract Studies in reward shaping for reinforcement learning (RL) have flourished in recent years due to its ability to speed up training. Based on the potential-based reward shaping framework, our previous work proposed an adaptive potential function (APF) [ 1 ] that is used as a component module to the regular value estimation in RL algorithms. There, we showed that APF can accelerate the Q-learning with a Multi-layer Perceptron algorithm in the low-dimensional domain. This paper proposes to extend APF with an encoder (APF+) for RL state representation, allowing applying APF to the pixel-based Atari games using a state-encoding method that projects high-dimensional game’s pixel frames to low-dimensional embeddings. We approach by designing the state-representation encoder as a W-shaped network (W-Net), by using which we are able to encode both the background as well as the moving entities in the game frames. Specifically, the embeddings derived from the pre-trained W-Net consist of two latent vectors: One represents the input state, and the other represents the deviation of the input state’s representation from itself. We then incorporate W-Net into APF to train a downstream Dueling Deep Q-Network (DDQN), obtain the APF-WNet-DDQN, and demonstrate its effectiveness in Atari game-playing tasks. To evaluate the APF+W-Net module in such high-dimensional tasks, we compare with two types of baseline methods: (i) the basic DDQN; and (ii) two encoder-replaced APF-DDQN methods where we replace W-Net by (a) an unsupervised state representation method called Spatiotemporal Deep Infomax (ST-DIM) [ 2 ] and (b) a ground truth state representation provided by the Atari Annotated RAM Interface (ARI) [ 2 ] . The experiment results show that out of 20 Atari games, APF-WNet-DDQN outperforms DDQN ( 14/20 games) and APF-STDIM-DDQN ( 13/20 games) significantly. In comparison against the APF-ARI-DDQN which employs embeddings directly of the detailed game-internal state information, the APF-WNet-DDQN achieves a comparable performance. Introduction Reinforcement learning (RL) has achieved increasing attention in many fields, such as games [ 3 ] , aviation [ 4 , 5 , 6 ] , and robotics [ 7 , 8 , 9 ] . In the development of RL algorithms, a proper reward signal is an inevitable component to work on. The design of the reward signal is critical and greatly determines the success of the training procedure. Among the research on reward signals, reward shaping [ 10 , 11 ] is a practical approach to speed up the RL agent’s learning process. However, it may also lead to undesired or unsafe behaviors [ 11 ] . Therefore, Ng et al. [ 12 ] proposed the potential-based reward shaping (PBRS) framework to guarantee policy invariance after reward shaping. This means the underlying optimal policy can still be achieved when applying the reward shaping method. In the PBRS framework, a potential function \Phi , which can be any function over states, was introduced to compose the reward-shaping function F . Based on the PBRS framework, our former work [ 1 ] proposed an adaptive potential function (APF) in the sense that the agent learns the potential function concurrently with the policy training, based on the RL agent’s experienced trajectories. Intuitively, we aim to extract information from the RL agent’s past experiences to shape the reward function. This procedure acknowledges the fact that a reward needs to be perceived by an agent from the current state, and this perception needs to be learned. While our previous application of APF concerned low-dimensional discrete state-space problems in the Q-learning paradigm, we intend to generalize its use to high-dimensional problems such as benchmark suits of video games. Ultimately, the APF method can be combined with any RL algorithms in various applications. However, due to the limitation of computing and data storage, storing many trajectories of high-dimensional states for the RL agent to learn from is not feasible. Therefore, this paper proposes the format of APF pluses an encoder, namely APF+, where the encoder projects high-dimensional frames to low-dimensional embeddings as the APF’s input. To effectively encode high-dimensional states, we propose a new state representation method, the W-shaped Network (W-Net), which extracts both static and dynamic features of states. By incorporating W-Net into the APF component to augment the training of a downstream Dueling Deep Q-Network (DDQN) [ 13 ] , we obtain the APF-WNet-DDQN method. We then empirically study APF-WNet-DDQN in 20 Atari game-playing tasks in the OpenAI Gym environment [ 14 ] . Fig 1 shows the screenshots of five Atari games as examples. To evaluate W-Net, we choose two other state representation encoders as our baselines: (1) an unsupervised representation method called Spatiotemporal Deep Infomax (ST-DIM) [ 2 ] and (2) the Atari Annotated RAM Interface (ARI) [ 2 ] which directly utilizes the ground truth RAM information, and obtain two baseline methods, APF-STDIM-DDQN and APF-ARI-DDQN, respectively. Additionally, we also compare the APF+ augmented DDQN algorithms with the bare (non-APF) baseline, the DDQN. The experimental results show that APF-WNet-DDQN agent outperforms the DDQN (14/20 games) and the APF-STDIM-DDQN (13/20 games) agents significantly. Furthermore, the APF-WNet-DDQN agent achieves indifferent performances with the APF-ARI-DDQN agent, which generates embeddings by directly consulting the ground-truth Atari game RAM information. Note that in this paper, the goal is not to beat the charts as regards the RL performance on Atari gameplay. We use the Atari benchmark to test whether the APF, a specialized transform to improve value estimation in RL, also works in these high-dimensional, pixel-image-based games. We use the Atari game set as an evaluation method to compare the viability of game-state representation learning for optimized exploitation of the APF transform in RL. Fig 1 : Screenshots of five Atari games. From left to right: Ms Pacman, Boxing, Breakout, Freeway, and Seaquest. Paper outline. Section Related Work reviews related research works on PBRS and state representation methods. Section Background introduces preliminaries of RL, PBRS, and the baseline RL algorithm. Section Methodology revisits the APF method and describes our proposed W-Net and the APF-WNet-DDQN algorithm in detail. The experimental parameters and results are presented and discussed in Section Experiments . Finally, we present a summary in Section Conclusion . Related Work In RL, reward shaping is an efficient way of incorporating external advice into a learning agent. Informative shaping rewards can accelerate the learning process of the RL agent. On the contrary, naive rewards like ad hoc rewards would lead to unwanted or unsafe control behaviors. Instead of handcrafted reward shaping, relying on a mechanism where the optimal computation of reward is learned, concurrent to learning the main task is advantageous. Therefore, the PBRS framework, which guarantees policy invariance by using the difference in subsequent values of a potential function [ 12 ] , has shown to be effective in authonomous RL reward shaping. Many RL reward-shaping research works are based on the PBRS framework. [ 15 ] first demonstrated how to apply human feedback in reward shaping effectively based on PBRS. In [ 16 ] , the authors reinforced the reward signal under the PBRS framework by tracking the maximum and minimum episodic rewards. [ 17 ] utilized PBRS for online partially observable Markov decision processes to improve approximate planning. Moreover,