Beyond ReLU: Chebyshev-DQN for Enhanced Deep Q-Networks
Saman Yazdannik, Morteza Tayefi, Shamim Sanisales · 2025
Read Beyond ReLU: Chebyshev-DQN for Enhanced Deep Q-Networks by Saman Yazdannik, Morteza Tayefi, Shamim Sanisales (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-14536.txt.
Introduction
Saman Yazdannik, Morteza Tayefi, and Shamim Sanisales present a new design for deep Q-networks in this 2025 paper. These networks learn by estimating the value of each action, and the usual building blocks can struggle with complex value patterns. The authors swap in Chebyshev polynomials, a classic set of curves, as the base for the value estimates.
On the CartPole balance task, their design beat a standard network of similar size by a wide margin at a moderate curve count. A much higher count hurt learning, though. The work shows that the choice of base curves matters, and that more complex is not always better.
Catalog record
| CREATOR | Saman Yazdannik, Morteza Tayefi, Shamim Sanisales |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 29 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-14536
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Beyond ReLU: Chebyshev-DQN for Enhanced Deep Q-Networks Saman Yazdannik, Morteza Tayefi, Shamim Sanisales Abstract The performance of Deep Q-Networks (DQN) is critically dependent on the ability of its underlying neural network to accurately approximate the action-value function. Standard function approximators, such as multi-layer perceptrons, may struggle to efficiently represent the complex value landscapes inherent in many reinforcement learning problems. This paper introduces a novel architecture, the Chebyshev-DQN (Ch-DQN), which integrates a Chebyshev polynomial basis into the DQN framework to create a more effective feature representation. By leveraging the powerful function approximation properties of Chebyshev polynomials, we hypothesize that the Ch-DQN can learn more efficiently and achieve higher performance. We evaluate our proposed model on the CartPole-v1 benchmark and compare it against a standard DQN with a comparable number of parameters. Our results demonstrate that the Ch-DQN with a moderate polynomial degree (N=4) achieves significantly better asymptotic performance, outperforming the baseline by approximately 39%. However, we also find that the choice of polynomial degree is a critical hyperparameter, as a high degree (N=8) can be detrimental to learning. This work validates the potential of using orthogonal polynomial bases in deep reinforcement learning while also highlighting the trade-offs involved in model complexity. 1 Introduction Deep Reinforcement Learning (DRL) has marked a significant breakthrough in artificial intelligence, enabling agents to achieve superhuman performance in a wide array of complex sequential decision-making tasks. Landmark successes range from mastering Atari games from raw pixel data [ 1 , 5 ] and defeating world champions in the game of Go [ 2 ] , to making significant strides in robotics, autonomous driving, and resource management [ 6 ] . A cornerstone of this success is the Deep Q-Network (DQN) algorithm, which demonstrated for the first time that a deep neural network could be trained to solve a diverse set of challenging control problems using a unified architecture [ 1 ] . The central mechanism of DQN involves using a deep neural network as a powerful function approximator to estimate the action-value (Q-value) function. The fidelity of this approximation is paramount, as it directly governs the quality of the agent’s learned policy. In the standard DQN framework, the function approximator is typically a multi-layer perceptron (MLP) or a convolutional neural network (CNN). While these architectures have proven immensely capable, their use is not without challenges. It is well-documented that the combination of off-policy learning, bootstrapping, and the expressive power of non-linear function approximators, often termed the “deadly triad” can lead to significant training instability and divergence [ 3 , 7 ] . Consequently, DQN requires specialized techniques such as experience replay and fixed target networks to mitigate these issues [ 1 ] . Furthermore, modern DRL algorithms are often criticized for their poor sample efficiency, frequently requiring millions of environment interactions to learn effective policies, a significant drawback compared to biological learning [ 8 ] . This raises a critical question: are conventional neural network architectures the most effective tool for representing the value functions inherent in reinforcement learning tasks? This paper posits that we can design more effective DRL agents by drawing upon foundational principles from classical approximation theory. We are not the first to consider alternative function bases; prior work has explored the use of linear models with Fourier bases and radial basis functions [ 9 ] . However, we turn our attention to Chebyshev polynomials, a family of orthogonal polynomials renowned for their powerful and theoretically grounded properties. A key attribute of Chebyshev polynomials is their “minimax” property, which states that they provide the best polynomial approximation to a continuous function under the L-infinity norm, minimizing the maximum possible error [ 4 , 10 ] . This suggests that a function basis built from Chebyshev polynomials could offer a more efficient and accurate representation of complex value functions. Building on this insight, we introduce the Chebyshev-DQN (Ch-DQN), a novel architecture that integrates a Chebyshev Neural Network as the primary function approximator within the DQN framework. Instead of relying on standard activation functions like ReLU in its initial layers, the Ch-DQN transforms the input state into a feature representation using a basis of Chebyshev polynomials. To our knowledge, this is the first work to explore such an integration for deep reinforcement learning. Our central hypothesis is that the superior approximation properties of the Chebyshev basis will enable the Ch-DQN to learn a more precise and stable representation of the Q-function. We anticipate this will yield significant performance benefits, particularly in the realm of sample efficiency. To validate this hypothesis, we conduct a series of experiments on classic continuous control benchmarks. This work aims to bridge the gap between approximation theory and modern deep reinforcement learning, presenting a new path toward developing more robust, efficient, and performant DRL agents. 2 Background and Related Work 2.1 Deep Q-Networks (DQN) Q-learning is a model-free reinforcement learning algorithm that aims to learn the optimal action-value function, Q^{*}(s,a) [ 12 ] . This function is defined as the maximum expected cumulative reward achievable from a given state-action pair (s,a) . The optimal Q-function follows the Bellman optimality equation [ 13 ] : Q^{*}(s,a)=\mathbb{E}\left[r+\gamma\max_{a^{\prime}}Q^{*}(s^{\prime},a^{\prime})\mid s,a\right] (1) where r is the immediate reward, \gamma is a discount factor, and s^{\prime} is the next state. For problems with small, discrete state spaces, Q-values can be stored in a lookup table (a Q-table). However, this approach becomes intractable in problems with large or continuous state spaces. The DQN algorithm, introduced by Mnih et al. [ 5 , 1 ] , solved this scalability problem by replacing the Q-table with a deep neural network, Q(s,a;\theta) , parameterized by weights \theta . To ensure stable training, DQN introduced two key innovations: • Experience Replay: The agent stores its experiences (s,a,r,s^{\prime}) in a replay buffer. During training, mini-batches of experiences are randomly sampled from this buffer to update the network weights. This breaks the temporal correlations between consecutive samples and smooths the training distribution. • Target Network: DQN uses a second, separate "target network," Q(s,a;\theta^{-}) , to generate the target values for the Bellman update. The weights of this target network are held fixed for a number of steps and are only periodically updated with the weights from the main "policy network." This prevents the instability that arises from using a rapidly changing network to estimate both the current Q-value and the target value. The network is trained by minimizing the Mean Squared Error (MSE) loss between the target Q-value and the value predicted by the policy network. 2.2 Function Approximation in Reinforcement Learning The use of function approximators is essential for applying reinforcement learning to real-world problems. The goal is to find a parameterized function that can generalize from a limited set of training samples to the entire state-action space [ 3 ] . This area of research predates deep learning, with early work focusing on linear function approximators, where the Q-function is represented as a linear combination of a set of basis functions or features [ 9 ] . While computationally efficient, linear methods may lack the expressive power to represent complex, non-linear value functions. The advent of deep learning provided a powerful toolkit for non-linear function approximation, with neural networks becoming a common choice.