On-Policy Optimization of ANFIS Policies Using Proximal Policy Optimization

Kaaustaaub Shankar, Wilhelm Louw, Kelly Cohen · 2025

Read On-Policy Optimization of ANFIS Policies Using Proximal Policy Optimization by Kaaustaaub Shankar, Wilhelm Louw, Kelly Cohen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-01039.txt.

Introduction

Kaaustaaub Shankar, Wilhelm Louw, and Kelly Cohen train fuzzy controllers with a stable policy method drawn from reinforcement learning. Fuzzy controllers are prized because their rules can be read by people. Past work paired them with a value based method that can wobble.

Their new pairing uses an actor and critic setup that learns on policy. On a classic pole balance task across many runs, the fuzzy agents hit the top score each time with no spread. They beat the older pairing on steadiness and speed. The result favors this route for clear, readable control agents.

Catalog record

CREATORKaaustaaub Shankar, Wilhelm Louw, Kelly Cohen
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE8 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-01039

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

On-Policy Optimization of ANFIS Policies Using Proximal Policy Optimization
Kaaustaaub Shankar, Wilhelm Louw, Kelly Cohen

Abstract
We present a reinforcement learning method for training neuro-fuzzy controllers using Proximal Policy Optimization (PPO). Unlike prior approaches that used Deep Q-Networks (DQN) with Adaptive Neuro-Fuzzy Inference Systems (ANFIS), our PPO-based framework leverages a stable on-policy actor-critic setup. Evaluated on the CartPole-v1 environment across multiple seeds, PPO-trained fuzzy agents consistently achieved the maximum return of 500 500 with zero variance after 20,000 20{,}000 updates, outperforming ANFIS-DQN baselines in both stability and convergence speed. This highlights PPO’s potential for training explainable neuro-fuzzy agents in reinforcement learning tasks.



1
Introduction

Deep reinforcement learning (RL) has shown the potential to display super-human skill in complex domains. An example of this is AlphaGo’s defeat of a Go world champion
[
1
]
. However, the policies learned by deep neural networks (DNNs) remain largely opaque, limiting trust in safety-critical settings such as autonomous driving and healthcare.
In contrast, fuzzy inference systems offer transparency while providing a robust solution. These systems fall into two families: Mamdani and Takagi-Sugeno-Kang (TSK).
Mamdani systems rely on linguistic IF–THEN rules with fuzzy outputs and subsequent defuzzification, making them highly interpretable but less amenable to gradient-based tuning
[
2
]
.
TSK models instead express rule consequents as linear functions of the inputs, producing smoother numeric outputs and enabling more robust numerical optimization
[
3
]
.
Yet both architectures still lack systematic training pipelines. Designing membership functions, rule bases, and consequents typically depends on expert heuristics or search methods such as genetic algorithms, which hampers scalability to high-dimensional or dynamic tasks
[
4
]
.

Neuro-fuzzy methods like ANFIS address this by using a neural network to transform the inputs into intermediate features, which feed into Gaussian membership functions that activate first-order TSK rules; their weighted outputs are aggregated to produce the final action logits. All trainable parameters like network weights, membership centres and sigmas, and rule consequents are updated through gradient descent
[
5
]
. Furthermore, deep applications like ANFIS-DQN hybrids have shown promise
[
6
]
but inherit the instability of off-policy Q-learning.

Proximal Policy Optimization (PPO) combats these issues with a clipped, on-policy surrogate objective that yields stable learning and strong sample efficiency
[
7
]
. We therefore integrate an ANFIS-style fuzzy module into PPO, forming a
PPO-Fuzzy
agent. Using the well-studied
CartPole-v1
benchmark, we evaluate whether the approach achieves the transparency of fuzzy rules without sacrificing the performance of modern policy-gradient RL.

2
Related Work

Recently, there has been growing interest in scaling up fuzzy RL and integrating it with deep learning. Zander
et al.
trained Takagi–Sugeno–Kang (TSK) fuzzy systems with Deep Q-Learning (DQN), reporting
CartPole-v1
performance on par with, or better than, ordinary DQNs yet exhibiting the training instabilities typical of off-policy methods
[
6
]
. This motivates exploring
on-policy
optimization such as PPO.

3
Methodologies

To isolate the effect of the optimization algorithm, we replicate the experimental setup of Zander
et al.
, replacing their DQN learner with a PPO-based actor–critic loop. Four agents, each initialized with a distinct random seed, are trained on the
CartPole-v1
environment.

As in the original work, the raw state vector is passed through a neural network comprising an input layer with 4 units, a hidden layer of 128 neurons with
ReLU
activation, and a second hidden layer of 127
ReLU
-activated neurons. These 127 intermediate features are used to activate 16 Gaussian membership functions, whose firing strengths are used in a weight sum of the first-order TSK rule consequents to produce the final action.

Figure 1
:
Architecture of the controller

The value function, used to estimate state values for PPO’s critic, is modeled by a separate neural network. This network consists of two fully connected hidden layers with 64 and 32 units respectively, each followed by a
Tanh
activation.

Figure 2
:
Architecture of the value function

The actor and critic networks are optimized jointly using the PPO objective, which balances clipped policy updates, value regression, and entropy regularization to ensure stable learning. The full training loop for the PPO-ANFIS agent is summarized in Algorithm
1
.

Algorithm 1

PPO-ANFIS training loop

1:

Initialise ANFIS policy
\pi_{\theta}
and value network
V_{\phi}

2:

for
each iteration
i=1,\dots,N_{\text{updates}}

do

3:

Collect
T
timesteps of on-policy data
\mathcal{D}_{i}
using
\pi_{\theta}

4:

Compute returns
R_{t}
and advantages
\hat{A}_{t}
for all
(s_{t},a_{t})\in\mathcal{D}_{i}

5:

for
epoch
k=1

to

K

do

6:

Sample minibatch
\mathcal{B}\subset\mathcal{D}_{i}

7:

\mathcal{L}_{\text{clip}}=\mathbb{E}_{\mathcal{B}}\!\Big[\min\!\big(r_{t}(\theta)\hat{A}_{t},\;\operatorname{clip}(r_{t}(\theta),1-\epsilon,1+\epsilon)\hat{A}_{t}\big)\Big]

8:

\mathcal{L}_{\text{VF}}=\frac{1}{2}(R_{t}-V_{\phi}(s_{t}))^{2}
(Value loss)

9:

\mathcal{L}=-\mathcal{L}_{\text{clip}}+c_{v}\mathcal{L}_{\text{VF}}-c_{e}\,\mathbb{E}_{\mathcal{B}}[H[\pi_{\theta}(\cdot|s_{t})]]

10:

Update
(\theta,\phi)
via Adam; clip gradient-norm to
10

Notation.

r_{t}(\theta)=\pi_{\theta}(a_{t}|s_{t})/\pi_{\theta_{\text{old}}}(a_{t}|s_{t})
;
c_{v}
and
c_{e}
are value and entropy weights;
\epsilon=0.2
is the PPO clip parameter.

All experiments use the
CartPole-v1
environment with
the episode length capped at 500 steps. We use four runs that employ the following seeds
\{9,\allowbreak 42,\allowbreak 109,\allowbreak 131\}
; each run trains for
1{\times}10^{5}
mini-batch updates. The hyperparameters are included in the Appendix.

4
Results

(a)

PPO-trained ANFIS agents on
CartPole-v1
.
Mean return over 10 deterministic episodes, averaged across seeds.

(b)

Same experiment with gradient-norm clipping at 0.5.

Figure 3
:
Evaluation curves for PPO-trained ANFIS controllers.

Figure 4
:
ANFIS-DQN and vanilla DQN agents on
CartPole-v1

(Referenced from
[
6
]
).

5
Discussion

Figure
3(a)
shows the training performance of PPO-trained ANFIS agents in the
CartPole-v1
environment across four random seeds, while Figure
4
presents the results for ANFIS-DQN and standard DQN. All PPO-based agents converge rapidly, reaching the maximum return of 500 within 20,000–40,000 mini-batch iterations.

Some fluctuations remain (seeds 9 and 131 around 50,000 iterations), partly due to hyperparameter choices. With more optimal settings such as gradient-norm clipping at 0.5, training stability improves as shown in Figure
3(b)
. By 100,000 iterations, all PPO agents consistently achieve the maximum return, demonstrating robustness to initialization and effective handling of the fuzzy policy structure.

In contrast, prior DQN-based ANFIS training
[
6
]
showed persistent instability. These results highlight PPO’s suitability for training fuzzy controllers and its potential for scalable, stable learning in higher-dimensional tasks.

6
Future Work

In future work, we aim to expand and test this framework to more complicated environments like
LunarLander-v3-Continuous
and
Hopper-v1
. We also plan to explore integration of interpretability tools like SHAP or LIME to attribute actions to specific fuzzy rules, guiding rule pruning and the discovery of the optimal rule count.

Disclosure of Interests.

The authors have no competing interests to disclose.

7
Appendix:

Parameter

Value

Discount factor
\gamma

0.99

Learning rate

1\times 10^{-5}

PPO clip
\epsilon

0.2

Entropy coefficient
c_{e}

0.02

Mini-batch size

64

Roll-out horizon
T

2048 steps

Gradient-norm clip

10

Membership centres
c_{i}

\mathcal{N}(0,0.1^{2})

Membership widths
\sigma_{i}

0.25+0.5\,\mathcal{U}(0,1)

Rule consequents
w_{i}

Read the complete text