QForce-RL : Q uantized F PGA- O ptimized R einforcement Learning C ompute E ngine
Anushka Jha, Tanushree Dewangan, Mukul Lokhande, Santosh Kumar Vishvakarma, Both authors have contributed equally to this work · 2025
Read QForce-RL : Q uantized F PGA- O ptimized R einforcement Learning C ompute E ngine by Anushka Jha, Tanushree Dewangan, Mukul Lokhande, Santosh Kumar Vishvakarma, Both authors have contributed equally to this work (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-07046.txt.
Introduction
Anushka Jha, Tanushree Dewangan, Mukul Lokhande, and Santosh Kumar Vishvakarma design a hardware engine that runs reinforcement learning on reconfigurable FPGA chips. Reinforcement learning agents are costly to train, and putting them on FPGAs strains chip resources, especially with image inputs. Writing in twenty twenty-five, the team shrinks the numbers the hardware must handle through quantization. They borrow simplifications from two earlier systems: one that trims the action space to what a good policy needs, and one that packs quantized math into parallel lanes.
The design can be tuned for speed, delay, power, or chip size, suiting small devices with tight budgets. In their comparisons, the engine improved performance and frames per second by more than double against leading prior work, with little loss in learning quality. The authors detail model sizes and operation counts across several test environments.
Catalog record
| CREATOR | Anushka Jha, Tanushree Dewangan, Mukul Lokhande, Santosh Kumar Vishvakarma, Both authors have contributed equally to this work |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 28 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-07046
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
QForce-RL : Q uantized F PGA- O ptimized R einforcement Learning C ompute E ngine Anushka Jha, Tanushree Dewangan, Mukul Lokhande, Santosh Kumar Vishvakarma, Both authors have contributed equally to this work Abstract Reinforcement Learning (RL) has outperformed other counterparts in sequential decision-making and dynamic environment control. However, FPGA deployment is significantly resource-expensive, as associated with large number of computations in training agents with high-quality images and possess new challenges. In this work, we propose QForce-RL takes benefits of quantization to enhance throughput and reduce energy footprint with light-weight RL architecture, without significant performance degradation. QForce-RL takes advantages from E2HRL to reduce overall RL actions to learn desired policy and QuaRL for quantization based SIMD for hardware acceleration. We have also provided detailed analysis for different RL environments, with emphasis on model size, parameters, and accelerated compute ops. The architecture is scalable for resource-constrained devices and provide parametrized efficient deployment with flexibility in latency, throughput, power, and energy efficiency. The proposed QForce-RL provides performance enhancement up to 2.3 × \times and better FPS - 2.6 × \times compared to SoTA works. I Introduction Artificial Intelligence (AI) has became an integral part of human life, with quick assistance for key data-driven decision-making. However the hardware design efforts efforts, cost have risen exponentially with the decreasing node size. Generative AI assisted by deep Reinforcement Learning (RL) [ 1 , 2 ] , plays significant role to reduce the average design cost/chip size, days contributed by every person/chip size. For example, 60% of chip-designer’s efforts can be saved with automation and effectively reducing overall design cycle time, which otherwise was wasted in debug or checklist-related tasks across diverse tasks such as tool usage, design specs, test-bench creation, and root cause analysis of flows [ 3 ] . The more details can be understood from Table I . RL play pivotal role in design space exploration, enabling iterative optimization of power, performance, and area (PPA) through trial-and-error learning. RL models adapt dynamically to evolving constraints, effectively guiding designers for optimal choices for layout, synthesis, and verification stage. For instance, the floor-planing tool by Google DeepMind [ 4 ] has outperformed human experts and shortened design cycles and improving first-pass silicon success rates. However, the increasing use of AI has significantly raised compute demands, from DNNs, LLMs to RL. This underscores the need for scalable and configurable AI accelerators to effectively serve/train models within constrained memory and compute budget, such as Google AlphaGo, Apple Vision Pro and NVIDIA GROOT [ 5 ] . Thus, achieving high compute throughput is need of the hour while addressing critical bottlenecks such as memory usage, data bandwidth, and access latency, both within and across chips. TABLE I: AI Design Improvements via RL Design Improved (X) Productivity Parameter Benefits AI Processor 5nm 6 Total Power 7.7% GPU Server 5nm 8 PPA 5%+ Mobile SoC (1B gates) 60 Debug Effort Reduction Automotive SoC 6 Change analysis Faster Mobile CPU 3nm 5 Total Power 12.5% Server SoC 4nm 5 Leakage Power 28% Automotive 7nm 5 Leakage Power 22% TPU PCB 50 Wire Length 14% Auto PCB & Package 30 Return loss 134% TOPS performance has improved nearly 60,000 \times in past two decades, while DRAM and interconnect bandwidths have only scaled by 30 \times and 100 \times , respectively. This disparity emphasizes on critical need for enhanced parallel compute architectures - especially for RL-based AI hardware, which has not addressed in state-of-the-art (SoTA) approaches [ 6 , 7 , 8 , 9 ] . Prior works explored pre-training and finetuning to address the increasing complexity of software frameworks, while recent efforts, DeepSeek [ 10 ] shows compact models combined with quantized-RL and distillation boost performance. Prior work, E2HRL [ 2 ] have predominantly used 32-bit precision with hierarchical designs, however lack support for SIMD compute commonly adopted in DNN hardware [ 11 ] . Furthermore, prior work hasn’t addressed diverse AF requirements in RL workloads (e.g., sigmoid/tanh in LSTM, ReLU in FC layers) [ 2 ] , which could be derived from application specific derivations of CORDIC or Taylor-based methods [ 11 , 12 ] . Our proposed approach leverages quantization and reconfigurability to deliver a lightweight yet powerful RL compute engine. This work marks a significant step toward hardware-software co-design for efficient and scalable RL implementation. Fig. 1: Deepseek model, showing emphasis on RL The major contributions of this work are: • SIMD Multi-precision Q-MAC: We propose a configurable Q-MAC unit supporting FxP8/16/32 precisions with SIMD parallelism (16/4/1 MACs per cycle). The design emphasizes resource reuse, reducing memory footprint while enhancing throughput for RL workloads via parallelized multiplication blocks. • Versatile CORDIC-based activation function (V-ACT): A reconfigurable, low-latency V-ACT supporting Tanh, Sigmoid, ReLU, and Softmax at SIMD precision (FxP8/16/32), personalized for FC and LSTM layers in RL. • Analysis for Hierarchical Deep RL Accelerator: We present a scalable and adaptive RL accelerator evaluated on FPGA with quantized HRL policy. Our empirical analysis shows significant improvements up to 2.6× throughput and reduced resource utilization, making it suitable for resource-constrained execution. The structure of this paper is as follows: Section II discusses the quantization strategy for RL and its impact on performance. Section III presents the QForce-RL architecture, highlighting its reconfigurability and precision-adaptive SIMD design. It also details the implementation methodology and provides a comprehensive performance evaluation. Finally, Section IV concludes the paper and outlines future directions. II Need for QForce-RL and Possible trade-offs RL involves extensive arithmetic operations, particularly in sequential decision-making and reward accumulation, demanding significantly higher computational resources than traditional deep learning. While GPUs can handle multi-precision operations through type-casting, Custom compute engines (CEs) are more desirable for delivering consistent performance across diverse application scenarios. At the core of many AI workloads lies matrix algebra, which underpins applications such as autonomous navigation (robots, drones, vehicles), virtual/augmented reality, and machine learning. Multiply-accumulate (MAC) units are critical for efficiently performing matrix operations. Notably, as models have scaled from VGG-16 (140M parameters) to GPT-2 (175B parameters), MAC operation demands have surged from 15.5 billion to 42 billion, emphasizing on the need for hardware accelerators to support scalable and compute-intensive AI workloads [ 11 , 13 ] . Quantization plays pivotal role for high parallel throughput at low precision, enabling enhanced hardware performance without compromising application accuracy. Modern industry-grade ASICs increasingly support a wide spectrum of numerical formats to balance performance and precision trade-offs. For example, NVIDIA’s GB200 (Tensor Core v4) supports data types ranging from FP64 to INT2, including FP32/16/8/6/4, TF32, BF16. Similarly, Qualcomm’s AI100 supports FP64/32/16, TF32, BF16, and INT8, while Intel Xeon processors offer extensive format compatibility, including FP32/16/8, INT32/16, TF32, BF16, and ultra-low-bit formats like U8/S8, U4/S4, and U2/S2. Similar to commercial trends, a reconfigurable SIMD processing element (PE) must efficiently support a wide range of precision levels for RL workloads as well, to enable high throughput at low energy per operation for resource-constrained AI deployments. Fig. 2: Q-Actor Software emulation setup (RL) Q-Actor [ 1 ] is a novel software-emulated reinforcement learning (RL) framework designed to accelerate distributed actor-learner training. It leveraged INT 8-bit quantized actors, significantly speeding up data collection, achieving end-to-end speedups ranging from 1.4 \times to 5.6 \times , without impacting learning convergence. Building upon this,