F-TIS: Harnessing Diverse Models in Collaborative GRPO

Nikolay Blagoev, Oğuzhan Ersoy, Wendelin Boehmer, Lydia Yiyu Chen · 2026

Read F-TIS: Harnessing Diverse Models in Collaborative GRPO by Nikolay Blagoev, Oğuzhan Ersoy, Wendelin Boehmer, Lydia Yiyu Chen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-22537.txt.

Introduction

Blagoev and team propose Filtered Truncated Importance Sampling (F-TIS), a GRPO-style paradigm for heterogeneous models. In GRPO, completions are rewarded, and policy updates toward better ones. Generation is slow, and prior work spreads inference, assuming homogeneous models. Decentralized systems need heterogeneous models, but off-policy samples hurt convergence. F-TIS uses off-policy samples to improve local learning and cross-model collaboration. F-TIS matches on-sample final convergence, and sometimes beats on-policy training out-of-distribution by up to 12 percent.

Catalog record

CREATORNikolay Blagoev, Oğuzhan Ersoy, Wendelin Boehmer, Lydia Yiyu Chen
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE21942
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.22537

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

F-TIS: Harnessing Diverse Models in Collaborative GRPO Nikolay Blagoev, Oğuzhan Ersoy, Wendelin Boehmer, Lydia Yiyu Chen Abstract Reinforcement learning methods such as GRPO have seen great popularity in LLM post-training. In GRPO, models produce completions to a set of prompts, which are rewarded, and the policy is updated towards the relatively high reward completions. Due to the auto-regressive nature of models, the generation phase of such style of training can be extremely time consuming. As a solution, prior work has sought to distribute the inference step across many nodes, working parallel. These works assume primarily homogeneous models in the training in order to keep samples as close to on-policy as possible. This assumption may be impractical in decentralized systems, where parties with various computes and preferences may wish to collaborate on the same task. Thus, decentralized training requires an approach that can handle heterogeneous models - different models collaborating on the same tasks. However, this leads to highly off-policy samples presented during training, which prior work has identified that off-policy samples can hurt GRPO convergence. To enable heterogeneity, we propose Filtered Truncated Importance Sampling (F-TIS) - a GRPO-style training paradigm that can use off-policy samples to improve local model’s learning. Our framework allows various models to collaborate in the same RL training run while being communication efficient. We extensively evaluate F-TIS in various heterogeneous setups and we show that it exhibits identical final model convergence to purely on-sample training. Furthermore, we observe in some setups better generalization on out-of-distribution tasks than on-policy training, increasing model’s performance by up to 12%. keywords LLM, Collaborative RL, GRPO 1 Introduction Reinforcement Learning (RL) has seen a great adoption for the post-training of Large Language Models (LLMs) 14 ; 12 ; 18 . Algorithms such as Proximal Policy Optimization (PPO) allow LLMs to learn user-preferred behaviour 13 , such as adhering to some ethical code during conversations. More recently, RL has been utilized to a great degree for the purposes of improving the reasoning of LLMs. This is in part due to the highly influential work of 14 , which proposed Group Relative Policy Optimization (GRPO). GRPO removes the need for a value-model in PPO, thus greatly reducing the computational and memory requirements, by instead relying on the group relative advantage. In GRPO, for each prompt, multiple completions are generated and each completion’s advantage is computed relative to the other completions of the same prompt. Several works have demonstrated the success of GRPO-based algorithms in improving an LLM’s reasoning for various tasks 18 ; 8 ; 4 . Despite introducing lower memory footprint relative to PPO, GRPO comes with a high computational cost of generating multiple completions for each group (often 8 or more) 17 . This results in a bottleneck during the generation step, as LLMs generate in an auto-regressive manner, i.e., one token at a time 16 . A natural step is to distribute the generation step across many workers thus speeding up the generation 17 ; 15 . Such systems, however, assume relatively homogeneous models and resources, making them impractical for decentralized training. Furthermore, even if participants start from the same models, if they solely communicate completions, their models will drift apart, due to floating-point non-associativity. Such drifts introduce off-policyness between the generator and the trainer, harming the final model’s performance 9 ; 20 . Beyond these emergent drifts, we identify three types of model heterogeneity in decentralized RL post-training: • Model Size : Models of various sizes (e.g. of 1 billion parameters and 3 billion parameters) can be trained together in the same loop. More constrained devices train the smaller models, while larger devices or clusters can accommodate larger models. Alternatively, one can alleviate the costs of sampling from a large model (as the auto-regressive inference of LLMs is the biggest bottleneck in GRPO training 16 ). Thus they can utilize some number of small models to populate the batch size. • Model Expertise : Models of similar size, but different parameters and expertise can be trained together, as different users might have a personalized model they want to collaboratively improve on some task. • Trainable Parameters : Some nodes may train a different subset of the same model’s parameters (e.g. with parameter efficient fine-tuning). Such cases can arise due to resource constraints on some devices, needing a smaller set of trainable parameter, or to avoid destructive interference with other tasks 10 . We empirically show that GRPO underperforms in such relative to training purely on-policy. To address this challenge, we propose F-TIS - a single unified framework that can make use of off-policy samples in decentralized heterogeneous Reinforcement Learning. This framework requires communication volume of only 8\times|p| bytes, i.e., 8 bytes per each token, as it only communicates the log-probabilities and tokens of completions between nodes. Despite this minimal overhead, F-TIS can achieve performance identical to homogeneous RL and can even improve model’s reasoning capabilities on out-of-distribution tasks (by up to 12%). Beyond its benefits in decentralized training, F-TIS provides benefits even for a single cluster-training, where multiple models can be post-trained in one training run, rather than several sequential ones. 2 Related Work Reinforcement Learning has been commonly used for LLM post-training. Proximal Policy Optimization allowed models to learn user preference, which were implicitly hidden in data 13 . More recently, GRPO and its derivatives have been utilized to boost model’s reasoning and instruction-following abilities. When a model \theta is trained with GRPO, it generates a number G of completions (responses) a_{i} per prompt p ( p\circ a_{i}\;\forall i\in G where \circ is a concatenation operation), which is termed a "group". Each of the completions is rewarded via some reward model, to get a single scalar value r_{i} . Commonly used reward mechanisms in GRPO are rule-based rewards ( 8 ) , which check if the formatting and final answer is correct for a math task or all tests pass for a coding tasks. To replace the need for a value model, GRPO uses the group advantage \hat{A}_{i} , \hat{A}_{i}:=\frac{r_{i}-\mu_{r}}{\sigma_{r}} , where \mu and \sigma are the mean and standard deviation of the rewards for the completions belonging to the same prompt. The advantage is then used to compute the loss: \displaystyle\mathcal{L}_{GRPO}=\frac{1}{G}\sum_{i=1}^{G}\frac{1}{|a_{i}|}\sum_{t=1}^{|a_{i}|}min[\frac{\pi_{\theta}(a_{i,t}|p\circ a_{i,<t})}{\pi_{\theta_{gen}}(a_{i,t}|p\circ a_{i,<t})}\hat{A}_{i}, \displaystyle clip(\frac{\pi_{\theta}(a_{i,t}|p\circ a_{i,<t})}{\pi_{\theta_{gen}}(a_{i,t}|p\circ a_{i,<t})}\hat{A}_{i},1-\epsilon,1+\epsilon)] where \pi_{\theta_{gen}} is the policy that generated the completion. Training with GRPO can be divided into two phases - completion generation (gathering as many completions for various prompts) and training (computing the gradient based on the loss for all completions and for all prompts) ( 17 ) . Prior work has identified that the KL term ( \mathcal{D}_{KL}(\pi_{\theta}\parallel\pi_{\theta_{ref}}) , does not benefit the training and increases memory and computational demands 8 . In line with these works, we omit the KL-term in our experiments. Distributed RL has gained popularity as a means of speeding up the generation phase of GRPO-style training 17 . Deployed systems have even successfully trained models in a decentralized manner, however still assuming homogeneous models in order to prevent off-policy issues 1 ; 15 . Such approaches have been classified into two categories: vertical and horizontal 2 . In vertical RL, each device/node generates the entire group for one prompt. In horizontal RL, each device generates

Read the complete text