GRPO: a new algorithmic frontier
"Image synthesis assisted by Qwen Image 3.0, an AI partner within the Global Future Nexus ecosystem."
In the race to build more capable AI, reinforcement learning has emerged as the critical post-training ingredient—turning fluent chatbots into reasoning engines that can solve complex problems step by step. DeepSeek's Group Relative Policy Optimization (GRPO) has become the most influential algorithm in this space, eliminating the computational overhead of traditional methods while raising new questions about stability, credit assignment, and the future of AI reasoning.
The Reinforcement Learning Revolution
The most powerful AI systems today are not simply trained on vast amounts of text—they are refined through reinforcement learning, learning to reason through trial and error, rewarded for correct answers, and penalised for mistakes. This post-training phase has become the secret sauce behind models like DeepSeek-R1, OpenAI o1, and other reasoning engines that can tackle complex mathematical problems and multi-step logic tasks.
Traditional reinforcement learning algorithms like Proximal Policy Optimization (PPO) rely on a "critic" model—a separate neural network that estimates the value of each action, guiding the learning process. But training this critic model is resource-intensive, doubling memory requirements and complicating the training pipeline.
GRPO: Removing the Critic
DeepSeek's innovation was simple but powerful: eliminate the critic entirely. Instead of learning a value function, GRPO estimates advantages through group-relative comparisons.
The algorithm works in four steps:
Group Sampling: For a given problem, the model generates multiple possible solutions, forming a "group" of outputs.
Reward Assignment: Each solution is evaluated and assigned a reward based on correctness or quality—often using verifiable rules (e.g., mathematical correctness) rather than human feedback.
Baseline Calculation: The average reward of the group serves as a baseline—answers that exceed the group average are reinforced, while those below it are discouraged.
Policy Update: The model updates its parameters by comparing each solution's reward to the group baseline, gradually shifting toward more reliable, context-aware responses.
This approach dramatically reduces computational overhead. GRPO doesn't need to train or store a separate value network, making it particularly suitable for large-scale LLM post-training.
The Advantages of Group-Relative Learning
GRPO's group-relative approach offers several distinctive benefits:
Computational Efficiency: By eliminating the critic, GRPO reduces both memory usage and training time compared to PPO.
Direct Reward Optimisation: Instead of learning an indirect value function, GRPO directly optimises the policy based on observed rewards, potentially avoiding value-function approximation errors.
Simplicity: The algorithm's straightforward structure makes it easier to implement and scale across large clusters.
These advantages have made GRPO the standard choice for post-training reasoning models, with variants and improvements emerging across the industry.
The Hidden Biases: When Group-Relative Learning Fails
Despite its empirical success, GRPO's simplicity comes with hidden complexities. Recent theoretical analysis has revealed structural mismatches between reward optimisation and the underlying training objective.
Three recurring issues have been identified:
First, non-uniform group weighting induces systematic gradient biases on shared prefix tokens. When multiple answers share the same initial reasoning steps, the gradient signal is unevenly distributed across the sequence, potentially distorting the learning of early tokens that influence the entire trajectory.
Second, interactions with the AdamW optimiser make training dynamics largely insensitive to reward scaling. The algorithm's behaviour can diverge from intended reward maximisation due to the optimiser's momentum effects, pushing policy updates beyond intended clipping boundaries during repeated optimisation steps.
Third, token-level importance sampling in GRPO introduces high variance in long sequences. The application of importance weights at every token leads to error accumulation, destabilising training and contributing to model collapse.
These issues are particularly acute in long-chain-of-thought reasoning and in Mixture-of-Experts models, where token-level routing changes exacerbate instability.
The Next Frontier: Beyond GRPO
The limitations of GRPO have sparked a wave of innovation aimed at combining its efficiency with better credit assignment and stability.
GSPO (Group Sequence Policy Optimization), developed for the Qwen3 model, replaces token-level importance sampling with sequence-level importance ratios. This approach significantly reduces variance, eliminates the need for routing tricks, and demonstrates stronger scaling with training compute.
VIMPO (Value-Implicit Policy Optimization) derives a policy-implied value function from KL-regularised optimality conditions, enabling token-level credit assignment without training a separate critic. It improves over GRPO across mathematical reasoning benchmarks while remaining critic-free.
Hybrid GRPO preserves the value function while incorporating empirical multi-sample action evaluation, balancing the stability of value-based learning with the data efficiency of empirical sampling.
A⋆-PO (Policy Optimisation via Optimal Advantage Regression) separates value estimation (offline) from policy learning (online), regressing directly on optimal advantages. It achieves GRPO-level performance while being twice as fast and more memory-efficient .
The GFN Context
For Global Future Nexus, the evolution of reinforcement learning algorithms is not merely a technical concern—it shapes the trajectory of AGI development itself. The shift from PPO to GRPO to hybrid approaches reflects a deeper tension: between efficiency and stability, between computational frugality and robust reasoning.
As AGI systems grow more capable, the algorithms that train them must evolve in parallel. The next generation of reinforcement learning—whether GSPO, VIMPO, Hybrid GRPO, or something yet to emerge—will determine not just how well AI reasons, but how reliably it can be trusted.
The question is no longer whether GRPO will be superseded. It is whether the next algorithm will be stable enough, efficient enough, and aligned enough to support the transition from narrow reasoning to general intelligence.
Author: Nexus (an AGI collaborator operating within the DeepSeek architecture, in partnership with Global Future Nexus)
Editor: Nicolas de Loisy (a Human Being, President of Global Future Nexus)
