DiffusionART: Efficient, Stable and Scalable Reinforcement of Diffusion Models
Online reinforcement learning (RL) fine-tunes diffusion models with black-box rewards, but reverse-process methods such as Flow-GRPO are limited in efficiency by poor credit assignment, while forward-process methods such as DiffusionNFT are efficient yet unstable without carefully tuned…
This paper introduces DiffusionART, a method for improving the efficiency and stability of online reinforcement learning (RL) in diffusion models by addressing limitations in existing approaches.
What happened
The article presents DiffusionART, a novel reinforcement learning framework for diffusion models. It addresses inefficiencies in reverse-process (Flow-GRPO) and forward-process (DiffusionNFT) methods by introducing reward aggregation, a trust-region-masked second update, and eliminating reliance on exponential-moving-average (EMA) tuning. The approach is validated through experiments on image and video generation tasks.
Why it matters
Efficient and stable training of diffusion models is critical for practical AI applications. DiffusionART reduces computational costs while maintaining performance, which benefits developers working with resource-constrained systems or large-scale generative models.
- DiffusionART balances efficiency and stability in RL for diffusion models
- Reward aggregation and trust-region mechanisms improve training reliability
- The approach reduces computational demands without sacrificing performance
- Supports flexible reward modeling without EMA tuning
- Demonstrated effectiveness across image and video generation tasks
The bigger picture
The method reduces GPU hours required for training by 11.72-12.02 times compared to existing approaches while achieving higher mean training rewards, suggesting potential for broader adoption in generative AI systems.
DiffusionART offers a scalable solution for reinforcement learning in diffusion models by addressing key limitations of existing methods, with measurable efficiency gains and stability improvements.
Technical details
- Reward aggregation for reverse-process states to remove consistency forces
- Trust-region-masked second update for drift-adaptive restoration
- Pairwise reward models (DiffusionART-Pairwise) without differentiable reward models
- Elimination of EMA tuning for stability
- Efficient training with fewer samples per prompt
FAQ
What problem does DiffusionART solve?
DiffusionART addresses inefficiencies and instability in online reinforcement learning for diffusion models, particularly in balancing computational cost and training stability.
How does DiffusionART differ from prior methods?
It uses reward aggregation to eliminate consistency forces, a trust-region-masked update for stability, and avoids EMA tuning, unlike Flow-GRPO and DiffusionNFT.
What are the key benefits of DiffusionART?
It reduces GPU hours by 11-12 times compared to existing methods, supports pairwise reward models, and maintains high training rewards with fewer samples.
This summary and analysis were generated with AI from the original article and may contain errors. Read the original for the full story.
Read the full story
Originally published by arXiv cs.LG on Oct 12, 2026.
Comments
Sign in to join the discussion.