2026-01-07 · 2 citations · club pick

Diffusion-DRF: Free, Rich, and Differentiable Reward for Video Diffusion Fine-Tuning

Yifan Wang, Yanyu Li, Gordon Guocheng Qian, Sergey Tulyakov, Yun Fu, Anil Kag

Reinforcement learning & control Video & generative modeling

No peer-reviewed venue on record yet. 2 citations, as of the last refresh.

Abstract

Video diffusion alignment has been heavily relied on scalar rewards. These rewards are typically derived from learned reward models in human preference datasets, requiring additional training and extensive collection. Moreover, scalar rewards provide coarse, global supervision, offering limited prompt-generation mismatch credit assignment and making models prone to reward exploitation and unstable optimization. We propose Diffusion-DRF, a free, rich, and differentiable reward framework for video diffusion fine-tuning. Diffusion-DRF employs a frozen, off-the-shelf Vision-Language Model (VLM) as the critic, eliminating the need for reward model training. Instead of relying on a single scalar reward, it decomposes each user prompt into multi-dimensional questions with freeform dense VQA explanation queries, yielding information-rich feedback. By direct differentiable optimization over this rich feedback, Diffusion-DRF achieves stable reward-based tuning without preference datasets collection. Diffusion-DRF achieves significant gains both quantitatively and qualitatively, outperforming state-of-the-art Flow-GRPO by 4.74% in overall performance on unseen VBench-2.0.

arXiv comment: Webpage: https://snap-research.github.io/diffusion-drf/

Ten-minute slide kit

Six slides is the whole talk: what was broken, what people tried, what these authors did, what the numbers say, where it falls over, and the sentence people should remember.

SLIDE 1

The problem

Recent advances in diffusion-based text-to-video generation \[18, 19, 25, 38, 47, [52\]](#page-17-1) have markedly improved fidelity, temporal coherence, and prompt adherence. Beyond architecture and scaling, a second wave of progress has come from post-training, inspired by alignment practices in LLMs \[28, 31, 34, [48\]](#page-17-2) and text-to-image diffusion \[7, 11, 14, [36\]](#page-16-4). The core motivation is to use preferencedriven objectives to steer pretrained generators toward human-preferred behaviors…

SLIDE 2

What came before

Recent progress in LLM post-training \[2, 27, [32\]](#page-16-6) has been adapted to visual generation to further improve output qual- Comparisons across reward tuning methods. ✓ indicates preferred. Existing approaches can generally be divided into two paradigms: reinforcement learning–based optimization and differentiable reward optimization. RLbased methods optimize generators using preference supervision through policystyle

SLIDE 3

The method

Given a text prompt c, a video diffusion model vθ(zt | c) generates a video latent z0 through iterative denoising from Gaussian noise zT in latent space, where T denotes the total number of diffusion steps. Our goal is to fine-tune the pretrained model vθ using feedback derived from a reward function f(v, c), where v denotes the predicted clean video in pixel space decoded from z0. Diffusion-DRF leverages a frozen, off-the-shelf Vision-Language Model (VLM) as a…

SLIDE 4

What they measured

We apply Diffusion-DRF to the pretrained Wan2.1-1.3B-T2V [\[38\]](#page-16-0) with Qwen2.5-VL-7B [\[1\]](#page-14-2) as the default VLM. We only train the DiT and freeze other components (VAE, text-encoder, and VLM). Videos are generated at 512×288 resolution with 49 frames under 25 denoise steps during training and 30 steps at inference. AdamW \[16, [23\]](#page-15-12) optimizer is used with a learning rate of 1e −5 , 32 A100 80GB GPUs, and batch size 1 per

SLIDE 5

Where it breaks

Not recoverable from the parsed text. Read this section in the paper yourself.

SLIDE 6

One-line takeaway

We'd also love to hear your hot takes on:

Assembled from the paper's own PDF, parsed with its layout intact, 78,209 characters of it, then split on the paper's own section headings. Extractive, not generated: every sentence here is lifted from the paper. The takeaway line is the club's own one-liner from its reading list. Check it before you present it.

Presented at

Saturday Robotics, session 11

🤖🥘 Saturday Robotics x Manycore Tech x Neural Motion | CVPR 2026 Denver Research Night | Robotics & World Models Reading Club 11

Denver, CO

Why the club picked it. We'd also love to hear your hot takes on: