NeurIPS 2026·Sydney

RAVEN: Real-time Autoregressive Video Extrapolation with Consistency-model GRPO

Yanzuo Lu·Ronglai Zuo·Jiankang Deng

Imperial College London

Paper Code Models ComfyUI

Real-time streaming comparison on Wan2.1-T2V-1.3B. Each cell is a different causal video distillation method; the two right-most cells are ours.

01 · Overview

Abstract and contributions

Causal autoregressive video diffusion models support real-time streaming generation by extrapolating future chunks from previously generated content. Distilling such generators from high-fidelity bidirectional teachers yields competitive few-step models, yet a persistent gap between the history distributions encountered during training and those arising at inference constrains generation quality over long horizons. We introduce the Real-time Autoregressive Video Extrapolation Network (RAVEN), a training-time test framework that repacks each self rollout into an interleaved sequence of clean historical endpoints and noisy denoising states. This formulation aligns training attention with inference-time extrapolation and allows downstream chunk losses to supervise the history representations on which future predictions depend. We further propose Consistency-model Group Relative Policy Optimization (CM-GRPO), which reformulates a consistency sampling step as a conditional Gaussian transition and applies online Reinforcement Learning (RL) directly to this kernel, avoiding the Euler–Maruyama auxiliary process adopted in prior flow-model RL formulations. Experiments demonstrate that RAVEN surpasses recent causal video distillation baselines across quality, semantic, and dynamic degree evaluations, and that CM-GRPO provides further gains when combined with RAVEN.

  1. 01

    We identify a history supervision gap in autoregressive video diffusion distillation, where existing methods are either optimized under history distributions that differ from inference or conditioned on rollout history without end-to-end supervision.

  2. 02

    We introduce RAVEN, a training-time test framework that repacks self rollouts into an interleaved sequence of clean historical endpoints and noisy denoising states, allowing supervision to propagate through the history representations used during extrapolation.

  3. 03

    We propose CM-GRPO, which reformulates a consistency sampling step as a conditional Gaussian transition kernel and applies group relative policy optimization directly to this kernel, matching the sampler interface used at inference.

  4. 04

    We demonstrate that RAVEN surpasses recent causal video distillation baselines and that CM-GRPO provides complementary gains when combined with RAVEN.

02 · Method

Training-time test and CM-GRPO

RAVEN: training-time test for autoregressive distillation

A causal generator distilled from a bidirectional teacher is trained on history drawn from data or from detached rollouts, yet at inference it extrapolates from frames it has generated itself, so the history distribution shifts between training and streaming. RAVEN closes this gap with a training-time test: each self rollout is repacked into an interleaved sequence of clean historical endpoints and noisy denoising states, processed under a causal attention mask that mirrors inference-time extrapolation. Because later chunks attend to the cached history that the generator itself produced, their losses propagate back into the history representations on which future predictions depend.

Attention mask configurations of Teacher Forcing, Diffusion Forcing, Self Forcing and RAVEN.
Attention mask configuration. Teacher Forcing and Diffusion Forcing rely on data-driven historical states, inducing a training distribution that differs from inference. Self Forcing shifts the history distribution toward inference but reuses the historical cache as detached context. RAVEN repacks each self rollout into clean historical endpoints and noisy denoising states, so later chunks attend to the same history used during extrapolation while their losses supervise the cached representations.
RAVEN training pipeline with the fake-score step and the generator step.
Training pipeline. In the fake-score step, the frozen generator performs autoregressive self rollout with KV cache reuse, producing the clean endpoints and noisy denoising states that are reused in the generator step. RAVEN repacks them into an interleaved sequence processed under a causal attention mask, so that each noisy state attends to the clean history the generator itself produced, and later chunk losses, scaled chunk-wise, supervise the history representations on which future predictions depend.

CM-GRPO: online RL on the consistency transition kernel

A consistency sampling step from noise level u to a lower level s predicts a clean endpoint x̂θ = fθ(z̃(u), u, c) and draws the next latent as z̃(s) = αs x̂θ + σs ε with ε ∼ 𝒩(0, I). This rule already induces a Gaussian transition

πθ(z̃(s) | z̃(u), c) = 𝒩(z̃(s); αs x̂θ, σs2 I)

so CM-GRPO applies group relative policy optimization directly to this kernel. For each condition the generator runs G independent consistency trajectories, scores every final output with a reward, and updates the policy with group-relative advantages through a stop-gradient regression objective. Unlike Flow-GRPO, no auxiliary Euler–Maruyama SDE is introduced, and the stochastic transitions being optimized are exactly the ones used by the sampler at inference.

03 · Results

Comparison with causal distillation baselines

Quantitative comparison

Method Total Score Quality Score Semantic Score Dynamic Degree
CausVid83.0184.1878.342.340
LongLive83.0583.7080.462.277
Rolling Forcing83.2584.0080.252.536
Self Forcing84.2785.1080.972.543
Reward Forcing84.3985.2780.872.508
Causal Forcing84.9686.0080.762.669
+CM-GRPO85.0886.1280.962.829
RAVENours85.1586.1881.042.951
+CM-GRPOours85.4686.5481.172.962

VBench Total, Quality and Semantic scores follow the protocol shared by all compared baselines. Dynamic Degree is evaluated with UnifiedReward-32B on the 6,220 videos generated under the VBench prompt suite.

Qualitative comparison

Each clip compares Causal Forcing and RAVEN, with and without CM-GRPO, on the same prompt across animal motion, urban scenery and human subjects.

Ablation on the training-time test

Teacher Forcing, Diffusion Forcing, Self Forcing, Diffusion Forcing with self rollout, and RAVEN, each generating the same prompt.

User study

We conduct a user study on 100 long and detailed prompts drawn from the qualitative showcases of the existing baselines, generating 4 samples per prompt for each method. The study covers the four baselines designed for 5-second short video generation, namely CausVid, Self Forcing, Reward Forcing, and Causal Forcing. For each sample pair, an individual user rates a RAVEN clip against its baseline counterpart presented in randomized order along Quality, Semantic, and Overall. Aggregate preference rates are reported below, where RAVEN is preferred on every dimension against all four baselines, with a more pronounced lead on Semantic than on Quality and a clear margin on Overall.

User study preference rates on Quality, Semantic, and Overall.
User study preference rates on Quality, Semantic, and Overall.
04 · Releases

Models, code and news

All weights are hosted in the mvp-lab/RAVEN collection on Hugging Face. Training, inference and evaluation code live in the GitHub repository.

Paper models

Wan2.1-T2V-1.3B

Base model
Wan-AI/Wan2.1-T2V-1.3B (causal variant)
Resolution
480 × 832
Frames
81
Frame rate
16 fps
Sampling
4 NFE, consistency sampler
License
CC BY-NC 4.0
Hugging Face
Beyond the paper · preview

MiniMax-H3 streaming LoRA

Base model
MiniMaxAI/MiniMax-H3
Adapter
LoRA, r = 128, α = 128
Resolution
768 × 1376
Frames
192
Frame rate
24 fps
Sampling
4 NFE (video and audio)
Output
text-to-video-and-audio
Causal chunking
sink = 2, window = 2
License
MiniMax-H3 Community License

Preview release. The adapter is still undertrained and its texture detail remains limited, but it establishes the complete end-to-end RAVEN training pipeline for streaming generation on MiniMax-H3. It is not one of the models evaluated in the paper, and reaching real-time generation on a model of this size still requires further inference acceleration. The ComfyUI nodes run 192-frame 1376 × 768 text-to-video-and-audio generation within a 24 GiB VRAM envelope.

Hugging Face ComfyUI nodes Comfy Registry

MiniMax-H3 base model at 50 NFE (left) versus the RAVEN streaming LoRA at 4 NFE (right). The audio track is generated jointly by the streaming model. Unmute to listen.

News

  1. Accepted to NeurIPS 2026. See you in Sydney!
  2. ComfyUI nodes for RAVEN streaming generation on MiniMax-H3 released, enabling 192-frame 1376 × 768 text-to-video-and-audio generation within a 24 GiB VRAM envelope.
  3. MiniMax-H3 training code and the initial 4-NFE preview LoRA released.
  4. Paper, code and Wan2.1-T2V-1.3B weights released.
05 · Acknowledgements

Acknowledgements

We thank Fei Ni and Changrui Chen for their valuable discussions and assistance. J. Deng was supported by the NVIDIA Academic Grant.

The authors acknowledge the use of resources provided by the Isambard-AI National AI Research Resource (AIRR). Isambard-AI is operated by the University of Bristol and is funded by the UK Government’s Department for Science, Innovation and Technology (DSIT) via UK Research and Innovation; and the Science and Technology Facilities Council [ST/AIRR/I-A-I/1023].

06 · Citation

Citation

If you find this work useful, please cite RAVEN.

@inproceedings{lu2026raven,
  title = {RAVEN: Real-time Autoregressive Video Extrapolation with Consistency-model GRPO},
  author = {Lu, Yanzuo and Zuo, Ronglai and Deng, Jiankang},
  booktitle = {Advances in Neural Information Processing Systems},
  year = 2026
}