Motivation

Supporting Wan Animate (https://huggingface.co/Wan-AI/Wan2.2-Animate-14B-Diffusers)

See issue #13867

Based on PR #15419

Modifications

This PR introduces a customized orchestration pipeline for the Wan Animate model. By implementing a Segment Loop mechanism, it overcomes VRAM limitations for long video generation. Additionally, it integrates a Data Preprocessing stage that supports direct extraction of pose and face features from raw videos, significantly lowering the barrier to entry for end-users.

Pipeline Design

WanAnimatePipeline:

Preprocessing -> Validation -> Text/Image Encoding -> VAE Encoding -> Video Processing -> Condition Construction -> Segment Loop

SegmentLoopStage

• The SegmentLoopStage manages the cur_segment index and orchestrates a sub-pipeline consisting of five stages: Conditioning → Timestep → Latent → Denoising → Decoding
• The WanAnimateConditioningStage dynamically slices the Pose/Face conditions for the current segment.
• The DecodingStage triggers postprocess_decoded_frames for incremental frame stitching and returns a Req object to drive the next iteration or returns OutputBatch to terminate.

Data Preprocessing Stage

We provide a Data Preprocessing Stage, if users provide the onnx model path of the preprocess model, we can directly process the original video passed in by the user instead of the user passing in the preprocessed pose video and face video.

With preprocess_model_path:

sglang generate --model-path Wan-AI/Wan2.2-Animate-14B-Diffusers \
 --prompt='People in the video are doing actions.' --image-path [ref image path] --video-path [video path] --preprocess-model-path [process model path] --width  1280 --height 720 --save-output 

Without preprocess_model_path:

sglang generate --model-path Wan-AI/Wan2.2-Animate-14B-Diffusers\\n --prompt='People in the video are doing actions.'  --image-path [ref image path] --pose-video-path [processed pose video path] --face-video-path [processed face video path] --width  1280 --height 720 --save-output 

TODO

• replace mode for wan animate
• retarget and use flux for data preprocess

Accuracy Tests

Replicate.com wan animate animation:

Our results:

Checklist

• Format your code according to the Format code with pre-commit.
• Add unit tests according to the Run and add unit tests.
• Update documentation according to Write documentations.
• Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
• Follow the SGLang code style guidance.
• Work with maintainers to merge your PR. See the PR Merge Process