Feat; Add support for Wan/Qwen TAEHV decoding #937
+368
−31
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
Author
|
Sorry for the unrelated whitespace changes and the debug spam, will fix later |
Contributor
Author
|
Oh a new version of the taew2.1 weights just came out, coincidentally.
|
Contributor
Author
|
Now tae decoding for the outputs of Wan2.1 models (and Wan2.2 A14B) works in txt2img mode. Video decoding is running as well, but the results are obviously incorrect (flashing lights warning) If someone can see what I'm doing wrong when decoding videos, let me know. |
|
After fixing the three bugs mentioned in review, image results look correct (tested on GH200 with diffsdiff --git a/tae.hpp b/tae.hpp
index ad0bd37..6a7951f 100644
--- a/tae.hpp
+++ b/tae.hpp
@@ -224,7 +224,7 @@ public:
h = conv1->forward(ctx, h);
h = ggml_relu_inplace(ctx->ggml_ctx, h);
h = conv2->forward(ctx, h);
- h = ggml_relu_inplace(ctx->ggml_ctx, h);
+ // h = ggml_relu_inplace(ctx->ggml_ctx, h);
auto skip = x;
if (has_skip_conv) {
@@ -323,7 +323,7 @@ public:
for (int i = 0; i < num_layers; i++) {
for (int j = 0; j < num_blocks; j++) {
auto block = std::dynamic_pointer_cast<MemBlock>(blocks[std::to_string(index++)]);
- auto mem = ggml_pad(ctx->ggml_ctx, h, 0, 0, 0, 1);
+ auto mem = ggml_pad_ext(ctx->ggml_ctx, h, 0, 0, 0, 0, 0, 0, 1, 0);
mem = ggml_view_4d(ctx->ggml_ctx, mem, h->ne[0], h->ne[1], h->ne[2], h->ne[3], h->nb[1], h->nb[2], h->nb[3], 0);
h = block->forward(ctx, h, mem);
}
@@ -341,7 +341,7 @@ public:
h = last_conv->forward(ctx, h);
// shape(W, H, 3, T+3) => shape(W, H, 3, T)
- h = ggml_view_4d(ctx->ggml_ctx, h, h->ne[0], h->ne[1], h->ne[2], h->ne[3] - 3, h->nb[1], h->nb[2], h->nb[3], 0);
+ h = ggml_view_4d(ctx->ggml_ctx, h, h->ne[0], h->ne[1], h->ne[2], h->ne[3] - 3, h->nb[1], h->nb[2], h->nb[3], 3*h->nb[3]);
return h; |
madebyollin
suggested changes
Dec 11, 2025
Co-authored-by: Ollin Boer Bohan <[email protected]>
Contributor
Author
|
Video is still completely broken, but image decoding works very well now. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
https://github.com/madebyollin/taehv
Model weights: https://github.com/madebyollin/taehv/blob/main/taew2_1.pth
Only tested "successfuly" for decoding Qwen-Image outputs, still need some work to support video models. Encoding seems to work too, at least in img2img mode.
.\bin\Release\sd.exe --diffusion-model ..\..\ComfyUI\models\diffusion_models\qwen-image-Q8_0.gguf --vae ..\..\ComfyUI\models\vae\qwen_image_vae.safetensors --qwen2vl ..\..\ComfyUI\models\text_encoders\Qwen2.5-VL-7B-Instruct-Q8_0.gguf -p '一个穿着"QWEN"标志的T恤的中国美女正拿着黑色的马克笔面相镜头微笑。她身后的玻璃板上手写体写着 “一、Qwen-Image的技术路线： 探索视觉生成基础模型的极限，开创理解与生成一体化的未来。二、Qwen-Image的模型特色：1、复杂文字渲染。支持中英渲染、自动布局； 2、精准图像编辑。支持文字编辑、物体增减、风格变换。三、Qwen-Image的未来愿景：赋能专业内容创作、助力生成式AI发展。”' --cfg-scale 2.5 --sampling-method euler -v --offload-to-cpu -H 1024 -W 1024 --diffusion-fa --flow-shift 3 --tae ..\ComfyUI\models\vae_approx\taew2_1.pth --vae-conv-directSpeedup and memory saving aren't that impressive yet, maybe it can be improved further?