1. Environment Setup
   • Set up onediff
   • Set up nexfort backend
   • Set up PixArt
2. Run
   • Run 1024*1024 without compile
   • Run 1024*1024 with compile
3. Performance Comparison
4. Quantization
   • Run
   • Metric
   • Optimizing Quantization Precision
5. Quality

https://github.com/siliconflow/onediff?tab=readme-ov-file#installation

https://github.com/siliconflow/onediff/tree/main/src/onediff/infer_compiler/backends/nexfort

HF model:

• PixArt-alpha: https://huggingface.co/PixArt-alpha/PixArt-XL-2-1024-MS
• PixArt-sigma: https://huggingface.co/PixArt-alpha/PixArt-Sigma-XL-2-1024-MS

HF pipeline: https://huggingface.co/docs/diffusers/main/en/api/pipelines/pixart

model_id_or_path_to_PixArt-XL-2-1024-MS is the model id or model path of pixart alpha, such as PixArt-alpha/PixArt-XL-2-1024-MS or /data/hf_models/PixArt-XL-2-1024-MS/

cd onediff

# To test sigma, specify the --model parameter as `PixArt-alpha/PixArt-Sigma-XL-2-1024-MS`.
python3 ./benchmarks/text_to_image.py \
--model PixArt-alpha/PixArt-XL-2-1024-MS \
--scheduler none \
--steps 20 \
--compiler none \
--output-image ./pixart_alpha.png \
--prompt "product photography, world of warcraft orc warrior, white background"

python3 ./benchmarks/text_to_image.py \
--model PixArt-alpha/PixArt-XL-2-1024-MS \
--scheduler none \
--steps 20 \
--compiler oneflow \
--output-image ./pixart_alpha_compile.png \
--prompt "product photography, world of warcraft orc warrior, white background"

python3 ./benchmarks/text_to_image.py \
--model PixArt-alpha/PixArt-XL-2-1024-MS \
--scheduler none \
--steps 20 \
--compiler nexfort \
--output-image ./pixart_alpha_compile.png \
--prompt "product photography, world of warcraft orc warrior, white background"

¹ OneDiff warmup with compilation time is tested on AMD EPYC 7543 32-Core Processor. Note this is just for reference, and it varies a lot on different CPU.

² Intel(R) Xeon(R) Gold 6348 CPU @ 2.60GHz.

³ Intel(R) Xeon(R) Platinum 8468.

• compiler-config
  • default is {"mode": "max-optimize:max-autotune:low-precision", "memory_format": "channels_last"} in /benchmarks/text_to_image.py. This mode supports dynamic shapes.
  • setting --compiler-config '{"mode": "max-autotune", "memory_format": "channels_last"}' will reduce compilation time and just slightly reduce the performance.
  • setting --compiler-config '{"mode": "max-optimize:max-autotune:freezing:benchmark:low-precision:cudagraphs", "memory_format": "channels_last"}' will help achieve the best performance, but it increases the compilation time and affects stability.
  • setting --compiler-config '{"mode": "jit:disable-runtime-fusion", "memory_format": "channels_last"}' will reduce compilation time to 20 seconds, but will reduce the performance.
• fuse_qkv_projections: True

Onediff's nexfort backend works closely with Torchao to support model quantization. Quant can reduce the runtime memory requirement and increase the inference speed.

python3 ./benchmarks/text_to_image.py \
--model PixArt-alpha/PixArt-XL-2-1024-MS \
--scheduler none \
--steps 20 \
--output-image ./pixart_alpha_fp8.png \
--prompt "product photography, world of warcraft orc warrior, white background" \
--compiler nexfort \
--compiler-config '{"mode": "quant:max-optimize:max-autotune:freezing:benchmark:low-precision:cudagraphs", "memory_format": "channels_last"}' \
--quantize \
--quantize-config '{"quant_type": "fp8_e4m3_e4m3_dynamic"}'

Optimal balance. Currently, multiple quant types are supported, such as int8_dynamic, fp8_e4m3_e4m3_weightonly, fp8_e4m3_e4m3_dynamic_per_tensor, fp8_e4m3_e4m3_weightonly_per_tensor. Additionally, the format of activations and weights supports switching between e4m3 and e5m2, such as fp8_e4m3_e5m2_dynamic.

It is recommended to try different types of quantization to find the optimal situation in terms of performance and quality.

NVIDIA H800 (1024 * 1024)

Quantization of the model's layers can be selectively performed based on precision. Download fp8_e4m3.json or per_tensor_fp8_e4m3.json from https://huggingface.co/siliconflow/PixArt-alpha-onediff-nexfort-fp8.

Run:

python3 ./benchmarks/text_to_image.py \
--model PixArt-alpha/PixArt-XL-2-1024-MS \
--scheduler none \
--steps 20 \
--output-image ./pixart_alpha_fp8_90%.png \
--prompt "product photography, world of warcraft orc warrior, white background" \
--compiler nexfort \
--compiler-config '{"mode": "quant:max-optimize:max-autotune:freezing:benchmark:low-precision:cudagraphs", "memory_format": "channels_last"}' \
--quantize \
--quantize-config '{"quant_type": "fp8_e4m3_e4m3_dynamic"}' \
--quant-submodules-config-path /path/to/fp8_e4m3.json

When using nexfort as the backend for onediff compilation acceleration, the generated images are lossless. With nexfort's quantized versions (fp8 or int8), there are only slight differences.