THIS REPOSITORY IS OBSOLETE, USE https://github.com/dakenf/diffusers.js
This is a pure typescript implementation of SD pipeline that runs ONNX versions of the model with patched ONNX node runtime
Warning: this project requires Node 18
Works out of the box with DirectML. No additional libraries required
You can speed up things by installing tfjs-node, but i haven't seen significant performance improvements https://github.com/tensorflow/tfjs/tree/master/tfjs-node
tfjs-node might require installing visual studio build tools and python 2.7 https://community.chocolatey.org/packages/visualstudio2022buildtools
- Install CUDA (tested only on 11-7 but 12 should be supported) https://docs.nvidia.com/cuda/cuda-installation-guide-linux/
- Install cuDNN https://developer.nvidia.com/rdp/cudnn-archive
- Install onnxruntime-linux-x64-gpu-1.14.1 https://github.com/microsoft/onnxruntime/releases/tag/v1.14.1
No requirements but can run only on CPU which is quite slow (about 0.2s/it for fp32 and 0.1s/it for fp16)
npm i stable-diffusion-nodejs
import { PNG } from 'pngjs'
import { StableDiffusionPipeline } from 'stable-diffusion-nodejs'
const pipe = await StableDiffusionPipeline.fromPretrained(
'directml', // can be 'cuda' on linux or 'cpu' on mac os
'aislamov/stable-diffusion-2-1-base-onnx', // relative path or huggingface repo with onnx model
)
const image = await pipe.run("A photo of a cat", undefined, 1, 9, 30)
const p = new PNG({ width: 512, height: 512, inputColorType: 2 })
p.data = Buffer.from((await image[0].data()))
p.pack().pipe(fs.createWriteStream('output.png')).on('finish', () => {
console.log('Image saved as output.png');
})import * as tf from "@tensorflow/tfjs-node"
import { StableDiffusionPipeline } from 'stable-diffusion-nodejs'
const pipe = await StableDiffusionPipeline.fromPretrained(
'directml', // can be 'cuda' on linux or 'cpu' on mac os
'aislamov/stable-diffusion-2-1-base-onnx', // relative path or huggingface repo with onnx model
)
const image = await pipe.run("A photo of a cat", undefined, 1, 9, 30)
const png = await tf.node.encodePng(image[0])
fs.writeFileSync("output.png", png);import * as tf from "@tensorflow/tfjs-node"
import { StableDiffusionPipeline } from 'stable-diffusion-nodejs'
const pipe = await StableDiffusionPipeline.fromPretrained(
'directml', // can be 'cuda' on linux or 'cpu' on mac os
'CompVis/stable-diffusion-v1-4',
'onnx', // hf hub revision
1, // SD version, cannot detect automatically yet
)
const image = await pipe.run("A photo of a cat", undefined, 1, 9, 30)
const png = await tf.node.encodePng(image[0])
fs.writeFileSync("output.png", png);To test inference run this command. It will download SD2.1 onnx version from huggingface hub
npm run txt2img -- --prompt "an astronaut riding a horse" --provider directml
npm run txt2img -- --prompt "an astronaut riding a horse" --provider cuda
You can also use --provider cpu on a mac or if you don't have a supported videocard
You can use this tool to convert any HF hub model to ONNX https://github.com/Amblyopius/Stable-Diffusion-ONNX-FP16 Use fp16 for Cuda/DirectML and fp32 for Apple M1 (it runs twice faster but still slow)
- Support different schedulers, like DDIMS and UniPCMultistepScheduler
- Support batch size > 1
- ControlNet support
- Add interop between ONNX backend and tensorflow.js to avoid copying data from and to GPU on each inference step