Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tracking issue for using WebGPU to render on the window of the desktop environment #23563

Open
5 of 8 tasks
chirsz-ever opened this issue Apr 25, 2024 · 8 comments
Open
5 of 8 tasks
Labels
webgpu WebGPU API

Comments

@chirsz-ever
Copy link
Contributor

chirsz-ever commented Apr 25, 2024

So far we have WebGPU and BYOW, but it need more work to use the front-end graphics ecology.

Repository for testing and integrating: https://github.com/chirsz-ever/deno-webgpu-window-demos

Seamlessly use WebGPU-backended frameworks

Bugs and requirements

Creating and managing native windows

We already have deno_sdl2 and dwm for creating windows. But they rely on third-party libraries SDL2 and GLFW.

Maybe we need some methods to create windows without any third-party libraries? I have two ideas:

  • import winit and design a set of interfaces to use it;1
  • only write JavaScript code, use FFI to create window on respective platform, as a third-party library.

The first idea is easier to implement, but add a built-in component increases deno's compatibility costs and technical debt.

Canvas, CanvasRenderingContext2D and WebGL

Related issues: #5701 #1629

If deno support Canvas, CanvasRenderingContext2D and WebGL, it would be more easy to use front-end graphics libraries.

Deno.UnsafeWindowSurface is analog to Canvas, and it can use getContext("webgpu") to create a GPUCanvasContext. It is worthwhile to support other contexts, getContext("2d"), getContext("webgl"), getContext("webgl2") and getContext("bitmaprenderer").

How to implement CanvasRenderingContext2D (Canvas.getContext("2d"))? There are many methods in #5701, which mostly introduces skia or similar graphics libraries like Vello.

For WebGL, we can just use the native OpenGL implementation (gluton)[https://github.com/deno-windowing/gluten], or import ANGLE to maximize compliance of the WebGL standard.1

Maybe we can do something named "Canvas2D over WebGPU" and "WebGL over WebGPU", which only use JavaScript to implement CanvasRenderingContext2D and WebGL with WebGPU.

If we resolve all above, implementing OffscreenCanvas and ImageBitmapRenderingContext (Canvas.getContext("bitmaprenderer")) seems not hard.

Footnotes

  1. Removed by https://github.com/denoland/deno/issues/23563#issuecomment-2077777430 2

@crowlKats
Copy link
Member

We will not add a dependency on winit or any other windowing system, the way we have it currently is the best compromise.

Regarding Canvas and related:
for OffscreenCanvas, I have a local branch that needs some further work. The idea is that UnsafeWindowSurface will extend OffscreenCavas.
We will not implement a webgl context or any webgl related APIs, however we likely will implemented 2dcontext via Vello.

@chirsz-ever
Copy link
Contributor Author

Why don't allow "none" colorSpaceConversion when createImageBitmap?

https://github.com/denoland/deno/blame/c5193556242117737fc3ba95ee3692b7831ca04a/ext/canvas/01_image.js#L307-L309

@chirsz-ever
Copy link
Contributor Author

Is this a bug? I think the alpha value do not need to multiply 255. And why there need a is_not_premultiplied flag calculated from every pixels?

https://github.com/denoland/deno/blame/c5193556242117737fc3ba95ee3692b7831ca04a/ext/canvas/lib.rs#L86

@crowlKats crowlKats added the webgpu WebGPU API label May 5, 2024
@chirsz-ever
Copy link
Contributor Author

chirsz-ever commented May 6, 2024

For ImageBitmap :

Footnotes

  1. https://stackoverflow.com/a/60033582

  2. https://github.com/gpuweb/gpuweb/issues/1154

@crowlKats
Copy link
Member

one you missed that uses ImageBitmap is https://gpuweb.github.io/gpuweb/#gpuimagecopyexternalimage

@crowlKats
Copy link
Member

  • Use pixel format other than RGBA, such as that in image::DynamicImage. This would save memory usage.

sounds reasonable

  • Use a GPUTexture as the underlying data resource when possible. This improves performance because you don't need to transfer data from and to the GPU.

I don't think this is viable, since to create a GPUTexture you need a GPUDevice, which is not feasible since a user could use a different GPUDevice, or same GPUDevice but with different features.

@chirsz-ever
Copy link
Contributor Author

We could hold the GPUDevice object in a ImageBitmap object when the underlying resource is a GPUTexture. When we want to perform some operations, we first check whether the devices on both sides are the same, and then directly perform the operation, or copy the data to the CPU side and then transfer it to the target.

This optimization would not be implemented in the near future, so it is recorded here just for reference.

@chances
Copy link

chances commented Aug 23, 2024

Regarding three.js, see jsr.io/@3d/three.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
webgpu WebGPU API
Projects
None yet
Development

No branches or pull requests

3 participants