Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support importing wasm modules #2552

Closed
ry opened this issue Jun 20, 2019 · 29 comments · Fixed by #26668
Closed

Support importing wasm modules #2552

ry opened this issue Jun 20, 2019 · 29 comments · Fixed by #26668
Labels
feat new feature (which has been agreed to/accepted) web related to Web APIs

Comments

@ry
Copy link
Member

ry commented Jun 20, 2019

import * as foo from "./foo.wasm";

where foo will correspond to the exports of the wasm module.

@ry ry mentioned this issue Jun 20, 2019
@kitsonk
Copy link
Contributor

kitsonk commented Jun 21, 2019

For reference, here is the README about the integration. It is a Stage 1 proposal (and the WebAssembly committee follows the TC39 stage process), which means it is still quite early. I think this could be important to Deno, so maybe getting involved early would be a good thing, but is likely to be a bit of a rough road.

This will present interesting challenges for TypeScript and trying to understand the exports of a WASM module, but I suspect this would be an iterative process, where it would be just an untyped import, but eventually could try to figure out a way to evaluate/type on the fly the imports and exports of a WASM Module. It is going to be fairly complex I suspect, with some mediation between V8 and the compiler.

@kitsonk
Copy link
Contributor

kitsonk commented Sep 22, 2019

Just an update, the standard has moved to Stage 2. An important note in the proposal is that TLA is potentially needed to evaluate the module and hand it back to the ESM importer, so it is likely we need a version of TLA in V8 before we can really address this.

@dsseng
Copy link
Contributor

dsseng commented Nov 10, 2019

I have some ideas and I probably will work on this one. Compiler for that should produce smth like this:

const { instance } = await WebAssembly.instantiate(Uint8Array.from(atob("<WASM as base64>"), c => c.charCodeAt(0)));
module.exports = instance.exports;

.wasm file gets transformed to Base64 in compiler file (written in Rust) and then decoded & instantiated by Deno runtime, then exported. Instantiation function reference is here
P.S. I checked this manually, it works just fine.
P.P.S. module.exports seems to be invalid in Deno, maybe we should use an API like this that'll allow imports from JS to WASM too? Or we can just export the exports object.
@ry what do you think?

@kevinkassimo
Copy link
Contributor

We should be able to close this with #3328

@hayd
Copy link
Contributor

hayd commented Nov 14, 2019

It'd be great to have a toy example in the docs/manual.

@kevinkassimo
Copy link
Contributor

@hayd that could be a good first issue 😄

@kevinkassimo
Copy link
Contributor

@sh7dm Ooops, did not realize that you are working on this too... Only noticed this issue just now.
But basically I also thought of the base64 solution (and I guess it might work well with bundling). I implemented the thing by generating a dynamic script.

Would love to hear your feedback since currently in #3348 we have double compilation issue (and I did a hack to placate TS compiler... which might not be the most elegant thing)

@hayd
Copy link
Contributor

hayd commented Nov 14, 2019

@kevinkassimo I will add an example to the manual.

@hayd
Copy link
Contributor

hayd commented Nov 15, 2019

@kevinkassimo am i missing something here:

 $ cat fib.as.ts
export function fib(n: i32): i32 {
  var a = 0, b = 1;
  for (let i = 0; i < n; i++) {
    let t = a + b; a = b; b = t;
  }
  return b;
}

 $ asc fib.as.ts -b fib.wasm -O3

 $ cat fib.ts
import { fib } from "./fib.wasm";

console.log(fib(5));

 $ ./target/release/deno fib.ts
Compile file:///Users/hayd/OSS/deno/fib.ts
error: Uncaught ImportPrefixMissing: relative import path "env" not prefixed with / or ./ or ../
► $deno$/dispatch_json.ts:40:11
    at DenoError ($deno$/errors.ts:20:5)
    at unwrapResponse ($deno$/dispatch_json.ts:40:11)
    at sendAsync ($deno$/dispatch_json.ts:91:10)

Or is asc not doing the right thing?

@kevinkassimo
Copy link
Contributor

@hayd can you get asc output .wat format?

@kevinkassimo
Copy link
Contributor

@hayd I tried asc -t, and saw the following lines:

(module
  ;; ...
  (import "env" "abort" (func $~lib/builtins/abort (param i32 i32 i32 i32)))
  ;; ...
)

Looks like this result is prepared for npx asinit ., and running this command generates script index.js of the following format:

const fs = require("fs");
const compiled = new WebAssembly.Module(fs.readFileSync(__dirname + "/build/optimized.wasm"));
const imports = {
  env: {
    abort(_msg, _file, line, column) {
       console.error("abort called at index.ts:" + line + ":" + column);
    }
  }
};
Object.defineProperty(module, "exports", {
  get: () => new WebAssembly.Instance(compiled, imports).exports
});

This is why there is an attempt to import env. It seems asc does not handle the possibility of wasm imports

@kevinkassimo
Copy link
Contributor

kevinkassimo commented Nov 15, 2019

@hayd Figured out. You need to compile with the following flags:

asc --use "abort=" fib.as.ts -b fib.wasm -O3

This basically disables builtin abort thus removing the invalid import statement.

Now fib.ts works for me:

$ ./target/debug/deno ../test/wasm/fib.ts
Compile file:///Users/kun/Projects/Deno/test/wasm/fib.ts
8

@dsseng
Copy link
Contributor

dsseng commented Nov 15, 2019

@kevinkassimo no problem, just had an idea, didn't work on anything yet.

@rsp
Copy link
Contributor

rsp commented Jan 20, 2020

Could it be useful to run/import text files (wat/wit/witx) in addition to binary files?

  • by Deno bundling a wat to wasm compiler, like it bundles the tsc
  • by having a wat to wasm compiler in std
  • by bundling a fast s-expression parser

Some formats to keep an eye on:

  • wat (wasm text format)
  • wit (for module types)
  • witx (for WASI APIs)

The formats that describe the types might be especially useful.

@SyrupThinker
Copy link
Contributor

Any reason why this hasn't been closed?

@sithumonline
Copy link

Any reason why this hasn't been closed?

Manual documentation for wasm import is not complete. #3351 (review)

@SyrupThinker
Copy link
Contributor

The manual notes that wasm can be imported as it had been added later that day:
411f53f
https://deno.land/std/manual.md#wasm-support

@sithumonline
Copy link

@SyrupThinker , I think @ry will close #2552 & #3351

@ry ry closed this as completed May 1, 2020
@icyJoseph
Copy link

As of #5135 this has been temporarily removed. Search engine results however still do not show #5135 above this issue.

@Ciantic
Copy link

Ciantic commented Jul 18, 2021

Can we re-open this? Or add some tag at least to indicate it's waiting for ECMAScript Stage 3.

I had to manually go through the thread to see why this was closed, only to find out the support was once added and then removed.

@lucacasonato lucacasonato reopened this Jul 18, 2021
@kitsonk kitsonk added suggestion suggestions for new features (yet to be agreed) web related to Web APIs labels Aug 3, 2021
@kitsonk
Copy link
Contributor

kitsonk commented Aug 3, 2021

Update

This is the status page on the platform status: https://www.chromestatus.com/feature/5987156463976448#details

There is no active development in Chromium and there doesn't look like there is a lot of interest or activity there. Seems everyone has accepted the "fetch" instantiate streaming method. The big challenges with that for Deno are:

  • Requires explicit permissions to be able to load web assembly
  • fetch() does not currently support local files, which means you have to offload it to Deno.readFile() but that isn't currently easily interoperable with web streams.

It won't come down the TC39 proposal route, because it has "nothing" to do with ECMAScript.

Currently the proposal does not use ECMAScript import assertions: https://webassembly.github.io/esm-integration/js-api/index.html#esm-integration. I haven't found a relevant discussion where the Web Assembly committee is considering it either. It was briefly discussed here, but no clear actions: WebAssembly/esm-integration#42 nor any obvious progress.

@guybedford
Copy link
Contributor

Has there been any progress on discussion on #2150?

@kitsonk
Copy link
Contributor

kitsonk commented Aug 3, 2021

Has there been any progress on discussion on #2150?

Yes. Within the core team. It is something we are going to discuss in depth soon. We just have to get over how we deal with window.location and related APIs when a --location is not provided.

@lucacasonato lucacasonato added feat new feature (which has been agreed to/accepted) and removed suggestion suggestions for new features (yet to be agreed) labels Apr 27, 2022
@lucacasonato
Copy link
Member

There has been considerable movement on this (thanks @takikawa). There is now an experimental implementation of WASM imports in WebKit, behind a flag. We should implement WASM imports again as per the most up to date specs: https://webassembly.github.io/esm-integration/js-api/index.html#esm-integration and whatwg/html#4372.

I think the general approach for us would be the following:

  1. deno_graph needs to understand WASM modules. It needs to be able to analyze imports from within WASM.
  2. deno_core needs to have a WASM import type.
    • We need to parse the module.
    • Instantiate the module with it's "imports" set to the imports parsed out by deno_graph.
    • Create a synthetic module with it's module namespace set to the exports of the instantiated module.

Part 2 could be simplified once V8 implements WASM modules itself. The deno_graph changes are required regardless.

@bartlomieju
Copy link
Member

I can work on the deno_core part, should be straight-forward. @kitsonk could you look into deno_graph support?

@sant123
Copy link

sant123 commented Jul 1, 2022

I'm curious in how the wasm module will be imported. If using the assert import or not.

@redradist
Copy link

Is there any progress on it ?

Wasm Modules is currently at Stage 2 and usually Deno implement interesting features at this stage ...

@nicolo-ribaudo
Copy link

Fyi, whatwg/html#10380 has been merged and the Wasm proposal is at phase 3 ("implementation phase").

@kevinkassimo
Copy link
Contributor

It's finally back after 5 years 🥲

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feat new feature (which has been agreed to/accepted) web related to Web APIs
Projects
None yet
Development

Successfully merging a pull request may close this issue.