Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Is it supported to build zlib-ng on emscripten for webassembly? #1758

Open
LugPaj opened this issue Jul 9, 2024 · 6 comments
Open

Is it supported to build zlib-ng on emscripten for webassembly? #1758

LugPaj opened this issue Jul 9, 2024 · 6 comments

Comments

@LugPaj
Copy link

LugPaj commented Jul 9, 2024

Currently it crashes here on our side, when we try a compiled version. Not sure, it our compilation is correct though. So are there any experiences, that it should run on webassembly?
@nmoinvaz
Copy link
Member

nmoinvaz commented Jul 9, 2024 •
edited
Loading

We do run emscripten in our CI, so if you can figure out what you are doing differently to cause it to crash that would be a good first step.

zlib-ng/.github/workflows/cmake.yml

Lines 452 to 455 in d54e376

- name: Ubuntu Emscripten WASM32
os: ubuntu-latest
chost: wasm32
cmake-args: -DCMAKE_TOOLCHAIN_FILE=${EMSDK}/upstream/emscripten/cmake/Modules/Platform/Emscripten.cmake -DCMAKE_C_COMPILER_TARGET=wasm32 -DCMAKE_CROSSCOMPILING_EMULATOR=${EMSDK_NODE} -DZLIB_COMPAT=ON

@kleisauke
Copy link

wasm-vips has been using zlib-ng since its initial release without any issues.
https://github.com/kleisauke/wasm-vips/blob/6b83b72ef69c7c17755d8533d290909d8e98408e/build.sh#L237-L248

@milesgranger
Copy link

milesgranger commented Sep 17, 2024 •
edited
Loading

I'm experiencing issues getting v2.07 (used by c-blosc2) to compile with target of wasm32 emscripten.

cpuid.h:14:2: error: this header is for x86 only
14 | #error this header is for x86 only

I've referenced the CI workflow here and tried to set the cmake args with no effect; maybe I'm doing something silly, any tips are appreciated. :)

Full traceback 
blosc2-rs on  emscripten-mode is 📦 v0.3.0+2.15.1 via 🦀 v1.81.0
❯ cargo build --target wasm32-unknown-emscripten --features deactivate-zlib
warning: version requirement `0.3.1+2.15.1` for dependency `blosc2-sys` includes semver metadata which will be ignored, removing the metadata is recommended t
   Compiling blosc2-sys v0.3.1+2.15.1 (/home/milesg/Projects/blosc2-rs/blosc2-sys)
   Compiling blosc2-rs v0.3.1+2.15.1 (/home/milesg/Projects/blosc2-rs)
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 10.59s

blosc2-rs on  emscripten-mode [!] is 📦 v0.3.1+2.15.1 via 🦀 v1.81.0 took 10s
❯ cargo build --target wasm32-unknown-emscripten
warning: version requirement `0.3.1+2.15.1` for dependency `blosc2-sys` includes semver metadata which will be ignored, removing the metadata is recommended t
   Compiling blosc2-sys v0.3.1+2.15.1 (/home/milesg/Projects/blosc2-rs/blosc2-sys)
error: failed to run custom build command for `blosc2-sys v0.3.1+2.15.1 (/home/milesg/Projects/blosc2-rs/blosc2-sys)`

Caused by:
  process didn't exit successfully: `/home/milesg/Projects/blosc2-rs/target/debug/build/blosc2-sys-d8b20029052f49c6/build-script-build` (exit status: 101)
  --- stdout
  cargo:rerun-if-changed=build.rs
  CMAKE_TOOLCHAIN_FILE_wasm32-unknown-emscripten = None
  CMAKE_TOOLCHAIN_FILE_wasm32_unknown_emscripten = None
  TARGET_CMAKE_TOOLCHAIN_FILE = None
  CMAKE_TOOLCHAIN_FILE = None
  CMAKE_GENERATOR_wasm32-unknown-emscripten = None
  CMAKE_GENERATOR_wasm32_unknown_emscripten = None
  TARGET_CMAKE_GENERATOR = None
  CMAKE_GENERATOR = None
  CMAKE_PREFIX_PATH_wasm32-unknown-emscripten = None
  CMAKE_PREFIX_PATH_wasm32_unknown_emscripten = None
  TARGET_CMAKE_PREFIX_PATH = None
  CMAKE_PREFIX_PATH = None
  EMCMAKE_wasm32-unknown-emscripten = None
  EMCMAKE_wasm32_unknown_emscripten = None
  TARGET_EMCMAKE = None
  EMCMAKE = None
  CMAKE_wasm32-unknown-emscripten = None
  CMAKE_wasm32_unknown_emscripten = None
  TARGET_CMAKE = None
  CMAKE = None
  running: cd "/home/milesg/Projects/blosc2-rs/target/wasm32-unknown-emscripten/debug/build/blosc2-sys-6e55cedb3f4350b4/out/build" && CMAKE_PREFIX_PATH="" "em
  -- The ASM compiler identification is unknown
  -- Found assembler: /home/milesg/Projects/emsdk/upstream/emscripten/emcc
  -- Warning: Did not find file Compiler/-ASM
  -- Performing Test CMAKE_HAVE_LIBC_PTHREAD
  -- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
  -- Found Threads: TRUE
  -- Using LZ4 internal sources.
  -- Using ZLIB-NG internal sources for ZLIB support.
  -- Using CMake version 3.28.2
  -- ZLIB_HEADER_VERSION: 1.2.11
  -- ZLIBNG_HEADER_VERSION: 2.0.7
  -- Arch not recognized, falling back to cmake arch: 'x86'
  -- Basearch 'x86' not recognized, defaulting to 'x86'.
  -- Basearch of 'x86' has been detected as: 'x86'
  -- Using CMake toolchain: /home/milesg/Projects/emsdk/upstream/emscripten/cmake/Modules/Platform/Emscripten.cmake
  -- Performing Test FNO_LTO_AVAILABLE
  -- Performing Test FNO_LTO_AVAILABLE - Success
  -- Architecture supports unaligned reads
  -- Architecture supports unaligned reads of > 4 bytes
  -- Looking for sys/sdt.h
  -- Looking for sys/sdt.h - not found
  -- Looking for unistd.h
  -- Looking for unistd.h - found
  -- Looking for sys/types.h
  -- Looking for sys/types.h - found
  -- Looking for stdint.h
  -- Looking for stdint.h - found
  -- Looking for stddef.h
  -- Looking for stddef.h - found
  -- Check size of off64_t
  -- Check size of off64_t - done
  -- Looking for fseeko
  -- Looking for fseeko - found
  -- Looking for strerror
  -- Looking for strerror - found
  -- Looking for posix_memalign
  -- Looking for posix_memalign - found
  -- Looking for aligned_alloc
  -- Looking for aligned_alloc - found
  -- Performing Test HAVE_NO_INTERPOSITION
  -- Performing Test HAVE_NO_INTERPOSITION - Success
  -- Performing Test HAVE_ATTRIBUTE_VISIBILITY_HIDDEN
  -- Performing Test HAVE_ATTRIBUTE_VISIBILITY_HIDDEN - Success
  -- Performing Test HAVE_ATTRIBUTE_VISIBILITY_INTERNAL
  -- Performing Test HAVE_ATTRIBUTE_VISIBILITY_INTERNAL - Success
  -- Performing Test HAVE_BUILTIN_CTZ
  -- Performing Test HAVE_BUILTIN_CTZ - Success
  -- Performing Test HAVE_BUILTIN_CTZLL
  -- Performing Test HAVE_BUILTIN_CTZLL - Success
  -- Performing Test HAVE_PTRDIFF_T
  -- Performing Test HAVE_PTRDIFF_T - Success
  -- Performing Test HAVE_SSE2_INTRIN
  -- Performing Test HAVE_SSE2_INTRIN - Failed
  -- Performing Test HAVE_SSSE3_INTRIN
  -- Performing Test HAVE_SSSE3_INTRIN - Failed
  -- Performing Test HAVE_SSE42CRC_INLINE_ASM
  -- Performing Test HAVE_SSE42CRC_INLINE_ASM - Failed
  -- Performing Test HAVE_SSE42CRC_INTRIN
  -- Performing Test HAVE_SSE42CRC_INTRIN - Failed
  -- Performing Test HAVE_SSE42CMPSTR_INTRIN
  -- Performing Test HAVE_SSE42CMPSTR_INTRIN - Failed
  -- Performing Test HAVE_PCLMULQDQ_INTRIN
  -- Performing Test HAVE_PCLMULQDQ_INTRIN - Failed
  -- Performing Test HAVE_AVX2_INTRIN
  -- Performing Test HAVE_AVX2_INTRIN - Failed
  -- Architecture-specific source files: arch/x86/x86.c
  -- Renaming
  --     /home/milesg/Projects/blosc2-rs/target/wasm32-unknown-emscripten/debug/build/blosc2-sys-6e55cedb3f4350b4/out/c-blosc2/internal-complibs/zlib-ng-2.0.7
  -- to 'zconf.h.included' because this file is included with zlib
  -- but CMake generates it automatically in the build directory.
  -- The following features have been enabled:

   * CMAKE_BUILD_TYPE, Build type: Debug (selected)
   * WITH_GZFILEOP, Compile with support for gzFile related functions
   * ZLIB_COMPAT, Compile with zlib compatible API
   * WITH_OPTIM, Build with optimisation
   * WITH_NEW_STRATEGIES, Use new strategies
   * WITH_UNALIGNED, Support unaligned reads on platforms that support it
   * WITH_AVX2, Build with AVX2
   * WITH_SSE2, Build with SSE2
   * WITH_SSSE3, Build with SSSE3
   * WITH_SSE4, Build with SSE4
   * WITH_PCLMULQDQ, Build with PCLMULQDQ
   * SSE2, Support the SSE2 instruction set, using "-msse2"

  -- The following REQUIRED packages have been found:

   * Threads

  -- The following features have been disabled:

   * ZLIB_ENABLE_TESTS, Build test binaries
   * ZLIB_DUAL_LINK, Dual link tests against system zlib
   * WITH_FUZZERS, Build test/fuzz
   * WITH_NATIVE_INSTRUCTIONS, Instruct the compiler to use the full instruction set on this host (gcc/clang -march=native)
   * WITH_MAINTAINER_WARNINGS, Build with project maintainer warnings
   * WITH_CODE_COVERAGE, Enable code coverage reporting
   * WITH_INFLATE_STRICT, Build with strict inflate distance checking
   * WITH_INFLATE_ALLOW_INVALID_DIST, Build with zero fill for inflate invalid distances
   * INSTALL_UTILS, Copy minigzip and minideflate during install
   * FORCE_TZCNT, Assume CPU is TZCNT capable

  -- Using ZSTD internal sources.
  -- Building for system processor x86
  -- Building for compiler ID Clang
  -- Disabling support for assembly sources in ZSTD
  -- Configuring done (15.6s)
  -- Generating done (0.0s)
  -- Build files have been written to: /home/milesg/Projects/blosc2-rs/target/wasm32-unknown-emscripten/debug/build/blosc2-sys-6e55cedb3f4350b4/out/build
  EMMAKE_wasm32-unknown-emscripten = None
  EMMAKE_wasm32_unknown_emscripten = None
  TARGET_EMMAKE = None
  EMMAKE = None
  running: cd "/home/milesg/Projects/blosc2-rs/target/wasm32-unknown-emscripten/debug/build/blosc2-sys-6e55cedb3f4350b4/out/build" && MAKEFLAGS="-j --jobserve
  [  1%] Building C object internal-complibs/zlib-ng-2.0.7/CMakeFiles/zlib.dir/adler32.c.o
  [  2%] Building C object internal-complibs/zlib-ng-2.0.7/CMakeFiles/zlib.dir/chunkset.c.o
  [  3%] Building C object internal-complibs/zlib-ng-2.0.7/CMakeFiles/zlib.dir/compare258.c.o
  [  4%] Building C object internal-complibs/zlib-ng-2.0.7/CMakeFiles/zlib.dir/compress.c.o
  [  5%] Building C object internal-complibs/zlib-ng-2.0.7/CMakeFiles/zlib.dir/crc32.c.o
  [  6%] Building C object internal-complibs/zlib-ng-2.0.7/CMakeFiles/zlib.dir/crc32_comb.c.o
  [  7%] Building C object internal-complibs/zlib-ng-2.0.7/CMakeFiles/zlib.dir/deflate.c.o
  [  8%] Building C object internal-complibs/zlib-ng-2.0.7/CMakeFiles/zlib.dir/deflate_fast.c.o
  [  9%] Building C object internal-complibs/zlib-ng-2.0.7/CMakeFiles/zlib.dir/deflate_medium.c.o
  [ 10%] Building C object internal-complibs/zlib-ng-2.0.7/CMakeFiles/zlib.dir/deflate_quick.c.o
  [ 11%] Building C object internal-complibs/zlib-ng-2.0.7/CMakeFiles/zlib.dir/deflate_slow.c.o
  [ 12%] Building C object internal-complibs/zlib-ng-2.0.7/CMakeFiles/zlib.dir/functable.c.o
  [ 13%] Building C object internal-complibs/zlib-ng-2.0.7/CMakeFiles/zlib.dir/infback.c.o
  [ 14%] Building C object internal-complibs/zlib-ng-2.0.7/CMakeFiles/zlib.dir/inffast.c.o
  [ 15%] Building C object internal-complibs/zlib-ng-2.0.7/CMakeFiles/zlib.dir/inflate.c.o
  [ 16%] Building C object internal-complibs/zlib-ng-2.0.7/CMakeFiles/zlib.dir/inftrees.c.o
  [ 17%] Building C object internal-complibs/zlib-ng-2.0.7/CMakeFiles/zlib.dir/insert_string.c.o
  [ 18%] Building C object internal-complibs/zlib-ng-2.0.7/CMakeFiles/zlib.dir/trees.c.o
  [ 19%] Building C object internal-complibs/zlib-ng-2.0.7/CMakeFiles/zlib.dir/uncompr.c.o
  [ 20%] Building C object internal-complibs/zlib-ng-2.0.7/CMakeFiles/zlib.dir/zutil.c.o
  [ 21%] Building C object internal-complibs/zlib-ng-2.0.7/CMakeFiles/zlib.dir/arch/x86/x86.c.o

  --- stderr
  configure: cmake /home/milesg/Projects/blosc2-rs/target/wasm32-unknown-emscripten/debug/build/blosc2-sys-6e55cedb3f4350b4/out/c-blosc2 -DCMAKE_POSITION_INDE
  Configuring for Blosc version: 2.15.1
  CMake Warning at CMakeLists.txt:357 (message):
    Unrecognized system processor x86.  Cannot determine which hardware
    features (Clang 20.0.0) supports, so hardware-accelerated implementations
    will not be available.


  CMake Warning (dev) at blosc/CMakeLists.txt:16 (add_library):
    ADD_LIBRARY called with SHARED option but the target platform does not
    support dynamic linking.  Building a STATIC library instead.  This may lead
    to problems.
  This warning is for project developers.  Use -Wno-dev to suppress it.

  CMake Warning:
    Manually-specified variables were not used by the project:

      CMAKE_CXX_FLAGS


  make: cmake --build . --target install --config Debug
  gmake: warning: jobserver unavailable: using -j1.  Add '+' to parent make rule.
  In file included from /home/milesg/Projects/blosc2-rs/target/wasm32-unknown-emscripten/debug/build/blosc2-sys-6e55cedb3f4350b4/out/c-blosc2/internal-complib
  /home/milesg/Projects/emsdk/upstream/lib/clang/20/include/cpuid.h:14:2: error: this header is for x86 only
     14 | #error this header is for x86 only
        |  ^
  /home/milesg/Projects/emsdk/upstream/lib/clang/20/include/cpuid.h:309:5: error: invalid output constraint '=a' in asm
    309 |     __cpuid(__leaf, __eax, __ebx, __ecx, __edx);
        |     ^
  /home/milesg/Projects/emsdk/upstream/lib/clang/20/include/cpuid.h:273:11: note: expanded from macro '__cpuid'
    273 |         : "=a"(__eax), "=r" (__ebx), "=c"(__ecx), "=d"(__edx) \
        |           ^
  /home/milesg/Projects/emsdk/upstream/lib/clang/20/include/cpuid.h:324:5: error: invalid output constraint '=a' in asm
    324 |     __cpuid(__leaf, *__eax, *__ebx, *__ecx, *__edx);
        |     ^
  /home/milesg/Projects/emsdk/upstream/lib/clang/20/include/cpuid.h:273:11: note: expanded from macro '__cpuid'
    273 |         : "=a"(__eax), "=r" (__ebx), "=c"(__ecx), "=d"(__edx) \
        |           ^
  /home/milesg/Projects/emsdk/upstream/lib/clang/20/include/cpuid.h:338:5: error: invalid output constraint '=a' in asm
    338 |     __cpuid_count(__leaf, __subleaf, *__eax, *__ebx, *__ecx, *__edx);
        |     ^
  /home/milesg/Projects/emsdk/upstream/lib/clang/20/include/cpuid.h:280:11: note: expanded from macro '__cpuid_count'
    280 |         : "=a"(__eax), "=r" (__ebx), "=c"(__ecx), "=d"(__edx) \
        |           ^
  /home/milesg/Projects/emsdk/upstream/lib/clang/20/include/cpuid.h:346:3: error: invalid output constraint '=a' in asm
    346 |   __cpuid_count(__leaf, __subleaf, __cpu_info[0], __cpu_info[1], __cpu_info[2],
        |   ^
  /home/milesg/Projects/emsdk/upstream/lib/clang/20/include/cpuid.h:280:11: note: expanded from macro '__cpuid_count'
    280 |         : "=a"(__eax), "=r" (__ebx), "=c"(__ecx), "=d"(__edx) \
        |           ^
  /home/milesg/Projects/blosc2-rs/target/wasm32-unknown-emscripten/debug/build/blosc2-sys-6e55cedb3f4350b4/out/c-blosc2/internal-complibs/zlib-ng-2.0.7/arch/x86/x86.c:37:5: error: invalid output constraint '=a' in asm
     37 |     __cpuid(info, *eax, *ebx, *ecx, *edx);
        |     ^
  /home/milesg/Projects/emsdk/upstream/lib/clang/20/include/cpuid.h:273:11: note: expanded from macro '__cpuid'
    273 |         : "=a"(__eax), "=r" (__ebx), "=c"(__ecx), "=d"(__edx) \
        |           ^
  /home/milesg/Projects/blosc2-rs/target/wasm32-unknown-emscripten/debug/build/blosc2-sys-6e55cedb3f4350b4/out/c-blosc2/internal-complibs/zlib-ng-2.0.7/arch/x86/x86.c:51:5: error: invalid output constraint '=a' in asm
     51 |     __cpuid_count(info, subinfo, *eax, *ebx, *ecx, *edx);
        |     ^
  /home/milesg/Projects/emsdk/upstream/lib/clang/20/include/cpuid.h:280:11: note: expanded from macro '__cpuid_count'
    280 |         : "=a"(__eax), "=r" (__ebx), "=c"(__ecx), "=d"(__edx) \
        |           ^
  7 errors generated.
  gmake[2]: *** [internal-complibs/zlib-ng-2.0.7/CMakeFiles/zlib.dir/build.make:377: internal-complibs/zlib-ng-2.0.7/CMakeFiles/zlib.dir/arch/x86/x86.c.o] Error 1
  gmake[1]: *** [CMakeFiles/Makefile2:143: internal-complibs/zlib-ng-2.0.7/CMakeFiles/zlib.dir/all] Error 2
  gmake: *** [Makefile:156: all] Error 2
  emmake: error: 'cmake --build . --target install --config Debug' failed (returned 2)
  thread 'main' panicked at /home/milesg/.cargo/registry/src/index.crates.io-6f17d22bba15001f/cmake-0.1.51/src/lib.rs:1100:5:

  command did not execute successfully, got: exit status: 1

  build script failed, must exit now
  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

Edit: I see now, if I set WITH_OPTIM FALSE then things are happy, but maybe it's not the 'correct' solution.
Avoiding specifically these two lines:

    list(APPEND ZLIB_ARCH_HDRS ${ARCHDIR}/x86.h)
    list(APPEND ZLIB_ARCH_SRCS ${ARCHDIR}/x86.c)

zlib-ng/CMakeLists.txt

Line 798 in a583e21

list(APPEND ZLIB_ARCH_HDRS ${ARCHDIR}/x86.h)

@milesgranger milesgranger mentioned this issue Sep 17, 2024
Merged
@kleisauke
Copy link

@milesgranger -DWITH_RUNTIME_CPU_DETECTION=FALSE ought to fix that, but it's only available since c694bcd (v2.2.0). I can only recommend -DWITH_OPTIM=FALSE if you cannot target Wasm SIMD.

@nmoinvaz
Copy link
Member

nmoinvaz commented Oct 4, 2024

@milesgranger were you able to get it working?

@milesgranger
Copy link

Yes, c-blosc2 is on v2.0.7 so -DWITH_OPTIM=FALSE works. I have the rust bindings now on CI w/ emscripten by setting this during the build stage. after adding that option in c-blosc2.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@nmoinvaz @kleisauke @milesgranger @LugPaj