DJS_USE_CUSTOM_ALLOCATOR doesn't seem to be actually useful / maintained, since any custom implementation still need to copy lots of code from Utility.h

Indeed. I think JS_USE_CUSTOM_ALLOCATOR should guard the code in this block: https://searchfox.org/firefox-main/rev/af3c77e22f88ca06e1b4ac822c1d056f263a64fe/js/public/Utility.h#355 not the thread code too.

Some code does make assumption that moz_arena_ allocated pointers can be freed / re-alloced by unprefixed free(), this lead to a patch to StringBuffer.h

Sounds like we could upstream some fixes.

This design ensures that the allocator functions can be inlined

I understand the need for this (we cannot inline without LTO - maybe we should investigate thinlto support in mozjs artifacts), but this design is complex (hate environment variables), I am wondering how we could make use of this in servo (for linux where we use jemalloc). One thing that I loved about webeef solution is that it's easy to pass overrides, and we could possibly do this without changing artifacts (always use prefixed alloc, but provide "own" in mozjs as default). Does inlining really makes difference in practice?

Does inlining really makes difference in practice?

Yes - the issue is that the happy path of allocators is highly optimized to be just a few instructions. Adding a function call is measurable overhead (since it saves and restores registers on the stack). How measurable depends of course on the frequency of allocations, but it really adds up. (This is based on discussions with a colleague who is responsible for our in-house custom allocator - I didn't measure myself).

I am wondering how we could make use of this in servo (for linux where we use jemalloc)

I have a half-baked PR for servo, which uses this feature. The main issue is that it adds another thing to mach (due to the environment variable). Otherwise, nothing much is really required on the servo side.

One thing that I loved about webeef solution is that it's easy to pass overrides, and we could possibly do this without changing artifacts

This is mainly a tradeoff between baking in the function calls statically (via static inline) or having real function declarations that are only resolved by the linker (which then requires LTO to inline the function calls again).
However, this approach does allow combining the two, by (the embedder / servo) providing a header file to SM which provides static inline function definitions for the allocator APIs, which call a servo_embedder_<malloc> function, which is forward declared and can be resolved at link-time.

Some code does make assumption that moz_arena_ allocated pointers can be freed / re-alloced by unprefixed free(), this lead to a patch to StringBuffer.h

Sounds like we could upstream some fixes.

I'm not sure if they would appreciate it. The intention seems to be that if the arena allocator is OOM, then they fall back to malloc, and thus need to always free with free. That's also why in practice the SM code seems to rely on the fact that jemalloc will be used unprefixed in many places.

If we published servo-allocator, then mozjs-sys could also depend on it and access information via DEP_ syntax (it's a bit hacky since it requires adding a links key, but I believe we already abuse this in a different servo crate).
An embedder could choose a different allocator via patch. Not quite as simple as via environment variables, but perhaps an option if we want more robustness, and offer a couple built-in solutions like jemalloc and mimalloc?

I haven't thought this compleltely through, so perhaps something prevents that solution. It would make the publish pipeline for alloc related changes quite ugly though.