and not “why”

The “why” was largely political, IMO: The spawn() family of functions were introduced into POSIX so that Windows NT could get a POSIX certification and win government contracts, since it had spawn() and not fork().

Tannenbaum’s Modern Operating Systems was written around this time, and you might find its discussion of process-spawning APIs interesting: He doesn’t mention performance, and indeed Linux’s fork+exec was faster than NT’s CreateProcess so I find it incredibly unlikely NT’s omission for fork() was for performance, but more likely to simplify other parts of the NT design.

I guess this kind of thing is what I’m looking for

The suggestion to run a subprocess that calls tcsetpgrp before exec isn’t a bad one, and maybe obviates some of the performance benefits you get from posix_spawn, but it might not be so bad because that subprocess can be a real simple tiny static binary that does what it needs to and calls exec(). One day maybe we won’t have to worry about this.

Another option is to just wait+WIFSTOPPED and then kill+SIGCONT it if it’s supposed to be in the foreground.

XNU has a lot of extensions to POSIX spawn that make it almost usable. Unfortunately, they’re not implemented anywhere else. The biggest problem with the API is that they constrained it to permit userspace implementations. As such, it is strictly less expressive than vfork + execve. That said, vfork isn’t actually that bad an API. It would be even better if the execve simply returned and vfork didn’t return twice. Then the sequence would be simply vfork, setup, execve, cleanup.

With a language like C++ that supports RAII, you can avoid the footguns of vfork by doing

pid_t pid = vfork();
if (pid == 0)
{
    {
         // Set up the child
     }
     execve(…);
     pid = -1;
}

This ensures that anything that you created in between the setup is cleaned up. I generally use a std::vector for the execve arguments. This must be declare in the enclosing scope, so it’s cleaned up in the parent. It’s pretty easy to wrap this in a function that takes a lambda and passes it a reference to the argv and envp vectors, and executes it before the execve . This ensures that you get the memory management right. As a caller, you just pass a lambda that does any file descriptor opening and so on. The wrapper that I use also takes a vector of file descriptors to inherit, so you can open some files before spawning the child, the do arbitrary additional setup in the child context (including things like entering capability mode or attaching to a jail).

Actually that tweet thread by the author of fish is very good.

This is exactly what Melvin just added, so looks like we can’t use it ? On what platforms?

8/ What’s missing? One hole is the syscall you’ve never heard of: tcsetpgrp(), which hands-off tty ownership. The “correct” usage with fork is a benign race, where both the parent (tty donator) and child (tty inheritor) request that the child own the tty.

9/ There is no “correct” tcsetpgrp usage with posix_spawn: no way to coax the child into claiming the tty. This means, when job control is on, you may start a process which immediately stops (SIGTTIN) or otherwise. Here’s ksh getting busted: https://mail-archive.com/ast-developers

posix_spawn is documented in POSIX: https://pubs.opengroup.org/onlinepubs/9699919799/ , specially here: https://pubs.opengroup.org/onlinepubs/9699919799/functions/posix_spawn.html . This manpage contains big “RATIONALE” section. And “SEE ALSO” section with posix_spawnattr_* functions

Linux does posix_spawn in user space, on top of fork - but sometimes vfork

Glibc and musl add more and more optimizations over time, allowing posix_spawn to use vfork (as opposed to fork) in more and more cases. It is quite possible that recent versions of glibc and musl call vfork in all cases.

AFAIK this glibc bug report https://sourceware.org/bugzilla/show_bug.cgi?id=10354 is resolved using patchset, which makes glibc always use vfork/CLONE_VFORK.

Using vfork is not simple. This article explains how hard it is: https://ewontfix.com/7/

Here is my util, I call it “asjail” for “Askar Safin’s jail”:

https://paste.gg/p/anonymous/4d26975181eb4223b10800911255c951

It is public domain. I’m sorry for Russian comments.

Compile with gcc -o asjail asjail.c. My program is x86 Linux specific. Run like so: asjail -p bash.

Program has options -imnpuU, which correspond to unshare(1) options (see its manpage). (Also, actual unshare has option -r, I have no such cool option.)

My program usually requires root privileges, but you can specify -U flag, which creates user namespace. So, you can run asjail -pU bash as normal user, this will create new user namespace and then create PID namespace inside it. (Again: unshare -pr bash is even better.)

But user namespace requires that they should be enabled in kernel. In some distros they are enabled by default, in others - not.

I wrote this util nearly 10 years ago. Back then I wanted some lightweight container solution. I was not aware of unshare(1) util. unshare(1) fully subsumes my util. (Don’t confuse with unshare(2) syscall.) Also, unshare(1) is low-level util, it’s lower then bubblewrap, runc, etc.

I don’t remember some details in this code, for example I don’t remember why I need this signal mask manipulations.

Today I use docker and I’m happy with it. Not only docker provide isolation, it is also allows you to write Dockerfiles. And partial results in dockerfiles are cached, so you can edit some line in dockerfile, and docker will rebuild exactly what is needed and no more. Dockerfiles are perfect for bug reports, i. e. you can simply send dockerfile instead of “steps to reproduce” section. The only problem with docker is inability to run systemd inside it. I have read that this is solved by podman, but I didn’t test it. Also, dockerfiles are not quite reproducible, because they are often rely on downloading something from internet. I have read that proper solution is Nix, but I didn’t test it

Additional comments on asjail (and other topics).

You need to add -- to make sure options are processed as part of target command, not asjail itself, i. e. asjail -p -- bash -c 'echo ok'.

asjail was written by careful reading of clone(2) manual page.

asjail is not complete solution. It doesn’t do chrooting, mounting needed directories (/proc, /dev, etc). So back then 10 years ago I wrote bash script called asjail-max-1, which does these additional steps. asjail-max-1 was written by careful reading of http://www.freedesktop.org/wiki/Software/systemd/ContainerInterface and so asjail-max-1 (together with asjail) can run systemd in container! asjail-max-1 is screen sized bash script.

But, unfortunately, to make all this work, you also need C program, which emulates terminal using forkpty(3). So I wrote such a program and I called it pty. It is some screens sized C program. Together asjail, asjail-max-1 and pty gives you complete small solution for running containers. In something like 5 screens of code.

I can post all this code.

But all this is not needed, because all this is subsumed by existing tools. asjail is subsumed by unshare. And asjail-max-1+asjail+pty is subsumed by systemd-nspawn.

Today I don’t use any of these my tools. When I need to run container I use docker. If I need to run existing file tree I use systemd-nspawn

Also, all these tools I discussed so far are designed for maximal isolation. I. e. to prevent container from accessing host resources. But sometimes you have opposite task, i. e. you want to run some program and give it access to host resources, for example to X server connection, to sound card etc. So I have another script, which is simple wrapper for chroot, which does exactly this: https://paste.gg/p/anonymous/84c3685e200347299cac0dbd23d31bf3