The "too small to fail" memory-allocation rule

Indeed it's bizarre that an automatic attempt to avert malloc failure would potentially re-enter its caller and therefore guarantee a lock-ordering violation. First, the allocator is at all called with a lock held, implying that allocation length follows from state which the lock is used to guard; which in turn implies overly eager design.

Second, that the allocator enters the OOM killer on its own volition, changing malloc's locking behaviour from "only ever the heap lock" to "everything an asynchronous OOM killer might end up with". That might well cover all of the kernel, when the caller is expecting malloc to be atomic by itself.

What's surprising isn't so much the reams of untested malloc-failure handling code, but that this bug comes up as late as 2014.