The "too small to fail" memory-allocation rule

The allocation is question is __GFP_NOWAIT which is equivalent to GFP_NOIO.
Such an allocation must not risk waiting for FS or IO activity. Yet waiting for the OOM killer can clearly do that. This is a violation of the interface whether you accept the "too small to fail" rule or not.

Invoking the OOM killer is reasonable, waiting a little while is reasonable, but waiting for the killed processes to flush data or close files is not..... Hmmm. The out_of_memory() function chooses a processes, kills it, then waits one second ("schedule_timeout_killable(1);"), so it seems to be behaving as I would expect.

Reading more of the email threads it seems that TIF_MEMDIE is an important part of the problem. This is set on a thread when it has been chosen to die. As long as any thread has this set, "select_bad_process()" will not select anything else so nothing else gets killed.

So the "real" problem seems to be that if a process with TIF_MEMDIE set enters filesystem (or IO) code and blocks, a GFP_NOIO allocation can be made to wait for that filesystem code to complete - which could deadlock.

Looking at the original patch which started this: http://marc.info/?l=linux-mm&m=141839249819519&w=2 the problem seems to involve a process with TIF_MEMDIE set, which has already released its memory, but is now stuck in the filesystem, probably closing some files.

As it has TIF_MEMDIE set, nothing else will be killed. So no memory will become available, so it will stay stuck in the filesystem...

This deadlock could possibly be avoided by having oom_scan_process_thread() not abort the scan if TIF_MEMDIE is set on a process which has already dropped its memory. Then another process can be killed even if the first one blocked

But that analysis was very hasty and probably misses something important. "Here be dragons" is how I would mark this code on a map of Linux.