The "too small to fail" memory-allocation rule

Posted Dec 25, 2014 20:23 UTC (Thu) by quotemstr (subscriber, #45331)
In reply to: The "too small to fail" memory-allocation rule by fw
Parent article: The "too small to fail" memory-allocation rule

The man page says this about MAP_NORESERVE:

Do not reserve swap space for this mapping. When swap space is reserved, one has the guarantee that it is possible to modify the mapping. When swap space is not reserved one might get SIGSEGV upon a write if no physical memory is available. See also the discussion of the file /proc/sys/vm/overcommit_memory in proc(5). In kernels before 2.6, this flag had effect only for private writable mappings.

That looks a lot like a a flag for mapping that doesn't reserve commit charge. There's no mention of the flag not actually working; there's no reference to better techniques. MAP_NORESERVE is just an attractive nuisance.

There's just tremendous confusion over exactly Linux does with commit charge even among people well-versed in memory management fundamentals. There's no clarity in the API either. MAP_NORESERVE is dangerous because it's the only publicly-documented knob for managing commit charge and it's a broken knob nobody is interested in fixing.

(And no, SIGSEGV is not something that will discourage use. Those who know what they're doing can write code that recovers perfectly safely from SIGSEGV.)

The "too small to fail" memory-allocation rule

Posted Apr 18, 2019 6:36 UTC (Thu) by thestinger (guest, #91827) [Link]

In case anyone comes across this old comment, the official documentation is at https://www.kernel.org/doc/Documentation/vm/overcommit-ac... and is accurate. It covers the rules properly.

The linux-man-pages project is not the official kernel documentation and has often been inaccurate about important things for many years. MAP_NORESERVE doesn't work as it describes at all.

It has no impact in the full overcommit mode or the memory accounting mode, which makes sense. It only has an impact in the heuristic overcommit mode, where it disables the heuristic for causing immediate failure.

Note how a read-only (or PROT_NONE) mapping with not data has no accounting cost:

For an anonymous or /dev/zero map
SHARED - size of mapping
PRIVATE READ-only - 0 cost (but of little use)
PRIVATE WRITABLE - size of mapping per instance