------------------------

HAProxy Management Guide

------------------------

version 3.1

This document describes how to start, stop, manage, and troubleshoot HAProxy,

as well as some known limitations and traps to avoid. It does not describe how

to configure it (for this please read configuration.txt).

Note to documentation contributors :

This document is formatted with 80 columns per line, with even number of

spaces for indentation and without tabs. Please follow these rules strictly

so that it remains easily printable everywhere. If you add sections, please

update the summary below for easier searching.

Summary

-------

1. Prerequisites

2. Quick reminder about HAProxy's architecture

3. Starting HAProxy

4. Stopping and restarting HAProxy

5. File-descriptor limitations

6. Memory management

7. CPU usage

8. Logging

9. Statistics and monitoring

9.1. CSV format

9.2. Typed output format

9.3. Unix Socket commands

9.4. Master CLI

9.4.1. Master CLI commands

9.5. Stats-file

10. Tricks for easier configuration management

11. Well-known traps to avoid

12. Debugging and performance issues

13. Security considerations

13.1. Linux capabilities support

1. Prerequisites

----------------

In this document it is assumed that the reader has sufficient administration

skills on a UNIX-like operating system, uses the shell on a daily basis and is

familiar with troubleshooting utilities such as strace and tcpdump.

2. Quick reminder about HAProxy's architecture

----------------------------------------------

HAProxy is a multi-threaded, event-driven, non-blocking daemon. This means it

uses event multiplexing to schedule all of its activities instead of relying on

the system to schedule between multiple activities. Most of the time it runs as

a single process, so the output of "ps aux" on a system will report only one

"haproxy" process, unless a soft reload is in progress and an older process is

finishing its job in parallel to the new one. It is thus always easy to trace

its activity using the strace utility. In order to scale with the number of

available processors, by default haproxy will start one worker thread per

processor it is allowed to run on. Unless explicitly configured differently,

the incoming traffic is spread over all these threads, all running the same

event loop. A great care is taken to limit inter-thread dependencies to the

strict minimum, so as to try to achieve near-linear scalability. This has some

impacts such as the fact that a given connection is served by a single thread.

Thus in order to use all available processing capacity, it is needed to have at

least as many connections as there are threads, which is almost always granted.

HAProxy is designed to isolate itself into a chroot jail during startup, where

it cannot perform any file-system access at all. This is also true for the

libraries it depends on (eg: libc, libssl, etc). The immediate effect is that

a running process will not be able to reload a configuration file to apply

changes, instead a new process will be started using the updated configuration

file. Some other less obvious effects are that some timezone files or resolver

files the libc might attempt to access at run time will not be found, though

this should generally not happen as they're not needed after startup. A nice

consequence of this principle is that the HAProxy process is totally stateless,

and no cleanup is needed after it's killed, so any killing method that works

will do the right thing.

HAProxy doesn't write log files, but it relies on the standard syslog protocol

to send logs to a remote server (which is often located on the same system).

HAProxy uses its internal clock to enforce timeouts, that is derived from the

system's time but where unexpected drift is corrected. This is done by limiting

the time spent waiting in poll() for an event, and measuring the time it really

took. In practice it never waits more than one second. This explains why, when

running strace over a completely idle process, periodic calls to poll() (or any

of its variants) surrounded by two gettimeofday() calls are noticed. They are

normal, completely harmless and so cheap that the load they imply is totally

undetectable at the system scale, so there's nothing abnormal there. Example :

16:35:40.002320 gettimeofday({1442759740, 2605}, NULL) = 0

16:35:40.002942 epoll_wait(0, {}, 200, 1000) = 0

16:35:41.007542 gettimeofday({1442759741, 7641}, NULL) = 0

16:35:41.007998 gettimeofday({1442759741, 8114}, NULL) = 0

16:35:41.008391 epoll_wait(0, {}, 200, 1000) = 0

16:35:42.011313 gettimeofday({1442759742, 11411}, NULL) = 0

HAProxy is a TCP proxy, not a router. It deals with established connections that

have been validated by the kernel, and not with packets of any form nor with

sockets in other states (eg: no SYN_RECV nor TIME_WAIT), though their existence

may prevent it from binding a port. It relies on the system to accept incoming

connections and to initiate outgoing connections. An immediate effect of this is

that there is no relation between packets observed on the two sides of a

forwarded connection, which can be of different size, numbers and even family.

Since a connection may only be accepted from a socket in LISTEN state, all the

sockets it is listening to are necessarily visible using the "netstat" utility

to show listening sockets. Example :

# netstat -ltnp

Active Internet connections (only servers)

Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name

tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1629/sshd

tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 2847/haproxy

tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN 2847/haproxy

3. Starting HAProxy

-------------------

HAProxy is started by invoking the "haproxy" program with a number of arguments

passed on the command line. The actual syntax is :

$ haproxy [<options>]*

where [<options>]* is any number of options. An option always starts with '-'

followed by one of more letters, and possibly followed by one or multiple extra

arguments. Without any option, HAProxy displays the help page with a reminder

about supported options. Available options may vary slightly based on the

operating system. A fair number of these options overlap with an equivalent one

in the "global" section. In this case, the command line always has precedence

over the configuration file, so that the command line can be used to quickly

enforce some settings without touching the configuration files. The current

list of options is :

-- <cfgfile>* : all the arguments following "--" are paths to configuration

file/directory to be loaded and processed in the declaration order. It is

mostly useful when relying on the shell to load many files that are

numerically ordered. See also "-f". The difference between "--" and "-f" is

that one "-f" must be placed before each file name, while a single "--" is

needed before all file names. Both options can be used together, the

command line ordering still applies. When more than one file is specified,

each file must start on a section boundary, so the first keyword of each

file must be one of "global", "defaults", "peers", "listen", "frontend",

"backend", and so on. A file cannot contain just a server list for example.

-f <cfgfile|cfgdir> : adds <cfgfile> to the list of configuration files to be

loaded. If <cfgdir> is a directory, all the files (and only files) it

contains are added in lexical order (using LC_COLLATE=C) to the list of

configuration files to be loaded ; only files with ".cfg" extension are

added, only non hidden files (not prefixed with ".") are added.

Configuration files are loaded and processed in their declaration order.

This option may be specified multiple times to load multiple files. See

also "--". The difference between "--" and "-f" is that one "-f" must be

placed before each file name, while a single "--" is needed before all file

names. Both options can be used together, the command line ordering still

applies. When more than one file is specified, each file must start on a

section boundary, so the first keyword of each file must be one of

"global", "defaults", "peers", "listen", "frontend", "backend", and so on.

A file cannot contain just a server list for example.

-C <dir> : changes to directory <dir> before loading configuration

files. This is useful when using relative paths. Warning when using

wildcards after "--" which are in fact replaced by the shell before

starting haproxy.

-D : start as a daemon. The process detaches from the current terminal after

forking, and errors are not reported anymore in the terminal. It is

equivalent to the "daemon" keyword in the "global" section of the

configuration. It is recommended to always force it in any init script so

that a faulty configuration doesn't prevent the system from booting.

-L <name> : change the local peer name to <name>, which defaults to the local

hostname. This is used only with peers replication. You can use the

variable $HAPROXY_LOCALPEER in the configuration file to reference the

peer name.

-N <limit> : sets the default per-proxy maxconn to <limit> instead of the

builtin default value (usually 2000). Only useful for debugging.

-V : enable verbose mode (disables quiet mode). Reverts the effect of "-q" or

"quiet".

-W : master-worker mode. It is equivalent to the "master-worker" keyword in

the "global" section of the configuration. This mode will launch a "master"

which will monitor the "workers". Using this mode, you can reload HAProxy

directly by sending a SIGUSR2 signal to the master. The master-worker mode

is compatible either with the foreground or daemon mode. It is

recommended to use this mode with multiprocess and systemd.

-Ws : master-worker mode with support of `notify` type of systemd service.

This option is only available when HAProxy was built with `USE_SYSTEMD`

build option enabled.

-c : only performs a check of the configuration files and exits before trying

to bind. The exit status is zero if everything is OK, or non-zero if an

error is encountered. Presence of warnings will be reported if any.

-cc : evaluates a condition as used within a conditional block of the

configuration. The exit status is zero if the condition is true, 1 if the

condition is false or 2 if an error is encountered.

-d : enable debug mode. This disables daemon mode, forces the process to stay

in foreground and to show incoming and outgoing events. It must never be

used in an init script.

-dC[key] : dump the configuration file. It is performed after the lines are

tokenized, so comments are stripped and indenting is forced. If a non-zero

key is specified, lines are truncated before sensitive/confidential fields,

and identifiers and addresses are emitted hashed with this key using the

same algorithm as the one used by the anonymized mode on the CLI. This

means that the output may safely be shared with a developer who needs it

to figure what's happening in a dump that was anonymized using the same

key. Please also see the CLI's "set anon" command.

-dD : enable diagnostic mode. This mode will output extra warnings about

suspicious configuration statements. This will never prevent startup even in

"zero-warning" mode nor change the exit status code.

-dF : disable data fast-forward. It is a mechanism to optimize the data

forwarding by passing data directly from a side to the other one without

waking the stream up. Thanks to this directive, it is possible to disable

this optimization. Note it also disable any kernel tcp splicing. This

command is not meant for regular use, it will generally only be suggested by

developers along complex debugging sessions.

-dG : disable use of getaddrinfo() to resolve host names into addresses. It

can be used when suspecting that getaddrinfo() doesn't work as expected.

This option was made available because many bogus implementations of

getaddrinfo() exist on various systems and cause anomalies that are

difficult to troubleshoot.

-dI : enable the insecure fork. This is the equivalent of the

"insecure-fork-wanted" in the global section. It can be useful when running

all the reg-tests with ASAN which need to fork addr2line to resolve the

addresses.

-dK<class[,class]*> : dumps the list of registered keywords in each class.

The list of classes is available with "-dKhelp". All classes may be dumped

using "-dKall", otherwise a selection of those shown in the help can be

specified as a comma-delimited list. The output format will vary depending

on what class of keywords is being dumped (e.g. "cfg" will show the known

configuration keywords in a format resembling the config file format while

"smp" will show sample fetch functions prefixed with a compatibility matrix

with each rule set). These may rarely be used as-is by humans but can be of

great help for external tools that try to detect the appearance of new

keywords at certain places to automatically update some documentation,

syntax highlighting files, configuration parsers, API etc. The output

format may evolve a bit over time so it is really recommended to use this

output mostly to detect differences with previous archives. Note that not

all keywords are listed because many keywords have existed long before the

different keyword registration subsystems were created, and they do not

appear there. However since new keywords are only added via the modern

mechanisms, it's reasonably safe to assume that this output may be used to

detect language additions with a good accuracy. The keywords are only

dumped after the configuration is fully parsed, so that even dynamically

created keywords can be dumped. A good way to dump and exit is to run a

silent config check on an existing configuration:

./haproxy -dKall -q -c -f foo.cfg

If no configuration file is available, using "-f /dev/null" will work as

well to dump all default keywords, but then the return status will not be

zero since there will be no listener, and will have to be ignored.

-dL : dumps the list of dynamic shared libraries that are loaded at the end

of the config processing. This will generally also include deep dependencies

such as anything loaded from Lua code for example, as well as the executable

itself. The list is printed in a format that ought to be easy enough to

sanitize to directly produce a tarball of all dependencies. Since it doesn't

stop the program's startup, it is recommended to only use it in combination

with "-c" and "-q" where only the list of loaded objects will be displayed

(or nothing in case of error). In addition, keep in mind that when providing

such a package to help with a core file analysis, most libraries are in fact

symbolic links that need to be dereferenced when creating the archive:

./haproxy -W -q -c -dL -f foo.cfg | tar -T - -hzcf archive.tgz

When started in verbose mode (-V) the shared libraries' address ranges are

also enumerated, unless the quiet mode is in use (-q).

-dM[<byte>[,]][help|options,...] : forces memory poisoning, and/or changes

memory other debugging options. Memory poisonning means that each and every

memory region allocated with malloc() or pool_alloc() will be filled with

<byte> before being passed to the caller. When <byte> is not specified, it

defaults to 0x50 ('P'). While this slightly slows down operations, it is

useful to reliably trigger issues resulting from missing initializations in

the code that cause random crashes. Note that -dM0 has the effect of

turning any malloc() into a calloc(). In any case if a bug appears or

disappears when using this option it means there is a bug in haproxy, so

please report it. A number of other options are available either alone or

after a comma following the byte. The special option "help" will list the

currently supported options and their current value. Each debugging option

may be forced on or off. The most optimal options are usually chosen at

build time based on the operating system and do not need to be adjusted,

unless suggested by a developer. Supported debugging options include

(set/clear):

- fail / no-fail:

This enables randomly failing memory allocations, in conjunction with

the global "tune.fail-alloc" setting. This is used to detect missing

error checks in the code. Setting the option presets the ratio to 1%

failure rate.

- no-merge / merge:

By default, pools of very similar sizes are merged, resulting in more

efficiency, but this complicates the analysis of certain memory dumps.

This option allows to disable this mechanism, and may slightly increase

the memory usage.

- cold-first / hot-first:

In order to optimize the CPU cache hit ratio, by default the most

recently released objects ("hot") are recycled for new allocations.

But doing so also complicates analysis of memory dumps and may hide

use-after-free bugs. This option allows to instead pick the coldest

objects first, which may result in a slight increase of CPU usage.

- integrity / no-integrity:

When this option is enabled, memory integrity checks are enabled on

the allocated area to verify that it hasn't been modified since it was

last released. This works best with "no-merge", "cold-first" and "tag".

Enabling this option will slightly increase the CPU usage.

- no-global / global:

Depending on the operating system, a process-wide global memory cache

may be enabled if it is estimated that the standard allocator is too

slow or inefficient with threads. This option allows to forcefully

disable it or enable it. Disabling it may result in a CPU usage

increase with inefficient allocators. Enabling it may result in a

higher memory usage with efficient allocators.

- no-cache / cache:

Each thread uses a very fast local object cache for allocations, which

is always enabled by default. This option allows to disable it. Since

the global cache also passes via the local caches, this will

effectively result in disabling all caches and allocating directly from

the default allocator. This may result in a significant increase of CPU

usage, but may also result in small memory savings on tiny systems.

- caller / no-caller:

Enabling this option reserves some extra space in each allocated object

to store the address of the last caller that allocated or released it.

This helps developers go back in time when analysing memory dumps and

to guess how something unexpected happened.

- tag / no-tag:

Enabling this option reserves some extra space in each allocated object

to store a tag that allows to detect bugs such as double-free, freeing

an invalid object, and buffer overflows. It offers much stronger

reliability guarantees at the expense of 4 or 8 extra bytes per

allocation. It usually is the first step to detect memory corruption.

- poison / no-poison:

Enabling this option will fill allocated objects with a fixed pattern

that will make sure that some accidental values such as 0 will not be

present if a newly added field was mistakenly forgotten in an

initialization routine. Such bugs tend to rarely reproduce, especially

when pools are not merged. This is normally enabled by directly passing

the byte's value to -dM but using this option allows to disable/enable

use of a previously set value.

-dR : disable SO_REUSEPORT socket option on listening ports. It is equivalent

to the "global" section's "noreuseport" keyword. This may be applied in

multi-threading scenarios, when load distribution issues observed among the

haproxy threads (could be monitored with top).

-dS : disable use of the splice() system call. It is equivalent to the

"global" section's "nosplice" keyword. This may be used when splice() is

suspected to behave improperly or to cause performance issues, or when

using strace to see the forwarded data (which do not appear when using

splice()).

-dV : disable SSL verify on the server side. It is equivalent to having

"ssl-server-verify none" in the "global" section. This is useful when

trying to reproduce production issues out of the production

environment. Never use this in an init script as it degrades SSL security

to the servers.

-dW : if set, haproxy will refuse to start if any warning was emitted while

processing the configuration. This helps detect subtle mistakes and keep the

configuration clean and portable across versions. It is recommended to set

this option in service scripts when configurations are managed by humans,

but it is recommended not to use it with generated configurations, which

tend to emit more warnings. It may be combined with "-c" to cause warnings

in checked configurations to fail. This is equivalent to global option

"zero-warning".

-dZ : disable forwarding of data in "zero-copy" mode. It is equivalent to the

"global" section's "tune.disable-zero-copy-forwarding" keyword. This may be

helpful in case of issues with data loss or data integrity, or when using

strace to see the forwarded data, as it also disables any kernel tcp

splicing.

-db : disable background mode and multi-process mode. The process remains in

foreground. It is mainly used during development or during small tests, as

Ctrl-C is enough to stop the process. Never use it in an init script.

-de : disable the use of the "epoll" poller. It is equivalent to the "global"

section's keyword "noepoll". It is mostly useful when suspecting a bug

related to this poller. On systems supporting epoll, the fallback will

generally be the "poll" poller.

-dk : disable the use of the "kqueue" poller. It is equivalent to the

"global" section's keyword "nokqueue". It is mostly useful when suspecting

a bug related to this poller. On systems supporting kqueue, the fallback

will generally be the "poll" poller.

-dp : disable the use of the "poll" poller. It is equivalent to the "global"

section's keyword "nopoll". It is mostly useful when suspecting a bug

related to this poller. On systems supporting poll, the fallback will

generally be the "select" poller, which cannot be disabled and is limited

to 1024 file descriptors.

-dr : ignore server address resolution failures. It is very common when

validating a configuration out of production not to have access to the same

resolvers and to fail on server address resolution, making it difficult to

test a configuration. This option simply appends the "none" method to the

list of address resolution methods for all servers, ensuring that even if

the libc fails to resolve an address, the startup sequence is not

interrupted.

-dt [<trace_desc>,...] : activates traces on stderr. Without argument, this

enables all trace sources on error level. This can notably be useful to

detect protocol violations from clients or servers. An optional argument

can be used to specify a list of various trace configurations using ',' as

separator. Each element activates one or all trace sources. Additionally,

level and verbosity can be optionally specified on each element using ':'

as inner separator with trace name. When entering an invalid verbosity or

level name, the list of available keywords is presented. For example it can

be convenient to pass 'help' for each field to consult the list first.

-dv : disable the use of the "evports" poller. It is equivalent to the

"global" section's keyword "noevports". It is mostly useful when suspecting

a bug related to this poller. On systems supporting event ports (SunOS

derived from Solaris 10 and later), the fallback will generally be the

"poll" poller.

-m <limit> : limit allocatable memory, which is used to keep process's data,

to <limit> megabytes. This may cause some connection refusals or some

slowdowns depending on the amount of memory needed for normal operations.

This is mostly used to force haproxy process to work in a constrained

resource consumption scenario. It is important to note that the memory is

not shared between haproxy processes and a child process created via fork()

system call inherits its parent's resource limits. So, in a master-worker

mode this memory limit is separately applied to the master and its forked

worker process.

-n <limit> : limits the per-process connection limit to <limit>. This is

equivalent to the global section's keyword "maxconn". It has precedence

over this keyword. This may be used to quickly force lower limits to avoid

a service outage on systems where resource limits are too low.

-p <file> : write all processes' pids into <file> during startup. This is

equivalent to the "global" section's keyword "pidfile". The file is opened

before entering the chroot jail, and after doing the chdir() implied by

"-C". Each pid appears on its own line.

-q : set "quiet" mode. This disables the output messages. It can be used in

combination with "-c" to just check if a configuration file is valid or not.

-S <bind>[,bind_options...]: in master-worker mode, bind a master CLI, which

allows the access to every processes, running or leaving ones.

For security reasons, it is recommended to bind the master CLI to a local

UNIX socket. The bind options are the same as the keyword "bind" in

the configuration file with words separated by commas instead of spaces.

Note that this socket can't be used to retrieve the listening sockets from

an old process during a seamless reload.

-sf <pid>* : send the "finish" signal (SIGUSR1) to older processes after boot

completion to ask them to finish what they are doing and to leave. <pid>

is a list of pids to signal (one per argument). The list ends on any

option starting with a "-". It is not a problem if the list of pids is

empty, so that it can be built on the fly based on the result of a command

like "pidof" or "pgrep".

-st <pid>* : send the "terminate" signal (SIGTERM) to older processes after

boot completion to terminate them immediately without finishing what they

were doing. <pid> is a list of pids to signal (one per argument). The list

ends on any option starting with a "-". It is not a problem if the list

of pids is empty, so that it can be built on the fly based on the result of

a command like "pidof" or "pgrep".

-v : report the version and build date.

-vv : display the version, build options, libraries versions and usable

pollers. This output is systematically requested when filing a bug report.

-x <unix_socket> : connect to the specified socket and try to retrieve any

listening sockets from the old process, and use them instead of trying to

bind new ones. This is useful to avoid missing any new connection when

reloading the configuration on Linux.

Without master-worker mode, the capability must be enable on the stats

socket using "expose-fd listeners" in your configuration.

In master-worker mode, it does not need "expose-fd listeners", the master

will use automatically this option upon a reload with the "sockpair@"

syntax, which allows the master to connect directly to a worker without using

any stats socket declared in the configuration. If you want to disable this,

you can pass -x /dev/null.

A safe way to start HAProxy from an init file consists in forcing the daemon

mode, storing existing pids to a pid file and using this pid file to notify

older processes to finish before leaving :

haproxy -f /etc/haproxy.cfg \

-D -p /var/run/haproxy.pid -sf $(cat /var/run/haproxy.pid)

When the configuration is split into a few specific files (eg: tcp vs http),

it is recommended to use the "-f" option :

haproxy -f /etc/haproxy/global.cfg -f /etc/haproxy/stats.cfg \

-f /etc/haproxy/default-tcp.cfg -f /etc/haproxy/tcp.cfg \

-f /etc/haproxy/default-http.cfg -f /etc/haproxy/http.cfg \

-D -p /var/run/haproxy.pid -sf $(cat /var/run/haproxy.pid)

When an unknown number of files is expected, such as customer-specific files,

it is recommended to assign them a name starting with a fixed-size sequence

number and to use "--" to load them, possibly after loading some defaults :

haproxy -f /etc/haproxy/global.cfg -f /etc/haproxy/stats.cfg \

-f /etc/haproxy/default-tcp.cfg -f /etc/haproxy/tcp.cfg \

-f /etc/haproxy/default-http.cfg -f /etc/haproxy/http.cfg \

-D -p /var/run/haproxy.pid -sf $(cat /var/run/haproxy.pid) \

-f /etc/haproxy/default-customers.cfg -- /etc/haproxy/customers/*

Sometimes a failure to start may happen for whatever reason. Then it is

important to verify if the version of HAProxy you are invoking is the expected

version and if it supports the features you are expecting (eg: SSL, PCRE,

compression, Lua, etc). This can be verified using "haproxy -vv". Some

important information such as certain build options, the target system and

the versions of the libraries being used are reported there. It is also what

you will systematically be asked for when posting a bug report :

$ haproxy -vv

HAProxy version 1.6-dev7-a088d3-4 2015/10/08

Copyright 2000-2015 Willy Tarreau <[email protected]>

Build options :

TARGET = linux2628

CPU = generic

CC = gcc

CFLAGS = -pg -O0 -g -fno-strict-aliasing -Wdeclaration-after-statement \

-DBUFSIZE=8030 -DMAXREWRITE=1030 -DSO_MARK=36 -DTCP_REPAIR=19

OPTIONS = USE_ZLIB=1 USE_DLMALLOC=1 USE_OPENSSL=1 USE_LUA=1 USE_PCRE=1

Default settings :

maxconn = 2000, bufsize = 8030, maxrewrite = 1030, maxpollevents = 200

Encrypted password support via crypt(3): yes

Built with zlib version : 1.2.6

Compression algorithms supported : identity("identity"), deflate("deflate"), \

raw-deflate("deflate"), gzip("gzip")

Built with OpenSSL version : OpenSSL 1.0.1o 12 Jun 2015

Running on OpenSSL version : OpenSSL 1.0.1o 12 Jun 2015

OpenSSL library supports TLS extensions : yes

OpenSSL library supports SNI : yes

OpenSSL library supports prefer-server-ciphers : yes

Built with PCRE version : 8.12 2011-01-15

PCRE library supports JIT : no (USE_PCRE_JIT not set)

Built with Lua version : Lua 5.3.1

Built with transparent proxy support using: IP_TRANSPARENT IP_FREEBIND

Available polling systems :

epoll : pref=300, test result OK

poll : pref=200, test result OK

select : pref=150, test result OK

Total: 3 (3 usable), will use epoll.

The relevant information that many non-developer users can verify here are :

- the version : 1.6-dev7-a088d3-4 above means the code is currently at commit

ID "a088d3" which is the 4th one after after official version "1.6-dev7".

Version 1.6-dev7 would show as "1.6-dev7-8c1ad7". What matters here is in

fact "1.6-dev7". This is the 7th development version of what will become

version 1.6 in the future. A development version not suitable for use in

production (unless you know exactly what you are doing). A stable version

will show as a 3-numbers version, such as "1.5.14-16f863", indicating the

14th level of fix on top of version 1.5. This is a production-ready version.

- the release date : 2015/10/08. It is represented in the universal

year/month/day format. Here this means August 8th, 2015. Given that stable

releases are issued every few months (1-2 months at the beginning, sometimes

6 months once the product becomes very stable), if you're seeing an old date

here, it means you're probably affected by a number of bugs or security

issues that have since been fixed and that it might be worth checking on the

official site.

- build options : they are relevant to people who build their packages

themselves, they can explain why things are not behaving as expected. For

example the development version above was built for Linux 2.6.28 or later,

targeting a generic CPU (no CPU-specific optimizations), and lacks any

code optimization (-O0) so it will perform poorly in terms of performance.

- libraries versions : zlib version is reported as found in the library

itself. In general zlib is considered a very stable product and upgrades

are almost never needed. OpenSSL reports two versions, the version used at

build time and the one being used, as found on the system. These ones may

differ by the last letter but never by the numbers. The build date is also

reported because most OpenSSL bugs are security issues and need to be taken

seriously, so this library absolutely needs to be kept up to date. Seeing a

4-months old version here is highly suspicious and indeed an update was

missed. PCRE provides very fast regular expressions and is highly

recommended. Certain of its extensions such as JIT are not present in all

versions and still young so some people prefer not to build with them,

which is why the build status is reported as well. Regarding the Lua

scripting language, HAProxy expects version 5.3 which is very young since

it was released a little time before HAProxy 1.6. It is important to check

on the Lua web site if some fixes are proposed for this branch.

- Available polling systems will affect the process's scalability when

dealing with more than about one thousand of concurrent connections. These

ones are only available when the correct system was indicated in the TARGET

variable during the build. The "epoll" mechanism is highly recommended on

Linux, and the kqueue mechanism is highly recommended on BSD. Lacking them

will result in poll() or even select() being used, causing a high CPU usage

when dealing with a lot of connections.

4. Stopping and restarting HAProxy

----------------------------------

HAProxy supports a graceful and a hard stop. The hard stop is simple, when the

SIGTERM signal is sent to the haproxy process, it immediately quits and all

established connections are closed. The graceful stop is triggered when the

SIGUSR1 signal is sent to the haproxy process. It consists in only unbinding

from listening ports, but continue to process existing connections until they

close. Once the last connection is closed, the process leaves.

The hard stop method is used for the "stop" or "restart" actions of the service

management script. The graceful stop is used for the "reload" action which

tries to seamlessly reload a new configuration in a new process.

Both of these signals may be sent by the new haproxy process itself during a

reload or restart, so that they are sent at the latest possible moment and only

if absolutely required. This is what is performed by the "-st" (hard) and "-sf"

(graceful) options respectively.

In master-worker mode, it is not needed to start a new haproxy process in

order to reload the configuration. The master process reacts to the SIGUSR2

signal by reexecuting itself with the -sf parameter followed by the PIDs of

the workers. The master will then parse the configuration file and fork new

workers.

To understand better how these signals are used, it is important to understand

the whole restart mechanism.

First, an existing haproxy process is running. The administrator uses a system

specific command such as "/etc/init.d/haproxy reload" to indicate they want to

take the new configuration file into effect. What happens then is the following.

First, the service script (/etc/init.d/haproxy or equivalent) will verify that

the configuration file parses correctly using "haproxy -c". After that it will

try to start haproxy with this configuration file, using "-st" or "-sf".

Then HAProxy tries to bind to all listening ports. If some fatal errors happen

(eg: address not present on the system, permission denied), the process quits

with an error. If a socket binding fails because a port is already in use, then

the process will first send a SIGTTOU signal to all the pids specified in the

"-st" or "-sf" pid list. This is what is called the "pause" signal. It instructs

all existing haproxy processes to temporarily stop listening to their ports so

that the new process can try to bind again. During this time, the old process

continues to process existing connections. If the binding still fails (because

for example a port is shared with another daemon), then the new process sends a

SIGTTIN signal to the old processes to instruct them to resume operations just

as if nothing happened. The old processes will then restart listening to the

ports and continue to accept connections. Note that this mechanism is system

dependent and some operating systems may not support it in multi-process mode.

If the new process manages to bind correctly to all ports, then it sends either

the SIGTERM (hard stop in case of "-st") or the SIGUSR1 (graceful stop in case

of "-sf") to all processes to notify them that it is now in charge of operations

and that the old processes will have to leave, either immediately or once they

have finished their job.

It is important to note that during this timeframe, there are two small windows

of a few milliseconds each where it is possible that a few connection failures

will be noticed during high loads. Typically observed failure rates are around

1 failure during a reload operation every 10000 new connections per second,

which means that a heavily loaded site running at 30000 new connections per

second may see about 3 failed connection upon every reload. The two situations

where this happens are :

- if the new process fails to bind due to the presence of the old process,

it will first have to go through the SIGTTOU+SIGTTIN sequence, which

typically lasts about one millisecond for a few tens of frontends, and

during which some ports will not be bound to the old process and not yet

bound to the new one. HAProxy works around this on systems that support the

SO_REUSEPORT socket options, as it allows the new process to bind without

first asking the old one to unbind. Most BSD systems have been supporting

this almost forever. Linux has been supporting this in version 2.0 and

dropped it around 2.2, but some patches were floating around by then. It

was reintroduced in kernel 3.9, so if you are observing a connection

failure rate above the one mentioned above, please ensure that your kernel

is 3.9 or newer, or that relevant patches were backported to your kernel

(less likely).

- when the old processes close the listening ports, the kernel may not always

redistribute any pending connection that was remaining in the socket's

backlog. Under high loads, a SYN packet may happen just before the socket

is closed, and will lead to an RST packet being sent to the client. In some

critical environments where even one drop is not acceptable, these ones are

sometimes dealt with using firewall rules to block SYN packets during the

reload, forcing the client to retransmit. This is totally system-dependent,

as some systems might be able to visit other listening queues and avoid

this RST. A second case concerns the ACK from the client on a local socket

that was in SYN_RECV state just before the close. This ACK will lead to an

RST packet while the haproxy process is still not aware of it. This one is

harder to get rid of, though the firewall filtering rules mentioned above

will work well if applied one second or so before restarting the process.

For the vast majority of users, such drops will never ever happen since they

don't have enough load to trigger the race conditions. And for most high traffic

users, the failure rate is still fairly within the noise margin provided that at

least SO_REUSEPORT is properly supported on their systems.

5. File-descriptor limitations

------------------------------

In order to ensure that all incoming connections will successfully be served,

HAProxy computes at load time the total number of file descriptors that will be

needed during the process's life. A regular Unix process is generally granted

1024 file descriptors by default, and a privileged process can raise this limit

itself. This is one reason for starting HAProxy as root and letting it adjust

the limit. The default limit of 1024 file descriptors roughly allow about 500

concurrent connections to be processed. The computation is based on the global

maxconn parameter which limits the total number of connections per process, the

number of listeners, the number of servers which have a health check enabled,

the agent checks, the peers, the loggers and possibly a few other technical

requirements. A simple rough estimate of this number consists in simply

doubling the maxconn value and adding a few tens to get the approximate number

of file descriptors needed.

Originally HAProxy did not know how to compute this value, and it was necessary

to pass the value using the "ulimit-n" setting in the global section. This

explains why even today a lot of configurations are seen with this setting

present. Unfortunately it was often miscalculated resulting in connection

failures when approaching maxconn instead of throttling incoming connection

while waiting for the needed resources. For this reason it is important to

remove any vestigial "ulimit-n" setting that can remain from very old versions.

Raising the number of file descriptors to accept even moderate loads is

mandatory but comes with some OS-specific adjustments. First, the select()

polling system is limited to 1024 file descriptors. In fact on Linux it used

to be capable of handling more but since certain OS ship with excessively

restrictive SELinux policies forbidding the use of select() with more than

1024 file descriptors, HAProxy now refuses to start in this case in order to

avoid any issue at run time. On all supported operating systems, poll() is

available and will not suffer from this limitation. It is automatically picked

so there is nothing to do to get a working configuration. But poll's becomes

very slow when the number of file descriptors increases. While HAProxy does its

best to limit this performance impact (eg: via the use of the internal file

descriptor cache and batched processing), a good rule of thumb is that using

poll() with more than a thousand concurrent connections will use a lot of CPU.

For Linux systems base on kernels 2.6 and above, the epoll() system call will

be used. It's a much more scalable mechanism relying on callbacks in the kernel

that guarantee a constant wake up time regardless of the number of registered

monitored file descriptors. It is automatically used where detected, provided

that HAProxy had been built for one of the Linux flavors. Its presence and

support can be verified using "haproxy -vv".

For BSD systems which support it, kqueue() is available as an alternative. It

is much faster than poll() and even slightly faster than epoll() thanks to its

batched handling of changes. At least FreeBSD and OpenBSD support it. Just like

with Linux's epoll(), its support and availability are reported in the output

of "haproxy -vv".

Having a good poller is one thing, but it is mandatory that the process can

reach the limits. When HAProxy starts, it immediately sets the new process's

file descriptor limits and verifies if it succeeds. In case of failure, it

reports it before forking so that the administrator can see the problem. As

long as the process is started by as root, there should be no reason for this

setting to fail. However, it can fail if the process is started by an

unprivileged user. If there is a compelling reason for *not* starting haproxy

as root (eg: started by end users, or by a per-application account), then the

file descriptor limit can be raised by the system administrator for this

specific user. The effectiveness of the setting can be verified by issuing

"ulimit -n" from the user's command line. It should reflect the new limit.

Warning: when an unprivileged user's limits are changed in this user's account,

it is fairly common that these values are only considered when the user logs in

and not at all in some scripts run at system boot time nor in crontabs. This is

totally dependent on the operating system, keep in mind to check "ulimit -n"

before starting haproxy when running this way. The general advice is never to

start haproxy as an unprivileged user for production purposes. Another good

reason is that it prevents haproxy from enabling some security protections.

Once it is certain that the system will allow the haproxy process to use the

requested number of file descriptors, two new system-specific limits may be

encountered. The first one is the system-wide file descriptor limit, which is

the total number of file descriptors opened on the system, covering all

processes. When this limit is reached, accept() or socket() will typically

return ENFILE. The second one is the per-process hard limit on the number of

file descriptors, it prevents setrlimit() from being set higher. Both are very

dependent on the operating system. On Linux, the system limit is set at boot

based on the amount of memory. It can be changed with the "fs.file-max" sysctl.

And the per-process hard limit is set to 1048576 by default, but it can be

changed using the "fs.nr_open" sysctl.

File descriptor limitations may be observed on a running process when they are

set too low. The strace utility will report that accept() and socket() return

"-1 EMFILE" when the process's limits have been reached. In this case, simply

raising the "ulimit-n" value (or removing it) will solve the problem. If these

system calls return "-1 ENFILE" then it means that the kernel's limits have

been reached and that something must be done on a system-wide parameter. These

trouble must absolutely be addressed, as they result in high CPU usage (when

accept() fails) and failed connections that are generally visible to the user.

One solution also consists in lowering the global maxconn value to enforce

serialization, and possibly to disable HTTP keep-alive to force connections

to be released and reused faster.

6. Memory management

--------------------

HAProxy uses a simple and fast pool-based memory management. Since it relies on

a small number of different object types, it's much more efficient to pick new

objects from a pool which already contains objects of the appropriate size than

to call malloc() for each different size. The pools are organized as a stack or

LIFO, so that newly allocated objects are taken from recently released objects

still hot in the CPU caches. Pools of similar sizes are merged together, in

order to limit memory fragmentation.

By default, since the focus is set on performance, each released object is put

back into the pool it came from, and allocated objects are never freed since

they are expected to be reused very soon.

On the CLI, it is possible to check how memory is being used in pools thanks to

the "show pools" command :

> show pools

Dumping pools usage. Use SIGQUIT to flush them.

- Pool cache_st (16 bytes) : 0 allocated (0 bytes), 0 used, 0 failures, 1 users, @0x9ccc40=03 [SHARED]

- Pool pipe (32 bytes) : 5 allocated (160 bytes), 5 used, 0 failures, 2 users, @0x9ccac0=00 [SHARED]

- Pool comp_state (48 bytes) : 3 allocated (144 bytes), 3 used, 0 failures, 5 users, @0x9cccc0=04 [SHARED]

- Pool filter (64 bytes) : 0 allocated (0 bytes), 0 used, 0 failures, 3 users, @0x9ccbc0=02 [SHARED]

- Pool vars (80 bytes) : 0 allocated (0 bytes), 0 used, 0 failures, 2 users, @0x9ccb40=01 [SHARED]

- Pool uniqueid (128 bytes) : 0 allocated (0 bytes), 0 used, 0 failures, 2 users, @0x9cd240=15 [SHARED]

- Pool task (144 bytes) : 55 allocated (7920 bytes), 55 used, 0 failures, 1 users, @0x9cd040=11 [SHARED]

- Pool session (160 bytes) : 1 allocated (160 bytes), 1 used, 0 failures, 1 users, @0x9cd140=13 [SHARED]

- Pool h2s (208 bytes) : 0 allocated (0 bytes), 0 used, 0 failures, 2 users, @0x9ccec0=08 [SHARED]

- Pool h2c (288 bytes) : 0 allocated (0 bytes), 0 used, 0 failures, 1 users, @0x9cce40=07 [SHARED]

- Pool spoe_ctx (304 bytes) : 0 allocated (0 bytes), 0 used, 0 failures, 2 users, @0x9ccf40=09 [SHARED]

- Pool connection (400 bytes) : 2 allocated (800 bytes), 2 used, 0 failures, 1 users, @0x9cd1c0=14 [SHARED]

- Pool hdr_idx (416 bytes) : 0 allocated (0 bytes), 0 used, 0 failures, 1 users, @0x9cd340=17 [SHARED]

- Pool dns_resolut (480 bytes) : 0 allocated (0 bytes), 0 used, 0 failures, 1 users, @0x9ccdc0=06 [SHARED]

- Pool dns_answer_ (576 bytes) : 0 allocated (0 bytes), 0 used, 0 failures, 1 users, @0x9ccd40=05 [SHARED]

- Pool stream (960 bytes) : 1 allocated (960 bytes), 1 used, 0 failures, 1 users, @0x9cd0c0=12 [SHARED]

- Pool requri (1024 bytes) : 0 allocated (0 bytes), 0 used, 0 failures, 1 users, @0x9cd2c0=16 [SHARED]

- Pool buffer (8030 bytes) : 3 allocated (24090 bytes), 2 used, 0 failures, 1 users, @0x9cd3c0=18 [SHARED]

- Pool trash (8062 bytes) : 1 allocated (8062 bytes), 1 used, 0 failures, 1 users, @0x9cd440=19

Total: 19 pools, 42296 bytes allocated, 34266 used.

The pool name is only indicative, it's the name of the first object type using

this pool. The size in parenthesis is the object size for objects in this pool.

Object sizes are always rounded up to the closest multiple of 16 bytes. The

number of objects currently allocated and the equivalent number of bytes is

reported so that it is easy to know which pool is responsible for the highest

memory usage. The number of objects currently in use is reported as well in the

"used" field. The difference between "allocated" and "used" corresponds to the

objects that have been freed and are available for immediate use. The address

at the end of the line is the pool's address, and the following number is the

pool index when it exists, or is reported as -1 if no index was assigned.

It is possible to limit the amount of memory allocated per process using the

"-m" command line option, followed by a number of megabytes. It covers all of

the process's addressable space, so that includes memory used by some libraries

as well as the stack, but it is a reliable limit when building a resource

constrained system. It works the same way as "ulimit -v" on systems which have

it, or "ulimit -d" for the other ones.

If a memory allocation fails due to the memory limit being reached or because

the system doesn't have any enough memory, then haproxy will first start to

free all available objects from all pools before attempting to allocate memory

again. This mechanism of releasing unused memory can be triggered by sending

the signal SIGQUIT to the haproxy process. When doing so, the pools state prior

to the flush will also be reported to stderr when the process runs in

foreground.

During a reload operation, the process switched to the graceful stop state also

automatically performs some flushes after releasing any connection so that all

possible memory is released to save it for the new process.

7. CPU usage

------------

HAProxy normally spends most of its time in the system and a smaller part in

userland. A finely tuned 3.5 GHz CPU can sustain a rate about 80000 end-to-end

connection setups and closes per second at 100% CPU on a single core. When one

core is saturated, typical figures are :

- 95% system, 5% user for long TCP connections or large HTTP objects

- 85% system and 15% user for short TCP connections or small HTTP objects in

close mode

- 70% system and 30% user for small HTTP objects in keep-alive mode

The amount of rules processing and regular expressions will increase the user

land part. The presence of firewall rules, connection tracking, complex routing

tables in the system will instead increase the system part.

On most systems, the CPU time observed during network transfers can be cut in 4

parts :

- the interrupt part, which concerns all the processing performed upon I/O

receipt, before the target process is even known. Typically Rx packets are

accounted for in interrupt. On some systems such as Linux where interrupt

processing may be deferred to a dedicated thread, it can appear as softirq,

and the thread is called ksoftirqd/0 (for CPU 0). The CPU taking care of

this load is generally defined by the hardware settings, though in the case

of softirq it is often possible to remap the processing to another CPU.

This interrupt part will often be perceived as parasitic since it's not

associated with any process, but it actually is some processing being done

to prepare the work for the process.

- the system part, which concerns all the processing done using kernel code

called from userland. System calls are accounted as system for example. All

synchronously delivered Tx packets will be accounted for as system time. If

some packets have to be deferred due to queues filling up, they may then be

processed in interrupt context later (eg: upon receipt of an ACK opening a

TCP window).

- the user part, which exclusively runs application code in userland. HAProxy

runs exclusively in this part, though it makes heavy use of system calls.

Rules processing, regular expressions, compression, encryption all add to

the user portion of CPU consumption.

- the idle part, which is what the CPU does when there is nothing to do. For

example HAProxy waits for an incoming connection, or waits for some data to

leave, meaning the system is waiting for an ACK from the client to push

these data.

In practice regarding HAProxy's activity, it is in general reasonably accurate

(but totally inexact) to consider that interrupt/softirq are caused by Rx

processing in kernel drivers, that user-land is caused by layer 7 processing

in HAProxy, and that system time is caused by network processing on the Tx

path.

Since HAProxy runs around an event loop, it waits for new events using poll()

(or any alternative) and processes all these events as fast as possible before

going back to poll() waiting for new events. It measures the time spent waiting

in poll() compared to the time spent doing processing events. The ratio of

polling time vs total time is called the "idle" time, it's the amount of time

spent waiting for something to happen. This ratio is reported in the stats page

on the "idle" line, or "Idle_pct" on the CLI. When it's close to 100%, it means

the load is extremely low. When it's close to 0%, it means that there is

constantly some activity. While it cannot be very accurate on an overloaded

system due to other processes possibly preempting the CPU from the haproxy

process, it still provides a good estimate about how HAProxy considers it is

working : if the load is low and the idle ratio is low as well, it may indicate

that HAProxy has a lot of work to do, possibly due to very expensive rules that

have to be processed. Conversely, if HAProxy indicates the idle is close to

100% while things are slow, it means that it cannot do anything to speed things

up because it is already waiting for incoming data to process. In the example

below, haproxy is completely idle :

$ echo "show info" | socat - /var/run/haproxy.sock | grep ^Idle

Idle_pct: 100

When the idle ratio starts to become very low, it is important to tune the

system and place processes and interrupts correctly to save the most possible

CPU resources for all tasks. If a firewall is present, it may be worth trying

to disable it or to tune it to ensure it is not responsible for a large part

of the performance limitation. It's worth noting that unloading a stateful

firewall generally reduces both the amount of interrupt/softirq and of system

usage since such firewalls act both on the Rx and the Tx paths. On Linux,

unloading the nf_conntrack and ip_conntrack modules will show whether there is

anything to gain. If so, then the module runs with default settings and you'll

have to figure how to tune it for better performance. In general this consists

in considerably increasing the hash table size. On FreeBSD, "pfctl -d" will

disable the "pf" firewall and its stateful engine at the same time.

If it is observed that a lot of time is spent in interrupt/softirq, it is

important to ensure that they don't run on the same CPU. Most systems tend to

pin the tasks on the CPU where they receive the network traffic because for

certain workloads it improves things. But with heavily network-bound workloads

it is the opposite as the haproxy process will have to fight against its kernel

counterpart. Pinning haproxy to one CPU core and the interrupts to another one,

all sharing the same L3 cache tends to sensibly increase network performance

because in practice the amount of work for haproxy and the network stack are

quite close, so they can almost fill an entire CPU each. On Linux this is done

using taskset (for haproxy) or using cpu-map (from the haproxy config), and the

interrupts are assigned under /proc/irq. Many network interfaces support

multiple queues and multiple interrupts. In general it helps to spread them

across a small number of CPU cores provided they all share the same L3 cache.

Please always stop irq_balance which always does the worst possible thing on

such workloads.

For CPU-bound workloads consisting in a lot of SSL traffic or a lot of

compression, it may be worth using multiple processes dedicated to certain

tasks, though there is no universal rule here and experimentation will have to

be performed.

In order to increase the CPU capacity, it is possible to make HAProxy run as

several processes, using the "nbproc" directive in the global section. There

are some limitations though :

- health checks are run per process, so the target servers will get as many

checks as there are running processes ;

- maxconn values and queues are per-process so the correct value must be set

to avoid overloading the servers ;

- outgoing connections should avoid using port ranges to avoid conflicts