In this demo, Serf is used to coordinate web servers entering a load balancer. The nodes in this example can come up in any order and they'll eventually converge to a running cluster. When visiting the load balancer, you'll see each of the nodes visited in round-robin.

After telling the cluster to launch, no further human interaction is required. The cluster will launch and orchestrate itself into a load balanced web server setup.

The nodes are launched on AWS using CloudFormation and therefore will incur a real cost. You can control this cost based on how many nodes you decide to launch.

To run the demo, log into your AWS console, go to the CloudFormation tab, and create a new stack using the "cloudformation.json" file in this directory.

Configure how many web nodes you'd like to launch then create the stack. When the stack is finished creating, the load balancer IP will be in the "outputs" tab of that stack.

Visit http://IP:9999/ to see the HAProxy stats page and list of configured backends. Or visit the IP directly in your browser to see it round robin between the web nodes.

It will take some time for the load balancer and web servers to install their software and configure themselves. If the load balancer stats page is not up yet, give it a few seconds and try again. Once it is up, it will automatically refresh so you can watch as nodes eventually come online.

All the nodes (webs and load balancer) come up at the same time, with no explicit ordering defined. If there is an order, it doesn't actually matter for the function of the demo. The web servers can all come up before the load balancer, the load balancer can come up in the middle of some web servers, etc.

Each of the nodes does an initial setup using a shell script. In a real environment, this initial setup would probably be done with a real configuration management system. For the purposes of this demo, we use shell scripts for ease of understanding.

The load balancer runs setup_load_balancer.sh which installs HAProxy. HAProxy is configured with no backend servers. Serf will handle this later.

The web server runs setup_web_server.sh which installs Apache and sets up an index.html to just say what host it is.

At this stage, the load balancer is running HAProxy, and the web servers are running Apache, but they're not aware of each other in any way.

Next, all nodes run setup_serf.sh which installs Serf and configures it to run. This script also starts the Serf agent. For web servers, it also starts a task that continuosly tries to join the Serf cluster at address 10.0.0.5, which happens to be the static IP assigned to the load balancer. Note that Serf can join any node in the cluster. We just set a static IP on the load balancer for ease of automation, but in a real production environment you can use any existing node, potentially just using DNS.

Web server roles also make use of a "serf-query" upstart task to dynamically update the index page they serve. By using a "load" query, each node asks the rest of the cluster for their current load. This information is used to dynamically generate the index page.

The Serf configuration is very simple. Each node runs a Serf agent. The web servers run the agent with a role of "web" and the load balancer runs the agent with a role of "lb".

On member-join, the following shell script is run. We'll talk about the function of the shell script after the code sample.

if [ "x${SERF_TAG_ROLE}" != "xlb" ]; then
    echo "Not an lb. Ignoring member join."
    exit 0
fi

while read line; do
    ROLE=`echo $line | awk '{print \$3 }'`
    if [ "x${ROLE}" != "xweb" ]; then
        continue
    fi

    echo $line | \
        awk '{ printf "    server %s %s check\n", $1, $2 }' >>/etc/haproxy/haproxy.cfg
done

/etc/init.d/haproxy reload

The script first checks if this node is a load balancer. If it isn't a load balancer, we don't do anything on the "member-join" event. Web servers simply don't care about other members.

Otherwise, if this is a load balancer, then it handles the "member-join" event and reads the event data and updates the configuration of HAProxy. In a production environment, you might use something like a template processor to do this rather than appending to the configuration. For the purposes of this demo, this works as well.

Finally, HAProxy is reloaded so the configuration is read.

The result of this is that as members join, they're automatically added into rotation on the load balancer.

On member-leave and member-failed evets, the following shell script is run. This demo doesn't differentiate between the two events, treating both the same way and simply removing the node from the HAProxy configuration.

if [ "x${SERF_TAG_ROLE}" != "xlb" ]; then
    echo "Not an lb. Ignoring member leave"
    exit 0
fi

while read line; do
    NAME=`echo $line | awk '{print \$1 }'`
    sed -i'' "/${NAME} /d" /etc/haproxy/haproxy.cfg
done

/etc/init.d/haproxy reload

This script also does nothing if this node is not a load balanacer. Otherwise, it uses sed to remove each of the nodes from the HAProxy configuration and reloads it.