A project for monitoring and alerting with cloudfoundry utilizing the InfluxData TICK stack

For this project we have packaged the InfluxDB, Telegraf, Kapacitor, firehose nozzle, and Grafana components into a docker compose enviornment to allow for a compact and easily portable solution.

To run the project, you will need the following:

1. A working bosh/cloud-foundry enviornment utilizing the Cloud Foundry Diego architecture
2. A docker host with docker and docker compose installed and configured. This project has been tested with docker and docker-compose versions 17.03.1-ce

Update the following section in your cloud foundry manifest and redeploy cf to enable a uaa client for the firehose nozzle:

properties:
  uaa:
    clients:
      influxdb-firehose-nozzle:
        access-token-validity: 1209600
        authorized-grant-types: authorization_code,client_credentials,refresh_token
        override: true
        secret: <password>
        scope: openid,oauth.approvals,doppler.firehose
        authorities: oauth.login,doppler.firehose

First clone this repo to the docker host and change the following files to reflect your environment:

docker-compose.yml: update this to reflect the name of your cf enviornment, which will be the database name in influx

  environment:
  - PRE_CREATE_DB=cf_np  

The firehose nozzle is configured via environment variables in the docker-compose.yml. Variables passed here override the ones built inside the nozzle container via the nozzle json file. At a minimum, you will want to change these six variables to reference your cf deployment. The database and deployment names given here should match the env given as the influxd database above:

  NOZZLE_UAAURL=https://uaa.cf-np.company.com
  NOZZLE_CLIENT=client_id
  NOZZLE_CLIENT_SECRET=secret
  NOZZLE_TRAFFICCONTROLLERURL=wss://doppler.cf-np.company.com:443
  NOZZLE_DEPLOYMENT=cf_np
  NOZZLE_INFLUXDB_DATABASE=cf_np

Additional environment variable options can be found in the upstream project

If you changed your PRE_CREATE_DB value above, then make the following changes:

grafana/load.sh: replace any string of 'cf_np' with your PRE_CREATE_DB value
grafana/dashboards/import_format/*.json: replace any string of 'cf_np' with your PRE_CREATE_DB value

docker-compose.yml:: update the following environment variables for your slack instance

  KAPACITOR_SLACK_URL=https://hooks.slack.com/services/XXXX/YYYYY/ZZZZZZZZZZZZZ
  KAPACITOR_SLACK_CHANNEL=#bot-testing

kapacitor/*.tick: If you are using the built in tick scripts, you will have to update the vars at the top of the tick scripts such as grfana_url,grafanaenv, database, or slackchannel to represent your enviornment.

Follow the directions in the bosh-hm-forwarder project to either deploy a dedicated bosh release for the forwarder, or add the job to an existing job in your cf deployment. Instructions for adding the job to the existing consul_z2 job of your cf deployment are below:

Upload the release to your director

bosh target <director IP>
bosh upload release https://bosh.io/d/github.com/cloudfoundry/bosh-hm-forwarder-release

Add the following items to your consul_z2 job section and the releases section of your cf.yml

jobs:
- default_networks:
  - name: cf2
  name: consul_z2
  templates:
  - name: boshhmforwarder
    release: bosh-hm-forwarder

releases:
- name: cf
  version: latest
- name: bosh-hm-forwarder
  version: latest

Do a bosh deploy for your cf deployement

Update the following section in your bosh manifest and redeploy bosh to enable the bosh monitor statistics:

    hm:
      tsdb_enabled: true
      tsdb:
        address: <ip of the bosh-hm-forwarder job (IP of consul_z2 VM in this example) >
        port: 4000
      consul_event_forwarder_enabled: true
      consul_event_forwarder:
        host: <ip of your docker host>
        port: 8125
        protocol: http
        events: true
        heartbeats_as_alerts: true

In the top level of the project directory, use the following command to create the docker compose application and verify successful start:

 docker-compose up -d && docker-compose ps

Auto-defining and enabling the included tick scripts is currently not supported in Kapacitor, so you need to manually log into kapacitor after it is up and enable any desired scripts. In the below example it assumes your influx database is cf_np. If you changed PRE_CREATE_DB above, then substitute your database name below for the -dbrp field

will be addressed by influxdata/kapacitor#1481

docker exec -it <idofkapacitorcontainer> bash

cd /etc/kapacitor/
kapacitor define etcd_alert_np -type stream -tick etcd_alert_np.tick -dbrp cf_np.autogen
kapacitor enable etcd_alert_np
kapacitor define slow_consumer_np -type stream -tick slow_consumer_np.tick -dbrp cf_np.autogen
kapacitor enable slow_consumer_np
kapacitor define swap_alert_np -type stream -tick swap_alert_np.tick -dbrp cf_np.autogen
kapacitor enable swap_alert_np
kapacitor define job_health_np -type stream -tick job_health_np.tick -dbrp cf_np.autogen
kapacitor enable job_health_np
kapacitor define persistent_disk_np -type stream -tick persistent_disk_np.tick -dbrp cf_np.autogen
kapacitor enable persistent_disk_np
kapacitor define cpu_wait_alert_np -type stream -tick cpu_wait_np.tick -dbrp cf_np.autogen
kapacitor enable cpu_wait_alert_np
kapacitor define bosh_event_np -type stream -tick bosh_event_np.tick -dbrp telegraf_np.autogen
kapacitor enable bosh_event_np
kapacitor define max_container_np -type stream -tick max_container_np.tick -dbrp cf_np.autogen
kapacitor enable max_container_np

kapacitor list tasks

Grafana will be configured for the datasources you specified in the config file (cf_np & influx by default) and will have some preloaded dashboards which show relevant cf and bosh metric data.

You can use grafana to create you're own dashboards or the following dashboards have been included:

Cloud Foundry Job specific dashboards use metrics from the Firehose to show things like Cell available memory ratios and CPU load. Annotations are included to show "bosh deploy" events and enable correlation between changes and incidents.

VM dashboards provide VM level statistics from BOSH Monitor to show things like ephemeral disk or swap utilzation.

Enabled tick alert scripts in Kapacitor (such as Cell memory and bosh deploy's) will go to your slack channel for team notification. You can also modify the Kapacitor tickscripts to send alerts to any number of supported alert event handlers like email or pagerduty.