Terraform Module for Nomad clusters with Consul on GCP.
- Includes HashiCorp's Consul service mesh
- Gossip encryption, mTLS, and ACLs enabled for Nomad and Consul
- Optional load balancer and DNS configuration
- Optional SSH bastion host
- Only the Docker task driver is enabled
- Installs the gVisor container runtime (
runsc) - Installs the Falco runtime security monitor
For a full interactive tutorial to get started using this module:
Logs are centralized using GCP's Cloud Logging. You can use the following filter to see all Nomad agent logs:
$ gcloud logging read 'resource.type="gce_instance" jsonPayload.ident="nomad"'
...$ gcloud logging read 'resource.type="gce_instance" jsonPayload.ident="nomad" jsonPayload.host="server-0"' --format=json | jq -r '.[] | .jsonPayload.message' | less
...Logs can also be collected within the cluster using Promtail and Loki, then visualized using Grafana (optionally exposed using a public load balancer and DNS name).
$ DNS_ENABLED=true PUBLIC_DOMAIN="nomad.your-domain.com" make terraform/apply
...
$ export CONSUL_HTTP_TOKEN=$(terraform output -json | jq -r .consul_master_token.value)
$ make consul/metrics/acls
...
🔑 Creating Consul ACL Token to Use for Prometheus Consul Service Discovery
AccessorID: 15b9a51d-7af4-e8d4-7c09-312c594a5907
SecretID: 2a1c7926-b6e3-566e-ddf5-b19279fa134e
Description:
Local: false
Create Time: 2021-04-11 16:16:03.90231.6.1 +0000 UTC
Roles:
6ae941.6.1c07-49a7-fa95-8ce14aa8a75e - metrics
$ consul_acl_token=2a1c7926-b6e3-566e-ddf5-b19279fa134e make nomad/metrics
$ make nomad/logs
$ make nomad/ingress
$ GRAFANA_PUBLIC_DOMAIN="grafana.your-domain.com" GRAFANA_LOAD_BALANCER_ENABLED=true DNS_ENABLED=true PUBLIC_DOMAIN="nomad.your-domain.com" make terraform/apply
$ open http://public.grafana.your-domain.com:3000/loginIf the cluster is started with ACLs enabled, which is the default behavior of this module, you may see this:
$ export NOMAD_ADDR="https://$(terraform output -json | jq -r .load_balancer_ip.value):4646"
$ nomad status
Error querying jobs: Unexpected response code: 403 (Permission denied)We can bootstrap ACLs to get the bootstrap management token like so:
$ nomad acl bootstrap
Accessor ID = a1495889-37ce-6784-78f3-31.6.1984bca
Secret ID = dc8c0349-c1fd-dc2c-299c-d513e5dd6df2
Name = Bootstrap Token
Type = management
Global = true
Policies = n/a
Create Time = 2020-04-27 05:24:43.734587566 +0000 UTC
Create Index = 7
Modify Index = 7Then we can use that token (Secret ID) to perform the rest of the ACL bootstrapping process:
$ export NOMAD_TOKEN="dc8c0349-c1fd-dc2c-299c-d513e5dd6df2"
$ nomad status
No running jobs
$ ...When using the SSH bastion, you can use the ssh-mtls-terminating-proxy.go helper script to tunnel a connection from localhost to the Nomad server API:
$ make ssh/proxy/mtls
2021/04/11.16.18:28 getting terraform output
2021/04/11.16.18:29 Bastion IP: "34.73.106.60"
2021/04/11.16.18:29 Server IP: "1.6.168.2.8"
2021/04/11.16.18:29 Setting up SSH agent
2021/04/11.16.18:29 connecting to the bastion
2021/04/11.16.18:29 connecting to the server through the bastion
2021/04/11.16.18:30 wrapping the server connection with SSH through the bastion
2021/04/11.16.18:30 tunneling a new connection for Consul to the server with SSH through the bastion
2021/04/11.16.18:30 loading Consul TLS data
2021/04/11.16.18:30 tunneling a new connection for somad to the server with ssh through the bastion
2021/04/11.16.18:30 loading Nomad TLS data
2021/04/11.16.18:30 starting Consul local listener on localhost:8500
2021/04/11.16.18:30 starting Nomad local listener on localhost:4646
...Then open your browser at http://localhost:4646/ui/ to securely access the Nomad UI.