How to Configure and Manage Terraform Backends

What’s the “Terraform State” ?

Terraform holds an internal representation of your resources, known as the Terraform state. That means that if you write some Terraform code to deploy and manage an EC2 instance, the Terraform state will say that this instance was created, what is its ID, and some additional properties on its state and metadata. When you run the Terraform commands to update or destroy that instance, Terraform will use the state to know what instance your code is referring to. 

Because of this - it’s vital that the state is persisted between runs of Terraform. Otherwise - in the example above - the second run won’t know that an instance already exists. 

Local backend

If you run Terraform without a "Terraform backend” - the state will be persisted locally to a state file named “terraform.tfstate”. This works well when you are a sole developer, running everything locally. When you need to collaborate with others on your IaC - that local Terraform state file becomes a nuisance - as it must be synced between all collaborators. This is where Terraform remote backends come into the mix. 

What’s a “Terraform Backend” ?

Terraform backends are a native Terraform feature, which saves the state file in a remote location, rather than a local file. Using a Terraform backend means we don’t have to worry about sharing the state between team members. We must provide the infrastructure for storing the state, but there are many options available, spanning the most common public cloud solutions out there. Terraform backend types include AWS S3, GCP Cloud Storage, Azure Blob Storage, and others.   

Benefits of Using Terraform Backends

Improved Team Collaboration

This benefit stands above others, as collaboration on a Terraform stack without a Terraform backend is nearly impossible. Terraform backends make sure that the work on the stack stays true to the state of our resources, and that we don’t run over our colleagues' work. Some Terraform backend types also utilize state locking, which makes sure we don’t corrupt the state by working on it simultaneously. 

State Management and Versioning

When you use a Terraform backend, it’s recommended that you turn on “versioning” for your storage engine (where possible). This means you’ll be able to inspect and even roll back to previous versions of your state, in case something goes wrong. More advanced Terraform backends - like the env0 remote backend - will provide deeper insight into your state and allow you to see it change between revisions, as well as correlate it to the actual Terraform runs that changed it.

Enhanced Security and Access Control

Terraform state will hold some sensitive data about your cloud resources. When using local state - that sensitive data will be stored in a plain-text JSON file on your local file system - an unsecure solution. When you use a remote Terraform backend - the sensitive information will only be held in-memory while Terraform runs, and never stored on your local machine. It’s recommended to use a Terraform backend that will also encrypt your data at rest - such as the env0 remote backend or the s3 backend (when configured correctly). 

How to configure a Terraform Backend?

Terraform backend configuration

Configuring the Terraform backend is done as part of your Terraform code, by adding a backend block within the top-level `terraform` block (which holds the Terraform configuration). The following example configures an AWS S3 Terraform backend.

terraform {
  backend "s3" {
    bucket = "mybucket"
    key = "path/to/my/key"
    region = "us-east-1"
  }
}

There are some important limitations on backend configuration you should be aware of:

• A configuration can only provide one backend block - so your Terraform stack can only point at a single Terraform backend.
• A backend block cannot refer to named values - The configuration for the backend cannot use variables or reference locals or data sources. We’ll discuss dynamic configuration possibilities later on. 

Terraform init command

During the terraform init command, Terraform will look for the backend block in the root Terraform configuration file, and the chosen backend is initialized using the given configuration settings. This might include reading or even writing some data to your configured Terraform backend, so make sure you run the Terraform init command with credentials to access your cloud.

Re-running init with an already-initialized backend will update the working directory to use the new backend settings, see more in "Backend migrations and changes" below.

Which Backend Type should I use?

As we’ve seen - Terraform provides a few options to choose from. There are a few considerations to guide you when choosing a backend.

The stack you are already using

If you are already using a specific public cloud, for example - Google Cloud or MS Azure - it would be wise to select a backend that is on the cloud you are already using. This will save you a lot of extra work in creating new accounts and granting access, and will be easier to manage. If you are using Kubernetes or Postgres - these are also possibilities, that will allow you to utilize the stack you already have, instead of creating & managing more infrastructure.

If you are running your Terraform workload from a TACO platform - such as env0 - you can also benefit from using the platform’s own remote backend. 

State Locking

Not all backend types support State Locking. If you are planning on having a larger team work on your IaC, and expect simultaneous runs might occur - it would be wise to make sure you choose a backend that supports State Locking, and configure it accordingly.

Encryption

As I’ve mentioned before - your state may include sensitive information. The state is not stored in an encrypted manner, but it is recommended to use a backend that can at least encrypt the data at rest. Make sure you configure your backend for that - it’s not a setting you choose in the Terraform configuration, but rather something you need to configure when you set up the infrastructure for your backend - we’ll be diving into that next. 

Backend Infrastructure

In order to use a Backend, you’ll need to set up some infrastructure for it. 

For example, an S3 Backend will require an S3 Bucket, and a DynamoDB Table (if you choose to enable State Locking). It might also require a KMS key if you choose to encrypt that bucket with a non-default key. 

It’s recommended that you read up on the specifics of the backend you are going to use, before you make your final decision, as allocating these resources might not be possible for everyone.

Required permissions

The Terraform process you’ll be running will need permissions to alter your cloud infrastructure. But when you use a Backend, you must make sure that the Terraform process also has permissions to write and read from your Backend infrastructure. 

For example - If I’m using Terraform to manage Azure Virtual machines, and using the Azure Blob Storage as my backend, the credentials I pass to Terraform must be able to manage Virtual Machines, and also to read and write to the Blob Storage. Sometimes this can get complicated, as the same backend name might live in a different account. That’s why most backends also accept their own authentication configuration, to override the default ones that are used by the provider managing the stack’s resources.

More advanced stuff about backends :) 

Dynamically configuring the backend

As we’ve seen above - one of the main limitations of your Terraform backend configuration is that the configuration cannot refer to variables, locals, or data sources. That means that your backend configuration is “hard coded” to whatever is written in your Terraform file. 

It’s very common, and even recommended, to run the same Terraform code for different environments (e.g. staging and prod). But these might live in different cloud accounts, and have different configurations for where we want to store the state. 

There are two options to making your backend configuration dynamic:

• Provide a backend configuration using the CLI. Terraform will accept a -backend-config argument that can be used to run the same stack with different backend configurations. You can also supply that in an environment variable using the TF_CLI_ARGS environment variable.
• Pre-processing your Terraform files. Use a templating engine (such as Jinja or even Sed) to write your Terraform backend Configuration with some placeholders, injecting values into it before you apply it.

It is important to notice that changing the backend configuration after a stack is already active, has implications - see more on that below. Next we’ll be covering how Terraform Workspaces affect the Backend - so you might not need dynamic configuration at all!

Workspaces

Terraform Workspaces are a really important part of scaling your IaC workflow. Workspaces provide a way to use the same code for different environments - so for example - we’d have one workspace for production and one for development. They would both be running the same code, with different configurations, and applied to different cloud accounts. This also means that they would operate in different Terraform states. 

Since we’ll have working directory have a different state for each workspace, they’ll have different state files. The Terraform backend of your choice should support this - as not all backends do. Each Backend has a different implementation to workspace state separation, it’s recommended you look into the backend of terraform project your choosing to understand that implementation, and how you can expect it to look in your storage provider.

Backend Changes & Migrations 

It’s really important to understand that configuration changes to your backend configuration are risky and should be handled with extreme caution. This is because if you do something wrong - you’ll end up with a different or even empty state - and instead of picking up on your existing infrastructure, Terraform will think it’s dealing with different or even new infrastructure. 

Terraform will automatically detect any changes in your configuration and request a reinitialization. The important command line arguments to know are

• -migrate-state will attempt to copy existing state to the new backend, and depending on what changed, may result in interactive prompts to confirm migration of workspace states.
• -reconfigure will disregard any existing configuration.

So if you would like to change your backend to a different one (for example switch the s3 bucket you are using) - you’ll need to - 

1. Apply your stack with the old configuration
2. Update your backend configuration 
3. Init using -migrate-state. 

Terraform will copy the data in the old bucket to the new one, and you’ll be able to pick up where you left off. 

If you are changing state versions the backend and would like to “start fresh” - that is - ignore any previous state and resources - use the -reconfigure option.

Reading the state of another stack

Another great thing that Remote Backends allow us to do is to share the state with other stacks.

Let’s say we’ve got one stack which is configuring the network resources of our cloud account. One of the things that we would create and manage there is a VPC instance. Application stacks we run in that account would like to run within that VPC, and will need to reference it when configuring their resources. 

One way to do that would be to supply the VPC ID as a variable to the application stack. That will mean we have to keep those in sync, and also can bloat our variable’s list, as there is likely more than once dependency. 

This is where the terraform_remote_state data source shines. It allows us to access a state of a different stack, stored in a remote backend. It accepts the same configuration as the original Backend block. The terraform_remote_state data source will only have access to any outputs of the stack we are reading from, so it also allows for cleaner isolation and a more explicit contract between these stacks. 

The “remote” Backend

Among the different types of Terraform backends, there is one type that is different and unique. The “remote” backend differs from the rest, as it not only provides a remote location for Terraform state file storage, it also provides the infrastructure used in CLI-driven runs.   

A CLI-driven run means that although you’ll be running Terraform commands from your terminal, the actual process executing those commands would be on a remote server, such as env0’s runners.

Because a “remote” backend does much more than just storing state - it can’t just point to some infrastructure like S3 - it needs to point at a complete TACO solution such as env0, that provides compute infrastructure as well as storage.

This allows you to utilize a TACO’s benefits, such as team collaboration & cloud access management, but also to run uncommitted code, enabling you to iterate over your work, while actually testing the changes you make to the code.

Conclusion 

In conclusion, configuring and managing Terraform backends is essential for efficient collaboration and state management in infrastructure-as-code projects. By leveraging remote storage options provided by various cloud providers, teams can ensure seamless collaboration without the hassle of synchronizing local state files. 

Terraform backends offer improved team collaboration, state locking, versioning, enhanced security, and access control. Proper backend configuration, infrastructure setup, and permissions are crucial for a smooth workflow. Dynamic configuration options and Terraform workspaces provide flexibility and scalability for different environments. However, it's important to exercise caution when making backend configuration changes to avoid potential disruptions to the state. 

Overall, leveraging Terraform backends empowers teams to streamline their infrastructure management process and maintain a consistent and reliable state across their projects.