Terraform State Locking with S3 & Dynamo DB

"The first year focus would be entirely on building a rock solid infrastructure"

This article aims to shed light on a common unclarity for beginning DevOps teams that plan on setting up their infrastructure through Terraform and version-control, "how do we collaborate?". I will explain how we store our infrastructures snapshots remotely and ensure that no concurrent modifications can be made to our infrastructures snapshots.

Without these snapshots, Terraform has no way to identify the resources it created during previous runs. Thus, when multiple people are collaborating on infrastructure, it's important to store the snapshots in a shared location.

What is a state?

Terraform its core concept is to provision and keep track of resources. Terraform does this through creating a state. The state is cached information about your managed infrastructure and configuration to map your resources to configuration that's keeping track of metadata, and to improve performance for large infrastructures. By default this state is stored in the root of your project named terraform.tfstate, instead of storing the state locally (local backend) it can also be stored remotely (remote backend), which works better in a team environment.

Terraform uses this state to create plans and make change to provisioned resources. Before any operation, Terraform executes a refresh command to reconcile the state Terraform knows about the provisioned resources. This can be used to detect any delta from the last-known state, and to update the state file accordingly.

"The primary purpose of Terraform state is to store bindings between objects in a remote system and resource instances declared in your configuration. When Terraform creates a remote object in response to a change of configuration, it will record the identity of that remote object against a particular resource instance, and then potentially update or delete that object in response to future configuration changes." - Terraform Documentation

For more information about state, visit Purpose of Terraform State.

Remote opposed to local backends

Terraform configurations perform operations backed by backends to store the state snapshot aswell as perform the operations from. Backends provide two areas of behaviour in terraform: Where state is stored and Where operations are performed. The state which we discussed in the previous section should be accessible by every team member if you collaborate on an infrastructure, your team members must have access to the same state data.

The local backend stores state as a local file in the root of your project, but every other backend stores this so called state in a remote storage. Some backends act like plain storage providers; others support locking the state while operations are being performed, which helps prevent conflicts and inconsistencies.

The local backend performs API operations directly from the machine where the terraform command is run. The remote backend can perform API operations remotely,When running remote operations, the local terraform command displays the output of the remote actions as though they were being performed locally.

Putting the resources together

Lets put ourselves to work and setup the remote backend by providing terraform with our remote state storage. Take a look at the below diagram, we are going to setup a global S3 Bucket and a Dynamo DB table. The first step is to make a new directory: $ mkdir terraform-sample and add a backend.tf file to it: $ touch backend.tf, this file will contain a provider block and backend block, don't worry... we will add this now.

Add the below provider block to the backend.tf file and change the profile property accordingly, after you've done that run $ terraform init

s3.tf

Time to add the global S3 Bucket as a remote data storage, Amazon S3 bucket names must be unique globally. Add a new file called s3.tf to our directory, copy the code in the file and change the bucket property accordingly. If you get the "Bucket name already exists" or "BucketAlreadyExists" error, then you must use a different bucket name to create the bucket.

Now that we have the global S3 Bucket in place to store our state files we should look for a way to ensure that no concurrent modifications can be made to the state file. Lets say that colleague A has to add a new Aurora Cluster but colleague B is at the same time adding a new EC2 Instance. This will conflict as only one person can make changes to the state a time.

Terraform can lock a state file to prevent other users from breaking our infrastructure using the same state file at the same time. However, not every backend supports this feature. But S3 supports consistency checking via Dynamo DB.

dynamodb.tf

A single DynamoDB table can be used to lock multiple remote state files. Terraform generates key names that include the values of the bucket and key variables. Lets create our Dynamo DB resource by adding a new file called dynamodb.tf to our directory, copy the code in the file and change the name property accordingly.

All resources are in place and we are ready to provision them through terraform by running: $ terraform plan, this will create an output of what's going to be created. Lets create our resources by running: $ terraform apply

After running the above commands you should have seen that terraform created the resources successfully but a terraform.tfstate file was created locally, this is because we have not told our terraform configuration to make use of the remote backend. Lets do this now by adding the terraform backend block to our backend.tf file.

The key is the path to the state file in the S3 bucket, you can point multiple terraform configurations to this key or provide different keys per terraform configuration and terraform will store the terraform.tfstate under a different path.

Lets now store the state of our newly created backend resources (S3 Bucket and Dynamo DB Table) in the S3 Bucket by running $ terraform init. Terraform will detect that we already have a locally stored state file and asks us to copy over the locally stored state file to our S3 Bucket, type yes.

As you can see, our terraform.tfstate has been stored in the S3 Bucket and is versioned. This allows for rolling-back to a state after something got messed up on accident.

Happy terraforming!

From now on re-use the backend block from backend.tf in your new projects to make use of the remote backend, you can choose to modify the key property in the backend block to store new terraform projects under a different state.


Did you enjoy this read? Feel free to buy me a coffee! :)

Contact me? You can do that through blog@bschaatsbergen.com or LinkedIn.

If you're looking for other articles I recommend you to look in the library.