Gitlab recently introduced a feature which allows users to store Terraform states in Gitlab via terraform http backend.
Pretty cool, right? However, I am not going to explain how it works. It’s pretty well documented. Instead, I’ll describe my take on integrating the managed state feature with terratest.
The first step first! Let’s define simple terraform manifest along with a very simple terratest test. Don’t mind the simplicity of the test. This time, we will care only about backend configuration which can be used with whatever complexity of infra/tests code.
// main.tf
output "hello_world" {
value = "Hello, World!"
}
// main_test.go
package test
import (
"testing"
"github.com/gruntwork-io/terratest/modules/terraform"
"github.com/stretchr/testify/assert"
)
func TestTerraformHelloWorldExample(t *testing.T) {
terraformOptions := terraform.WithDefaultRetryableErrors(t, &terraform.Options{
TerraformDir: ".",
})
defer terraform.Destroy(t, terraformOptions)
terraform.InitAndApply(t, terraformOptions)
output := terraform.Output(t, terraformOptions, "hello_world")
assert.Equal(t, "Hello, World!", output)
}
Nothing new so far. Let’s verify if we can execute and pass the test.
$ # initialize module to vendor dependencies
$ go mod init test
$ # execute tests
$ go test
TestTerraformHelloWorldExample 2021-02-10T21:45:45+01:00 retry.go:91: terraform [init -upgrade=false]
TestTerraformHelloWorldExample 2021-02-10T21:45:45+01:00 logger.go:66: Running command terraform with args [init -upgrade=false]
TestTerraformHelloWorldExample 2021-02-10T21:45:45+01:00 logger.go:66:
TestTerraformHelloWorldExample 2021-02-10T21:45:45+01:00 logger.go:66: Initializing the backend...
TestTerraformHelloWorldExample 2021-02-10T21:45:45+01:00 logger.go:66:
TestTerraformHelloWorldExample 2021-02-10T21:45:45+01:00 logger.go:66: Initializing provider plugins...
TestTerraformHelloWorldExample 2021-02-10T21:45:45+01:00 logger.go:66:
TestTerraformHelloWorldExample 2021-02-10T21:45:45+01:00 logger.go:66: Terraform has been successfully initialized!
TestTerraformHelloWorldExample 2021-02-10T21:45:45+01:00 logger.go:66:
TestTerraformHelloWorldExample 2021-02-10T21:45:45+01:00 logger.go:66: You may now begin working with Terraform. Try running "terraform plan" to see
TestTerraformHelloWorldExample 2021-02-10T21:45:45+01:00 logger.go:66: any changes that are required for your infrastructure. All Terraform commands
TestTerraformHelloWorldExample 2021-02-10T21:45:45+01:00 logger.go:66: should now work.
TestTerraformHelloWorldExample 2021-02-10T21:45:45+01:00 logger.go:66:
TestTerraformHelloWorldExample 2021-02-10T21:45:45+01:00 logger.go:66: If you ever set or change modules or backend configuration for Terraform,
TestTerraformHelloWorldExample 2021-02-10T21:45:45+01:00 logger.go:66: rerun this command to reinitialize your working directory. If you forget, other
TestTerraformHelloWorldExample 2021-02-10T21:45:45+01:00 logger.go:66: commands will detect it and remind you to do so if necessary.
TestTerraformHelloWorldExample 2021-02-10T21:45:45+01:00 retry.go:91: terraform [get -update]
TestTerraformHelloWorldExample 2021-02-10T21:45:45+01:00 logger.go:66: Running command terraform with args [get -update]
TestTerraformHelloWorldExample 2021-02-10T21:45:46+01:00 retry.go:91: terraform [apply -input=false -auto-approve -lock=false]
TestTerraformHelloWorldExample 2021-02-10T21:45:46+01:00 logger.go:66: Running command terraform with args [apply -input=false -auto-approve -lock=false]
TestTerraformHelloWorldExample 2021-02-10T21:45:46+01:00 logger.go:66:
TestTerraformHelloWorldExample 2021-02-10T21:45:46+01:00 logger.go:66: Apply complete! Resources: 0 added, 0 changed, 0 destroyed.
TestTerraformHelloWorldExample 2021-02-10T21:45:46+01:00 logger.go:66:
TestTerraformHelloWorldExample 2021-02-10T21:45:46+01:00 logger.go:66: Outputs:
TestTerraformHelloWorldExample 2021-02-10T21:45:46+01:00 logger.go:66:
TestTerraformHelloWorldExample 2021-02-10T21:45:46+01:00 logger.go:66: hello_world = Hello, World!
TestTerraformHelloWorldExample 2021-02-10T21:45:46+01:00 retry.go:91: terraform [output -no-color -json hello_world]
TestTerraformHelloWorldExample 2021-02-10T21:45:46+01:00 logger.go:66: Running command terraform with args [output -no-color -json hello_world]
TestTerraformHelloWorldExample 2021-02-10T21:45:47+01:00 logger.go:66: "Hello, World!"
TestTerraformHelloWorldExample 2021-02-10T21:45:47+01:00 retry.go:91: terraform [destroy -auto-approve -input=false -lock=false]
TestTerraformHelloWorldExample 2021-02-10T21:45:47+01:00 logger.go:66: Running command terraform with args [destroy -auto-approve -input=false -lock=false]
TestTerraformHelloWorldExample 2021-02-10T21:45:47+01:00 logger.go:66:
TestTerraformHelloWorldExample 2021-02-10T21:45:47+01:00 logger.go:66: Destroy complete! Resources: 0 destroyed.
PASS
ok test 2.784s
All good!
This works tho. So why do I need to mess around with remote state?!
For a simple reason. Usually, we want to run tests in CI/CD systems to check if terraform code works as expected. The issue with the local state is the case when terratest/terraform fails and leaves hanging resources burning your money. Nobody wants to clean infrastructure menually after every test failure … unless you are masochist.
Therefore it’s a very good idea to persist your state file even in terratest. Even better, persist the state for each branch separately! Standalone state per branch helps to avoid locks and it also allows you to develop multiple features at the same time. And most importantly, you don’t have to worry about failed terraform applies because everything will be refreshed on the next apply.
So let’s modify the code.
As mentioned in the docs, Gitlab State needs to be configured via terraform http backend.
Edit main.tf (or feel free to create another file) and configure the provider.
terraform {
backend "http" {
}
}
As you can see, the backend does not contain any configuration. This is expected as we don’t know gitlab credentials..
and of course, we don’t want them to be static! If you look at the Gitlab guide, you will notice that they use a wrapper
for terraform binary - gitlab-terraform. Maybe we could force terratest to use this wrapper, but that would not be
any fun, wouldn’t it? Moreover, the wrapper is freaking simple so let’s just translate it to Go so we don’t have to
use their image.
Fire up a code editor again, and create identical 1:1 wrapper for backend config.
// just simple helper which allow me to set default value if environment variable is not present.
func getEnv(key, fallback string) string {
if value, ok := os.LookupEnv(key); ok {
return value
}
return fallback
}
// identical copy of https://gitlab.com/gitlab-org/terraform-images/-/blob/ca3883884b857c77f13962d89c20d3748228d2af/src/bin/gitlab-terraform.sh#L64
func gitlabHTTPBackendConfig() map[string]interface{} {
username := getEnv("TF_USERNAME", os.Getenv("GITLAB_USER_LOGIN"))
password := os.Getenv("TF_PASSWORD")
address := os.Getenv("TF_ADDRESS")
// If TF_PASSWORD is unset then default to gitlab-ci-token/CI_JOB_TOKEN
if password == "" {
username = "gitlab-ci-token"
password = os.Getenv("CI_JOB_TOKEN")
}
// let's build address from CI variables if address is not provided
if address == "" {
address = fmt.Sprintf(
"%s/projects/%s/terraform/state/%s",
os.Getenv("CI_API_V4_URL"),
os.Getenv("CI_PROJECT_ID"),
getEnv("TF_STATE_NAME", "terratest"),
)
}
// return backend config
return map[string]interface{}{
"address": getEnv("TF_HTTP_ADDRESS", address),
"lock_address": getEnv("TF_HTTP_LOCK_ADDRESS", fmt.Sprintf("%s/lock", address)),
"lock_method": getEnv("TF_HTTP_LOCK_METHOD", "POST"),
"unlock_address": getEnv("TF_HTTP_UNLOCK_ADDRESS", fmt.Sprintf("%s/lock", address)),
"unlock_method": getEnv("TF_HTTP_UNLOCK_METHOD", "DELETE"),
"username": getEnv("TF_HTTP_USERNAME", username),
"password": getEnv("TF_HTTP_PASSWORD", password),
"retry_wait_min": getEnv("TF_HTTP_RETRY_WAIT_MIN", "5"),
}
}
Yeah, maybe too unnecessary env wrapping but I wanted to have it 100% compatible with the official guide and configuration. Now, let’s add this backend config to terraform options.
terraformOptions := terraform.WithDefaultRetryableErrors(t, &terraform.Options{
TerraformDir: ".",
BackendConfig: gitlabHTTPBackendConfig(),
})
And the last thing missing is CI job so we can actually run this in gitlab runners.
# .gitlab-ci.yml
stages:
- test
terratest:
stage: test
image:
name: golang:alpine
variables:
CGO_ENABLED: 0
# ensure that each branch has it's own terraform state
TF_STATE_NAME: $CI_COMMIT_REF_SLUG
before_script:
- apk add terraform
script:
- go test
Done! Push your code and be happy from green pipelines :)
Note: I don’t recommend to keep long living infra produced by tests. Especially when you are using some nuking tool like cloud-nuke which does periodic infrastructure cleanups.
In next part I’ll describe how to clean up states for merged branches. Stay tuned!