Advanced Terraform: managing Docker containers¶
Overview¶
This advanced tutorial will guide you through setting up a Terraform project with Docker.
Goals
Understand how to use registry.terraform.io
Understand practical terraform advanced language concepts by orchestrating Docker containers
Understand how to install software and provision multiple VMs
Understand how Terraform is used to orchestrate VMs together
Undertand how Terraform is used to create a multi-node JupyterHub
Things we won't cover
Basic management of Kubernetes clusters (that is a different lesson)
All of Terraform's features
Using registry.terraform.io¶
Terraform maintains an active registry site located at https://registry.terraform.io.
This is the Terraform's defacto catalog for
- providers
- modules
- policies (enforcement rules via Terraform Cloud)
- runtasks (integrations with other services via Terraform Cloud)
Examples of providers:
- https://registry.terraform.io/providers/terraform-provider-openstack/openstack/latest
- https://registry.terraform.io/providers/kreuzwerker/docker/latest
- https://registry.terraform.io/providers/hashicorp/kubernetes/latest
Prerequisites in doing Terraform exercises¶
-
Basic understanding of OpenStack and Terraform as was covered in the prior lesson
-
Access to an OpenStack cloud (we will use Jetstream2)
-
Terraform installed on your workstation or local machine
Outcomes
By the end of this tutorial, you will
have used Terraform to manage Docker containers while learning advanced Terraform concepts
Using a Terraform to simply manage Docker¶
We will use VMs with Docker installed and learn how to launch containers in a declarative way, rather than using the docker command. Once you gain access to your VM, perform the following steps
- ssh into your VM (alternatively, you can declare use the Docker Terraform provider using ssh access)
git clone https://gitlab.com/stack0/terraform-docker-helloworld.gitcd terraform-docker-helloworld-
Review
input.tf,main.tf, andterraform.tfvars.exampleto get a sense of how to manage a Docker container using Terraform- Review the concept of a Terraform
resource - How many resources are created?
- What is the relationship between the resources?
- Is there any question that comes to mind about the port property of
docker_containerresource
- Review the concept of a Terraform
-
cp terraform.tfvars.example terraform.tfvars -
edit
terraform.tfvars- feel free to edit the
imageorcontainer_namewith a container you prefer, but if you do, please select a container with a port that you can access - if you are using local docker access, use the default value; if you are accessing the docker host remotely, use the string "ssh://myuser@1.2.3.4", where
myuseris replaced with your vm username and1.2.3.4is replaced with your vm's ip address
- feel free to edit the
-
terraform apply -auto-approveExpected Response
Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols: + create Terraform will perform the following actions: # docker_container.mycontainer will be created + resource "docker_container" "mycontainer" { + attach = false + bridge = (known after apply) + command = (known after apply) + container_logs = (known after apply) + container_read_refresh_timeout_milliseconds = 15000 + entrypoint = (known after apply) + env = (known after apply) + exit_code = (known after apply) + hostname = (known after apply) + id = (known after apply) + image = (known after apply) + init = (known after apply) + ipc_mode = (known after apply) + log_driver = (known after apply) + logs = false + must_run = true + name = "edwins-container" + network_data = (known after apply) + read_only = false + remove_volumes = true + restart = "no" + rm = false + runtime = (known after apply) + security_opts = (known after apply) + shm_size = (known after apply) + start = true + stdin_open = false + stop_signal = (known after apply) + stop_timeout = (known after apply) + tty = false + wait = false + wait_timeout = 60 } # docker_image.mydocker will be created + resource "docker_image" "mydocker" { + id = (known after apply) + image_id = (known after apply) + name = "nginx:latest" + repo_digest = (known after apply) } Plan: 2 to add, 0 to change, 0 to destroy. docker_image.mydocker: Creating... docker_image.mydocker: Creation complete after 3s [id=sha256:89da1fb6dcb964dd35c3f41b7b93ffc35eaf20bc61f2e1335fea710a18424287nginx:latest] docker_container.mycontainer: Creating... docker_container.mycontainer: Creation complete after 1s [id=54f9987cde7fe5b474348ca3e89955c24eb076d6d4ea49aee7e44928f3f8e711] Apply complete! Resources: 2 added, 0 changed, 0 destroyed. -
terraform destroy -auto-approve
Increasing the number of containers using count¶
We will next update Terraform to create multiple containers of the same image.
-
Modify the
input.tfwith a new variable -
Modify the
main.tfwith the following changes (highlighted lines):resource "docker_container" "mycontainer" { count = var.num_containers image = docker_image.mydocker.image_id name = "${format("%s%02d", var.container_name, count.index)}" ports { internal = 80 external = "${8080 + count.index}" # illustrates using count.index to indicate which port to use } } output "docker_containers" { value = keys({ for index, d in docker_container.mycontainer.* : "${format("%s,%s,%s", index, d.name,d.id)}" => d }) } -
Edit your
terraform.tfvarsto include a newnum_containersinput variable with a value5 -
terraform apply -auto-approveExpected Response
$ terraform apply -auto-approve Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols: + create Terraform will perform the following actions: # docker_container.mycontainer[0] will be created + resource "docker_container" "mycontainer" { + attach = false + bridge = (known after apply) + command = (known after apply) + container_logs = (known after apply) + container_read_refresh_timeout_milliseconds = 15000 + entrypoint = (known after apply) + env = (known after apply) + exit_code = (known after apply) + hostname = (known after apply) + id = (known after apply) + image = (known after apply) + init = (known after apply) + ipc_mode = (known after apply) + log_driver = (known after apply) + logs = false + must_run = true + name = "mycontainer00" + network_data = (known after apply) + read_only = false + remove_volumes = true + restart = "no" + rm = false + runtime = (known after apply) + security_opts = (known after apply) + shm_size = (known after apply) + start = true + stdin_open = false + stop_signal = (known after apply) + stop_timeout = (known after apply) + tty = false + wait = false + wait_timeout = 60 + ports { + external = 8080 + internal = 80 + ip = "0.0.0.0" + protocol = "tcp" } } # docker_container.mycontainer[1] will be created + resource "docker_container" "mycontainer" { + attach = false + bridge = (known after apply) + command = (known after apply) + container_logs = (known after apply) + container_read_refresh_timeout_milliseconds = 15000 + entrypoint = (known after apply) + env = (known after apply) + exit_code = (known after apply) + hostname = (known after apply) + id = (known after apply) + image = (known after apply) + init = (known after apply) + ipc_mode = (known after apply) + log_driver = (known after apply) + logs = false + must_run = true + name = "mycontainer01" + network_data = (known after apply) + read_only = false + remove_volumes = true + restart = "no" + rm = false + runtime = (known after apply) + security_opts = (known after apply) + shm_size = (known after apply) + start = true + stdin_open = false + stop_signal = (known after apply) + stop_timeout = (known after apply) + tty = false + wait = false + wait_timeout = 60 + ports { + external = 8081 + internal = 80 + ip = "0.0.0.0" + protocol = "tcp" } } # docker_container.mycontainer[2] will be created + resource "docker_container" "mycontainer" { + attach = false + bridge = (known after apply) + command = (known after apply) + container_logs = (known after apply) + container_read_refresh_timeout_milliseconds = 15000 + entrypoint = (known after apply) + env = (known after apply) + exit_code = (known after apply) + hostname = (known after apply) + id = (known after apply) + image = (known after apply) + init = (known after apply) + ipc_mode = (known after apply) + log_driver = (known after apply) + logs = false + must_run = true + name = "mycontainer02" + network_data = (known after apply) + read_only = false + remove_volumes = true + restart = "no" + rm = false + runtime = (known after apply) + security_opts = (known after apply) + shm_size = (known after apply) + start = true + stdin_open = false + stop_signal = (known after apply) + stop_timeout = (known after apply) + tty = false + wait = false + wait_timeout = 60 + ports { + external = 8082 + internal = 80 + ip = "0.0.0.0" + protocol = "tcp" } } # docker_container.mycontainer[3] will be created + resource "docker_container" "mycontainer" { + attach = false + bridge = (known after apply) + command = (known after apply) + container_logs = (known after apply) + container_read_refresh_timeout_milliseconds = 15000 + entrypoint = (known after apply) + env = (known after apply) + exit_code = (known after apply) + hostname = (known after apply) + id = (known after apply) + image = (known after apply) + init = (known after apply) + ipc_mode = (known after apply) + log_driver = (known after apply) + logs = false + must_run = true + name = "mycontainer03" + network_data = (known after apply) + read_only = false + remove_volumes = true + restart = "no" + rm = false + runtime = (known after apply) + security_opts = (known after apply) + shm_size = (known after apply) + start = true + stdin_open = false + stop_signal = (known after apply) + stop_timeout = (known after apply) + tty = false + wait = false + wait_timeout = 60 + ports { + external = 8083 + internal = 80 + ip = "0.0.0.0" + protocol = "tcp" } } # docker_container.mycontainer[4] will be created + resource "docker_container" "mycontainer" { + attach = false + bridge = (known after apply) + command = (known after apply) + container_logs = (known after apply) + container_read_refresh_timeout_milliseconds = 15000 + entrypoint = (known after apply) + env = (known after apply) + exit_code = (known after apply) + hostname = (known after apply) + id = (known after apply) + image = (known after apply) + init = (known after apply) + ipc_mode = (known after apply) + log_driver = (known after apply) + logs = false + must_run = true + name = "mycontainer04" + network_data = (known after apply) + read_only = false + remove_volumes = true + restart = "no" + rm = false + runtime = (known after apply) + security_opts = (known after apply) + shm_size = (known after apply) + start = true + stdin_open = false + stop_signal = (known after apply) + stop_timeout = (known after apply) + tty = false + wait = false + wait_timeout = 60 + ports { + external = 8084 + internal = 80 + ip = "0.0.0.0" + protocol = "tcp" } } # docker_image.mydocker will be created + resource "docker_image" "mydocker" { + id = (known after apply) + image_id = (known after apply) + name = "nginx:latest" + repo_digest = (known after apply) } Plan: 6 to add, 0 to change, 0 to destroy. Changes to Outputs: + docker_containers = (known after apply) docker_image.mydocker: Creating... docker_image.mydocker: Creation complete after 3s [id=sha256:89da1fb6dcb964dd35c3f41b7b93ffc35eaf20bc61f2e1335fea710a18424287nginx:latest] docker_container.mycontainer[1]: Creating... docker_container.mycontainer[4]: Creating... docker_container.mycontainer[2]: Creating... docker_container.mycontainer[0]: Creating... docker_container.mycontainer[3]: Creating... docker_container.mycontainer[1]: Creation complete after 1s [id=8c80b558394f62fe004ff156fa1fee06ce723f2b0721cfdfd0f4e64fb6fb28f8] docker_container.mycontainer[3]: Creation complete after 1s [id=c0bc456f847a4e403ee4bcd133f79a21f22b4ec8f94db8793160b28807789e06] docker_container.mycontainer[0]: Creation complete after 1s [id=4ce1d4e0617994ba225701e5e0ef243daa8762968c8d7202be2612ecaeaca064] docker_container.mycontainer[2]: Creation complete after 1s [id=36cf1496a418395408a9d5630c8e1cf8b12fa948ab73fe95a3b1a3d8ce767e99] docker_container.mycontainer[4]: Creation complete after 1s [id=46ab03a8cd2616f20d34058d20b298df8ecc84bf5e8cfaf8013086f21ba1f501] Apply complete! Resources: 6 added, 0 changed, 0 destroyed. Outputs: docker_containers = [ "0,mycontainer00,4ce1d4e0617994ba225701e5e0ef243daa8762968c8d7202be2612ecaeaca064", "1,mycontainer01,8c80b558394f62fe004ff156fa1fee06ce723f2b0721cfdfd0f4e64fb6fb28f8", "2,mycontainer02,36cf1496a418395408a9d5630c8e1cf8b12fa948ab73fe95a3b1a3d8ce767e99", "3,mycontainer03,c0bc456f847a4e403ee4bcd133f79a21f22b4ec8f94db8793160b28807789e06", "4,mycontainer04,46ab03a8cd2616f20d34058d20b298df8ecc84bf5e8cfaf8013086f21ba1f501", ] -
Notice some pieces of the codes that were introduced
- What is the
count.index? - What is the format function and why use it?
- What will index start at?
- Terraform supports math operations
- An example of using the keys() method and ad hoc dictionary construction
- What is the
-
terraform destroy -auto-approve
When resources change outside of Terraform¶
In this exercise we'll discover how to use Terraform to handle change.
-
Use
docker stopanddocker rmto stop and delete docker containers 2 and 3.Expected Response
$ docker stop 36cf1496a418395408a9d5630c8e1cf8b12fa948ab73fe95a3b1a3d8ce767e99 c0bc456f847a4e403ee4bcd133f79a21f22b4ec8f94db8793160b28807789e06 36cf1496a418395408a9d5630c8e1cf8b12fa948ab73fe95a3b1a3d8ce767e99 c0bc456f847a4e403ee4bcd133f79a21f22b4ec8f94db8793160b28807789e06 $ docker rm 36cf1496a418395408a9d5630c8e1cf8b12fa948ab73fe95a3b1a3d8ce767e99 c0bc456f847a4e403ee4bcd133f79a21f22b4ec8f94db8793160b28807789e06 36cf1496a418395408a9d5630c8e1cf8b12fa948ab73fe95a3b1a3d8ce767e99 c0bc456f847a4e403ee4bcd133f79a21f22b4ec8f94db8793160b28807789e06 -
Execute a
terraform showand count the number of instances in the state -
Execute a
terraform refreshExpected Response
$ terraform refresh docker_image.mydocker: Refreshing state... [id=sha256:89da1fb6dcb964dd35c3f41b7b93ffc35eaf20bc61f2e1335fea710a18424287nginx:latest] docker_container.mycontainer[3]: Refreshing state... [id=c0bc456f847a4e403ee4bcd133f79a21f22b4ec8f94db8793160b28807789e06] docker_container.mycontainer[2]: Refreshing state... [id=36cf1496a418395408a9d5630c8e1cf8b12fa948ab73fe95a3b1a3d8ce767e99] docker_container.mycontainer[1]: Refreshing state... [id=8c80b558394f62fe004ff156fa1fee06ce723f2b0721cfdfd0f4e64fb6fb28f8] docker_container.mycontainer[4]: Refreshing state... [id=46ab03a8cd2616f20d34058d20b298df8ecc84bf5e8cfaf8013086f21ba1f501] docker_container.mycontainer[0]: Refreshing state... [id=4ce1d4e0617994ba225701e5e0ef243daa8762968c8d7202be2612ecaeaca064] Outputs: docker_containers = [ "0,mycontainer00,4ce1d4e0617994ba225701e5e0ef243daa8762968c8d7202be2612ecaeaca064", "1,mycontainer01,8c80b558394f62fe004ff156fa1fee06ce723f2b0721cfdfd0f4e64fb6fb28f8", "2,mycontainer02,36cf1496a418395408a9d5630c8e1cf8b12fa948ab73fe95a3b1a3d8ce767e99", "3,mycontainer03,c0bc456f847a4e403ee4bcd133f79a21f22b4ec8f94db8793160b28807789e06", "4,mycontainer04,46ab03a8cd2616f20d34058d20b298df8ecc84bf5e8cfaf8013086f21ba1f501", ] -
Execute a
terraform showagain and recount the instance in the state. Why doesn't the output variable change? -
Execute a
terraform apply(without the-auto-approve) and review what will be updated. Once you are satisfied with the changes that will happen, enteryesat the prompt.Expected Response
$ terraform apply docker_image.mydocker: Refreshing state... [id=sha256:89da1fb6dcb964dd35c3f41b7b93ffc35eaf20bc61f2e1335fea710a18424287nginx:latest] docker_container.mycontainer[0]: Refreshing state... [id=4ce1d4e0617994ba225701e5e0ef243daa8762968c8d7202be2612ecaeaca064] docker_container.mycontainer[1]: Refreshing state... [id=8c80b558394f62fe004ff156fa1fee06ce723f2b0721cfdfd0f4e64fb6fb28f8] docker_container.mycontainer[4]: Refreshing state... [id=46ab03a8cd2616f20d34058d20b298df8ecc84bf5e8cfaf8013086f21ba1f501] Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols: + create Terraform will perform the following actions: # docker_container.mycontainer[2] will be created + resource "docker_container" "mycontainer" { + attach = false + bridge = (known after apply) + command = (known after apply) + container_logs = (known after apply) + container_read_refresh_timeout_milliseconds = 15000 + entrypoint = (known after apply) + env = (known after apply) + exit_code = (known after apply) + hostname = (known after apply) + id = (known after apply) + image = "sha256:89da1fb6dcb964dd35c3f41b7b93ffc35eaf20bc61f2e1335fea710a18424287" + init = (known after apply) + ipc_mode = (known after apply) + log_driver = (known after apply) + logs = false + must_run = true + name = "mycontainer02" + network_data = (known after apply) + read_only = false + remove_volumes = true + restart = "no" + rm = false + runtime = (known after apply) + security_opts = (known after apply) + shm_size = (known after apply) + start = true + stdin_open = false + stop_signal = (known after apply) + stop_timeout = (known after apply) + tty = false + wait = false + wait_timeout = 60 + ports { + external = 8082 + internal = 80 + ip = "0.0.0.0" + protocol = "tcp" } } # docker_container.mycontainer[3] will be created + resource "docker_container" "mycontainer" { + attach = false + bridge = (known after apply) + command = (known after apply) + container_logs = (known after apply) + container_read_refresh_timeout_milliseconds = 15000 + entrypoint = (known after apply) + env = (known after apply) + exit_code = (known after apply) + hostname = (known after apply) + id = (known after apply) + image = "sha256:89da1fb6dcb964dd35c3f41b7b93ffc35eaf20bc61f2e1335fea710a18424287" + init = (known after apply) + ipc_mode = (known after apply) + log_driver = (known after apply) + logs = false + must_run = true + name = "mycontainer03" + network_data = (known after apply) + read_only = false + remove_volumes = true + restart = "no" + rm = false + runtime = (known after apply) + security_opts = (known after apply) + shm_size = (known after apply) + start = true + stdin_open = false + stop_signal = (known after apply) + stop_timeout = (known after apply) + tty = false + wait = false + wait_timeout = 60 + ports { + external = 8083 + internal = 80 + ip = "0.0.0.0" + protocol = "tcp" } } Plan: 2 to add, 0 to change, 0 to destroy. Changes to Outputs: ~ docker_containers = [ - "0,mycontainer00,4ce1d4e0617994ba225701e5e0ef243daa8762968c8d7202be2612ecaeaca064", - "1,mycontainer01,8c80b558394f62fe004ff156fa1fee06ce723f2b0721cfdfd0f4e64fb6fb28f8", - "2,mycontainer02,36cf1496a418395408a9d5630c8e1cf8b12fa948ab73fe95a3b1a3d8ce767e99", - "3,mycontainer03,c0bc456f847a4e403ee4bcd133f79a21f22b4ec8f94db8793160b28807789e06", - "4,mycontainer04,46ab03a8cd2616f20d34058d20b298df8ecc84bf5e8cfaf8013086f21ba1f501", ] -> (known after apply) Do you want to perform these actions? Terraform will perform the actions described above. Only 'yes' will be accepted to approve. Enter a value: -
terraform destroy -auto-approve
Validating your input data¶
Next we will show how to add validation to your inputs
-
In your
input.tf, add the following variable -
In your
main.tf, replace your docker_container resource to the following:resource "docker_container" "mycontainer" { count = var.num_containers image = docker_image.mydocker.image_id name = "${format("%s%02d", var.container_name, count.index)}" ports { internal = 80 external = var.port_assignment_list[count.index] # illustrates using count.index to indicate which port to use } } -
Add/update the following variables in your
terraform.tfvars -
terraform apply -auto-approve - What do you expect?
- Try updating the condition such that length must equal
num_containers
- Try updating the condition such that length must equal
Now, let's figure out how to make our validation slightly more useful
-
If necessary, restore your
input.tfto include the original validation block -
Update your
main.tfto include a validation block like the following:resource "docker_container" "mycontainer" { count = var.num_containers image = docker_image.mydocker.image_id name = "${format("%s%02d", var.container_name, count.index)}" ports { internal = 80 external = var.port_assignment_list[count.index] # illustrates using count.index to indicate which port to use } lifecycle { precondition { condition = length(var.port_assignment_list) == num_containers error_message = "length != num_containers" } } } -
Update your
terraform.tfvarswith the following settings -
terraform apply -auto-approve -
What do you expect?
- Try adding another port (or remove a port) so that
num_containersand the length of the list is equal
- Try adding another port (or remove a port) so that
-
terraform destroy -auto-approve
Increasing the number of containers using for_each¶
Next we will create multiple docker containers, but using a different method, for_each.
-
copy
input.tf,main.tffrom01c-multiple-containers(overwrite your existing files)- review the differences in
input.tf - review the differences in
main.tf
- review the differences in
-
remove the
num_containersfrom yourterraform.tfvarsand add a new variable calledcontainers_map, something like -
terraform apply -auto-approve -
Notice some pieces of the codes that were introduced
- Notice the declaration of a
map, the default value, and how to set it interraform.tfvars - Map keys do not need quoting. What about values?
- How would you compare the how ports are configured in between the use of
countandfor_each - Why might you use
countandfor_each
- Notice the declaration of a
-
terraform destroy -auto-approve
Using dynamic blocks create repeatable nestable blocks in resources¶
Next we will see an example of resource properties that can be repeated
-
Copy
input.tf,main.tffrom02a-using-ports(overwrite your existing files)- Review the differences in
main.tf
- Review the differences in
-
Edit
input.tfwith a new variable -
edit the main.tf with the following
docker_containerresource definition -
Add the following variable in your
terraform.tfvars -
terraform apply -auto-approve - Verify the ports were added when you use
docker ps terraform destroy -auto-approve
Adapted from CyVerse Container Camp (last source update 2023-08-17), CC BY 4.0. Spotted a problem? Open an issue.