# Container Camp — full corpus Each page below begins with its canonical URL followed by its original Markdown, OKF frontmatter included. Relative links have been rewritten to absolute URLs. ---8<--- https://unm-carc.github.io/container-camp/getting-started/overview/ --- title: "Workshop overview" description: "What CyVerse Container Camp covered in its Basics and Cloud Native tracks, the workshop learning objectives, and CyVerse's NSF funding." type: Guide tags: - Containers - Workshop - Overview generated: by: "claude/opus-5" at: "2026-09-11T00:00:00Z" sources: - id: cyverse-container-camp resource: "https://github.com/CyVerse-learning-materials/container-camp/blob/1746ee8873bcbed2bd3e62500262a9e9571cff16/docs/index.md" title: "CyVerse Container Camp: docs/index.md" author: "team:cyverse" last_modified: "2023-08-04T13:19:03-07:00" --- # Workshop overview [![CyVerse Learning Center](https://unm-carc.github.io/container-camp/assets/de/logos/cyverse_logo_2022.png "CyVerse Learning Center"){width="400"}](https://unm-carc.github.io/cyverse/){target=_blank} ![Container Camp banner](https://unm-carc.github.io/container-camp/assets/cc_banner_01.png) Welcome to our documentation on containers and orchestration in academic research and education. CyVerse ran Container Camp as two paired workshops. In 2023, **Container Camp Basics** met March 6-8 ([agenda](https://unm-carc.github.io/container-camp/getting-started/schedule-basics/)) and the **Cloud Native Camp** met August 16-18 ([agenda](https://unm-carc.github.io/container-camp/getting-started/schedule/)). The schedules are kept as a record of those offerings; the lessons below can be worked through on your own at any time. ## Basic Container Camp ### Docker containers for scientific research - [Introduction to Docker](https://unm-carc.github.io/container-camp/docker/intro/) - [Finding the right container](https://unm-carc.github.io/container-camp/docker/registry/) - [Building Docker images](https://unm-carc.github.io/container-camp/docker/build/) ### Containers on high performance and high throughput computing - [Introduction to Apptainer/Singularity](https://unm-carc.github.io/container-camp/singularity/intro/) - [Building Apptainer/Singularity images](https://unm-carc.github.io/container-camp/singularity/advanced/) - [Apptainer/Singularity on HPC](https://unm-carc.github.io/container-camp/singularity/hpc/) ## Cloud Native Camp ### Orchestration - [Container orchestration with Kubernetes](https://unm-carc.github.io/container-camp/orchestration/kubernetes/) ### Infrastructure as Code - [Infrastructure as Code with HashiCorp Terraform](https://unm-carc.github.io/container-camp/orchestration/terraform/) - [CyVerse's Cloud Automation & Continuous Analysis Orchestration (CACAO)](https://unm-carc.github.io/container-camp/orchestration/cacao/) !!! success "Learning Objectives" After completing these two paired workshops, you should be able to: * Explain why containers and orchestration are used in research computing * Create your own containers and deploy your own orchestrated frameworks * Understand how and when to use containers and IaC in your daily work You will also leave with - Understanding of how to use the most powerful public research computing infrastructure in the world via [ACCESS-CI](https://access-ci.org){target=_blank}. - The ability to launch and manage distributed resources using IaC templates on commercial (AWS, GCloud, Azure) or public research (OpenStack) clouds. - Insight into commercial cloud services, their costs, and how to best utilize them for scientific research. --- **Funding and Citations:** CyVerse is generously funded by the National Science Foundation [![NSF](https://unm-carc.github.io/container-camp/assets/nsf.png){width="25"}](https://nsf.gov){target=_blank}. Here are our award numbers: - [![NSF-0735191](https://img.shields.io/badge/NSF-0735191-blue.svg){target=_blank}](https://www.nsf.gov/awardsearch/showAward?AWD_ID=0735191) - [![NSF-1265383](https://img.shields.io/badge/NSF-1265383-blue.svg){target=_blank}](https://www.nsf.gov/awardsearch/showAward?AWD_ID=1265383) - [![NSF-1743442](https://img.shields.io/badge/NSF-1743442-blue.svg){target=_blank}](https://www.nsf.gov/awardsearch/showAward?AWD_ID=1743442) For any utilization of our resources, please adhere to the [CyVerse citation policy](https://cyverse.org/policies/cite-cyverse){target=_blank}.

Adapted from [CyVerse Container Camp](https://github.com/CyVerse-learning-materials/container-camp/blob/1746ee8873bcbed2bd3e62500262a9e9571cff16/docs/index.md){target=_blank} (last source update 2023-08-04), CC BY 4.0. Spotted a problem? [Open an issue](https://github.com/UNM-CARC/container-camp/issues){target=_blank}.

---8<--- https://unm-carc.github.io/container-camp/getting-started/code-of-conduct/ --- title: "Code of Conduct" description: "The anti-harassment code of conduct that applied to all Container Camp attendees, speakers, staff, and volunteers." type: Policy tags: - Code of Conduct - Workshop generated: by: "claude/opus-5" at: "2026-09-11T00:00:00Z" sources: - id: cyverse-container-camp resource: "https://github.com/CyVerse-learning-materials/container-camp/blob/1746ee8873bcbed2bd3e62500262a9e9571cff16/docs/getting_started/code_conduct.md" title: "CyVerse Container Camp: docs/getting_started/code_conduct.md" author: "team:cyverse" last_modified: "2022-04-20T19:25:56Z" --- # Code of Conduct All attendees, speakers, staff and volunteers at Container Camp are required to follow our code of conduct. CyVerse expects and appreciates cooperation from all participants to help ensure a safe, collaborative environment for everyone. Harrassment by any individual will not be tolerated and may result in the individual being removed from the Camp. Harassment includes: offensive verbal comments related to gender, gender identity and expression, age, sexual orientation, disability, physical appearance, body size, race, ethnicity, religion, technology choices, sexual images in public spaces, deliberate intimidation, stalking, following, harassing photography or recording, sustained disruption of talks or other events, inappropriate physical contact, and unwelcome sexual attention. Participants who are asked to stop any harassing behavior are expected to comply immediately. Workshop staff are also subject to the anti-harassment policy. In particular, staff should not use sexualised images, activities, or other material. If a participant engages in harassing behavior, the workshop organisers may take any action they deem appropriate, including warning the offender or expulsion from the workshop with no refund. If you are being harassed, or notice that someone else is being harassed, or have any other concerns, please contact a member of the workshop staff immediately. Staff can be identified as they'll be wearing badges or nametags. Workshop staff will be happy to help participants contact local law enforcement, provide escorts, or otherwise assist those experiencing harassment to feel safe for the duration of the workshop. We value your attendance. We expect participants to follow these rules at conference and workshop venues and conference-related social events. See http://www.ashedryden.com/blog/codes-of-conduct-101-faq for more information on codes of conduct.

Adapted from [CyVerse Container Camp](https://github.com/CyVerse-learning-materials/container-camp/blob/1746ee8873bcbed2bd3e62500262a9e9571cff16/docs/getting_started/code_conduct.md){target=_blank} (last source update 2022-04-20), CC BY 4.0. Spotted a problem? [Open an issue](https://github.com/UNM-CARC/container-camp/issues){target=_blank}.

---8<--- https://unm-carc.github.io/container-camp/getting-started/setup/ --- title: "Pre-camp setup" description: "Accounts to create before starting the lessons: GitHub and Docker Hub, plus optional CyVerse and NSF ACCESS accounts." type: Guide tags: - Setup - Accounts generated: by: "claude/opus-5" at: "2026-09-11T00:00:00Z" sources: - id: cyverse-container-camp resource: "https://github.com/CyVerse-learning-materials/container-camp/blob/1746ee8873bcbed2bd3e62500262a9e9571cff16/docs/getting_started/setup.md" title: "CyVerse Container Camp: docs/getting_started/setup.md" author: "team:cyverse" last_modified: "2023-07-28T08:51:15-07:00" --- # Pre-camp setup ## :material-clipboard-check-multiple: Prerequisites Before attending Container Camp (Basics) please do the following: [Create a GitHub Account](https://github.com){target=_blank} [Create a Docker Hub Account](https://hub.docker.com){target=_blank} **Optional** [Create a CyVerse Account](https://user.cyverse.org){target=_blank} Before attending Cloud Native Camp please do the following: :fontawesome-regular-hand-point-up: **(all of the above)** [Create an ACCESS-CI profile](https://access-ci.org/){target=_blank}

Adapted from [CyVerse Container Camp](https://github.com/CyVerse-learning-materials/container-camp/blob/1746ee8873bcbed2bd3e62500262a9e9571cff16/docs/getting_started/setup.md){target=_blank} (last source update 2023-07-28), CC BY 4.0. Spotted a problem? [Open an issue](https://github.com/UNM-CARC/container-camp/issues){target=_blank}.

---8<--- https://unm-carc.github.io/container-camp/getting-started/schedule-basics/ --- title: "Container Camp Basics schedule (March 2023)" description: "Day-by-day agenda of the March 6-8, 2023 Container Camp Basics: Docker, Docker Compose, and Apptainer/Singularity." type: Schedule tags: - Schedule - Docker - Apptainer stale_after: "2023-03-09T00:00:00Z" generated: by: "claude/opus-5" at: "2026-09-11T00:00:00Z" sources: - id: cyverse-container-camp resource: "https://github.com/CyVerse-learning-materials/container-camp/blob/1746ee8873bcbed2bd3e62500262a9e9571cff16/docs/getting_started/schedule_basics.md" title: "CyVerse Container Camp: docs/getting_started/schedule_basics.md" author: "team:cyverse" last_modified: "2023-07-27T11:00:45-07:00" --- # Container Camp Basics schedule (March 2023) This page records the agenda of the 2023 Container Camp Basics, held March 6-8, 2023 (times are Mountain Standard Time, Arizona). The session times and instructor assignments are historical; the linked lessons remain available to work through on your own. ## Day 1 - Introduction to Docker ??? note "Content" - Introduction to Docker and its uses in reproducible science - Launching development environments on CodeSpaces for container testing - Using Docker on the commandline. ??? success "Goals" - Introduction to containers & [where to find them](https://unm-carc.github.io/container-camp/docker/registry/) - Command line containers with CodeSpaces (optional: run locally) - Find and use official Docker images ???+ example "Activities" | Time (MST/AZ)| Activity | Instructor | Outcome | |-----------|----------|------------|-------| | 09:00 | Welcome | All | | 09:05 | Overview of Website | Michele | [Code of Conduct](https://unm-carc.github.io/container-camp/getting-started/code-of-conduct/) | | 09:10 | Why use Containers? | Carlos | What containers are used for in science | | 09:25 | [Start CodeSpace :material-microsoft-visual-studio-code: :material-github:](https://unm-carc.github.io/container-camp/cloud/codespaces/) | Tyson | Using Dev Environments to create containers| | 09:30 | [:material-docker: Docker Commands and Execution](https://unm-carc.github.io/container-camp/docker/intro/#fundamental-docker-commands) | Carlos | Basic command line use of Docker | | 09:55 | Break | | | 10:00 | [:material-docker: Docker Commands and Execution](https://unm-carc.github.io/container-camp/docker/intro/#fundamental-docker-commands) | Carlos & Tyson | Basic command line use of Docker | | 10:55 | Break | | | 11:00 | [:material-docker: Managing Docker and Data](https://unm-carc.github.io/container-camp/docker/intro/#managing-data-in-docker) | Michele | Volumes and Interactive Use inside containers | | 11:55 | Break | | | | 12:00 | [:material-docker: Managing Docker and Data](https://unm-carc.github.io/container-camp/docker/intro/) | Michele & Tyson | | | 12:55 | Conclude for the day | All | push changes to your GitHub | ??? warning "Optional Homework" - Test other Docker container images on CodeSpaces or locally ## Day 2 - Building Docker Containers ??? note "Content" - Use GitHub to browse for public Dockerfiles - Build Dockerfiles and push them to public registry - Use Version Control to set up automated container builds with GitHub Actions ??? success "Goals" - Introduction to what Dockerfiles are and what you use them for - Start thinking about how to modify them for your own applications ???+ example "Activities" | Time (MST/AZ)| Activity | Instructor | Notes | |-----------|----------|------------|-------| | 09:00 | Welcome back | All | | | 09:05 | Discuss previous day, answer questions | | | | 09:15 | [(re)start Dev Environment :material-microsoft-visual-studio-code: :material-github:](https://unm-carc.github.io/container-camp/cloud/codespaces/) | Tyson | | | 09:20 | [Finding the right container :material-docker:](https://unm-carc.github.io/container-camp/docker/registry/) | Tyson | | | 09:55 | Break | | | 10:00 | [Building Docker Images :material-docker:](https://unm-carc.github.io/container-camp/docker/build/) | Carlos | | | 10:55 | Break | | | 11:00 | [Using Docker Compose :material-docker:](https://unm-carc.github.io/container-camp/docker/compose/) | Michele | | | 11:55 | Break | | | | 12:00 | [Integrating your Containers into CyVerse](https://unm-carc.github.io/cyverse/discovery-environment/create-apps/){target=_blank} | Tyson | | | 12:55 | Conclude | | | ## Day 3 - Singularity, Orchestration, and Containers on the HPC ??? note "Content" - Introduction to Singularity - Introduction to Kubernetes ??? success "Goals" - Being able to execute containers on the HPC - Create a small Kubernetes cluster ???+ example "Activities" | Time (MST/AZ)| Activity | Instructor | Notes | |-----------|----------|------------|-------| | 09:00 | Welcome back | All | | | 09:05 | Discuss previous day, answer questions | | | | 09:15 | [(re(re))start Dev Environment :material-microsoft-visual-studio-code: :material-github:](https://unm-carc.github.io/container-camp/cloud/codespaces/) | | | | 09:20 | [Introduction to Singularity](https://unm-carc.github.io/container-camp/singularity/intro/) | | | 09:30 | [Obtaining Singularity](https://unm-carc.github.io/container-camp/singularity/intro/#installing-apptainer-or-singularityce) and [the Singularity CLI](https://unm-carc.github.io/container-camp/singularity/intro/#apptainer-and-singularity-cli) | | | 09:55 | Break | | | 10:00 | [Singularity Commands & Interacting with Singularity Images](https://unm-carc.github.io/container-camp/singularity/intro/#interact-with-images) | | | 10:55 | Break || | 11:00 | [Singularity Images Continued](https://unm-carc.github.io/container-camp/singularity/advanced/#apptainer-and-singularity-cli-continued)* | | | 11:55 | Break || | 12:00 | [Integrating your Containers into CyVerse](https://unm-carc.github.io/cyverse/discovery-environment/create-apps/){target=_blank}, questions, inquiries | | | 12:55 | Conclude** | | \* If we still have time, we will also discuss using [Singularity on the HPC](https://unm-carc.github.io/container-camp/singularity/hpc/#how-do-hpc-systems-fit-into-the-development-workflow). \*\* For additional questions and inquiries, we will make time for one-on-ones on either Thursday or Friday (flexible schedule).

Adapted from [CyVerse Container Camp](https://github.com/CyVerse-learning-materials/container-camp/blob/1746ee8873bcbed2bd3e62500262a9e9571cff16/docs/getting_started/schedule_basics.md){target=_blank} (last source update 2023-07-27), CC BY 4.0. Spotted a problem? [Open an issue](https://github.com/UNM-CARC/container-camp/issues){target=_blank}.

---8<--- https://unm-carc.github.io/container-camp/getting-started/schedule/ --- title: "Cloud Native Camp schedule (August 2023)" description: "Day-by-day agenda of the August 16-18, 2023 Cloud Native Camp: Kubernetes, Terraform, and CyVerse CACAO." type: Schedule tags: - Schedule - Kubernetes - Terraform - CACAO stale_after: "2023-08-19T00:00:00Z" generated: by: "claude/opus-5" at: "2026-09-11T00:00:00Z" sources: - id: cyverse-container-camp resource: "https://github.com/CyVerse-learning-materials/container-camp/blob/1746ee8873bcbed2bd3e62500262a9e9571cff16/docs/getting_started/schedule.md" title: "CyVerse Container Camp: docs/getting_started/schedule.md" author: "team:cyverse" last_modified: "2023-08-17T01:01:22Z" --- # Cloud Native Camp schedule (August 2023) This page records the agenda of the 2023 Cloud Native Camp, held August 16-18, 2023 (times are Pacific Daylight Time). The session times and instructor assignments are historical; the linked lessons remain available to work through on your own. ## Day 1 - Container Orchestration !!! abstract "Content" * Introduction to Kubernetes * K8s CLI * Deploying K8s clusters !!! success "Learning Objectives" * Comfort working in the terminal with a remote K8s cluster * Understanding of K8s as it is used to manage containers * Start your own Zero2JupyterHub !!! example "Agenda" | Time ([PDT](https://time.is/PDT){target=_blank}) | Activity | Instructor | Outcome | |-----------|----------|------------|-------| | 09:00 | Welcome | All | | 09:05 | Overview of Website | Tyson | [Code of Conduct](https://unm-carc.github.io/container-camp/getting-started/code-of-conduct/) | | 09:10 | What is Kubernetes? | Tyson | overview of K8s ecosystem | | 09:30 | [:simple-kubernetes: K8s CLI](https://unm-carc.github.io/container-camp/orchestration/kubernetes/#k8s-cli-kubectl) | Tyson | Basic command line use of K8s | | 09:55 | Break | | | 10:00 | [:simple-kubernetes: K8s CLI cont.](https://unm-carc.github.io/container-camp/orchestration/kubernetes/#k8s-cli-kubectl) | Tyson & Michele | Connecting to a K8s cluster | | 10:55 | Break | | | 11:00 | [:material-docker: Deploying K8s clusters](https://unm-carc.github.io/container-camp/orchestration/kubernetes-advanced/) | Michele | Zero2JupyterHub | | 11:55 | Break | | | | 12:00 | [:simple-kubernetes: Managing K8s clusters](https://unm-carc.github.io/container-camp/orchestration/kubernetes-advanced/) | Michele & Tyson | | | 12:55 | Conclude for the day | All | push changes to GitHub | ??? warning "Homework" * Consider what type of hardware & container orchestration you're most interested in and come ready with your own ideas for deploying something with Terraform on Day 2. ## Day 2 - Infrastructure as Code !!! abstract "Content" * Use Terraform to provision hardware and deploy applications & containers * Build Terraform templates !!! success "Learning Objectives" * Understanding of what IaC is and how it fits along with the orchestration of containers and cloud * Ability to launch your own cloud instances on at least one cloud platform (OpenStack) * Ability to provision your instances with Terraform and access them !!! example "Agenda" | Time ([PDT](https://time.is/PDT){target=_blank}) | Activity | Instructor | Outcome | |-----------|----------|------------|-------| | 09:00 | Welcome back | All | | | 09:05 | Discuss previous day, answer questions | | | | 09:20 | [:simple-terraform: What is Terraform](https://unm-carc.github.io/container-camp/orchestration/terraform/) | Edwin & Tyson | | | 09:55 | Break | | | 10:00 | [:simple-terraform: Terraform CLI](https://unm-carc.github.io/container-camp/orchestration/terraform/) | Tyson | | | 10:55 | Break | | | 11:00 | [:simple-terraform: Using a Terraform to simply manage Docker](https://unm-carc.github.io/container-camp/orchestration/terraform-advanced/#using-a-terraform-to-simply-manage-docker) | Edwin & Michele | | | 11:55 | Break | | | | 12:00 | [:simple-terraform: Terraform management continued](https://unm-carc.github.io/container-camp/orchestration/terraform-advanced/) | Edwin & Tyson | | | 12:55 | Conclude | | | ## Day 3 - CACAO !!! abstract "Content" - Introduction to CyVerse CACAO - Writing advanced Terraform templates for CACAO !!! success "Learning Objectives" - Being able to execute containers on the HPC - Create a small Kubernetes cluster !!! example "Agenda" | Time ([PDT](https://time.is/PDT){target=_blank}) | Activity | Instructor | Outcome | |-----------|----------|------------|-------| | 09:00 | Welcome back | All | | | 09:05 | Discuss previous day, answer questions | | | | 09:20 | [Introduction to CACAO](https://unm-carc.github.io/container-camp/orchestration/cacao/) | Edwin | | 09:30 | [Using the CACAO UI](https://unm-carc.github.io/container-camp/orchestration/cacao/) | Edwin | | 09:55 | Break | | | 10:00 | [Overview of CACAO templates](https://unm-carc.github.io/container-camp/orchestration/cacao-terraform/)| Edwin | | 10:55 | Break || | 11:00 | [Creating a CACAO template](https://unm-carc.github.io/container-camp/orchestration/cacao-terraform/) | Edwin | | 11:55 | Break || | 12:00 | Show and Tell - present your use cases | everyone | | 12:55 | Conclude | |

Adapted from [CyVerse Container Camp](https://github.com/CyVerse-learning-materials/container-camp/blob/1746ee8873bcbed2bd3e62500262a9e9571cff16/docs/getting_started/schedule.md){target=_blank} (last source update 2023-08-17), CC BY 4.0. Spotted a problem? [Open an issue](https://github.com/UNM-CARC/container-camp/issues){target=_blank}.

---8<--- https://unm-carc.github.io/container-camp/docker/intro/ --- title: "Introduction to Docker" description: "Run, inspect, and clean up Docker containers from the command line, mount data with volumes, and expose ports for browser-based IDEs." type: Lesson tags: - Docker - Containers - Command Line generated: by: "claude/opus-5" at: "2026-09-11T00:00:00Z" sources: - id: cyverse-container-camp resource: "https://github.com/CyVerse-learning-materials/container-camp/blob/1746ee8873bcbed2bd3e62500262a9e9571cff16/docs/docker/intro.md" title: "CyVerse Container Camp: docs/docker/intro.md" author: "team:cyverse" last_modified: "2023-08-15T20:54:48-07:00" --- # Introduction to Docker [![docker](https://unm-carc.github.io/container-camp/assets/docker/docker.png){ width="400" }](https://hub.docker.com){target=_blank} ### Prerequisites In order to complete these exercises we **STRONGLY** recommend that you set up a personal [:material-github: GitHub](https://github.com){target=_blank} and [:material-docker: DockerHub](https://hub.docker.com){target=_blank} account (account creation for both services is free). There are no specific skills needed for this tutorial beyond elementary command line ability and using a text editor. We are going to be using [:material-github: GitHub CodeSpaces](https://github.com/features/codespaces){target=_blank} for the hands on portion of the workshop, which features [:material-microsoft-visual-studio-code: VS Code](https://code.visualstudio.com/){target=_blank} as a fully enabled development environment with Docker already installed. CodeSpaces is a featured product from GitHub and requires a paid subscription or Academic account for access. Your account will temporarily be integrated with the course GitHub Organization for the next steps in the workshop. Our instructions on starting a new CodeSpace are [here](https://unm-carc.github.io/container-camp/cloud/codespaces/). ??? info "Installing Docker on your personal computer" We are going to be using virtual machines on the cloud for this course, and we will explain why this is a good thing, but there may be a time when you want to run Docker on your own computer. Installing Docker takes a little time but it is reasonably straight forward and it is a one-time setup. Installation instructions from Docker Official Docs for common OS and chip architectures: - [:fontawesome-brands-apple: Mac OS X](https://docs.docker.com/docker-for-mac/){target=_blank} - [:fontawesome-brands-windows: Windows](https://docs.docker.com/docker-for-windows){target=_blank} - [:fontawesome-brands-ubuntu: Ubuntu Linux](https://docs.docker.com/install/linux/docker-ce/ubuntu/){target=_blank} ??? failure "Never used a terminal before?" That is 💯 OK! (This person never used a terminal until after their terminal degree, and now they actually PREFER to work in it for writing code) Don't be afraid or ashamed, but be ready to learn some new skills -- we promise it will be worth your while and even FUN! Before venturing much further, you should review the [Software Carpentry](https://software-carpentry.org/lessons/){target=_blank} lessons on "The Unix Shell" and "Version Control with Git" -- these are great introductory lessons related to the skills we're teaching here. You've given up on ever using a terminal? No problem, Docker can be used from graphic interfaces, like [Docker Desktop](https://www.docker.com/products/docker-desktop/){target=_blank}, or platforms like [Portainer](https://www.portainer.io/){target=_blank}. We suggest you read through their documentation on how to use Docker. ## Fundamental Docker Commands :octicons-terminal-16: Docker commands in the terminal use the prefix `docker`. !!! note For every command listed, the correct execution of the commands through the command line is by using `docker` in front of the command: for example `docker help` or `docker search`. Thus, every :material-docker: = `docker`. ### :material-docker: help Like many other command line applications the most helpful flag is the `help` command which can be used with the Management Commands: ``` $ docker $ docker --help ``` ### :material-docker: search We talk about the concept of [Docker Registries](https://unm-carc.github.io/container-camp/docker/registry/) in the next section, but you can search the public list of registeries by using the `docker search` command to find public containers on the Official [Docker Hub Registry](https://hub.docker.com){target=_blank}: ``` $ docker search ``` ### :material-docker: pull Go to the [Docker Hub](https://hub.docker.com){target=_blank} and type `hello-world` in the search bar at the top of the page. Click on the 'tag' tab to see all the available 'hello-world' images. Click the 'copy' icon at the right to copy the `docker pull` command, or type it into your terminal: ``` $ docker pull hello-world ``` !!! note If you leave off the `:` and the tag name, it will by default pull the `latest` image ``` $ docker pull hello-world Using default tag: latest latest: Pulling from library/hello-world 2db29710123e: Pull complete Digest: sha256:bfea6278a0a267fad2634554f4f0c6f31981eea41c553fdf5a83e95a41d40c38 Status: Downloaded newer image for hello-world:latest docker.io/library/hello-world:latest ``` Now try to list the files in your current working directory: ``` $ ls -l ``` ??? question "Where is the image you just pulled?" Docker saves container images to the Docker directory (where Docker is installed). You won't ever see them in your working directory. Use 'docker images' to see all the images on your computer: ``` $ docker images ``` ??? tip "adding yourself to the Docker group on Linux" Depending on how and where you've installed Docker, you may see a `permission denied` error after running `$ docker run hello-world` command. If you're on Linux, you may need to prefix your Docker commands with `sudo`. Alternatively to run docker command without `sudo`, you need to add your user name (who has root privileges) to the docker "group". Create the docker group: ``` $ sudo groupadd docker ``` Add your user to the docker group:: ``` $ sudo usermod -aG docker $USER ``` Log out or close terminal and log back in and your group membership will be initiated ### :material-docker: run The single most common command that you'll use with Docker is `docker run` ([see official help manual](https://docs.docker.com/engine/reference/commandline/run/){target=_blank} for more details). `docker run` starts a container and executes the default "entrypoint", or any other "command" that follows `run` and any optional flags. ??? tip "What is an *entrypoint*?" An entrypoint is the initial command(s) executed upon starting the Docker container. It is listed in the `Dockerfile` as `ENTRYPOINT` and can take 2 forms: as commands followed by parameters (`ENTRYPOINT command param1 param2`) or as an executable (`ENTRYPOINT [“executable”, “param1”, “param2”]`) ``` $ docker run hello-world:latest ``` In the demo above, you used the `docker pull` command to download the `hello-world:latest` image. What about if you run a container that you haven't downloaded? ``` $ docker run alpine:latest ls -l ``` When you executed the command `docker run alpine:latest`, Docker first looked for the cached image locally, but did not find it, it then ran a `docker pull` behind the scenes to download the `alpine:latest` image and then execute your command. When you ran `docker run alpine:latest`, you provided a command `ls -l`, so Docker started the command specified and you saw the listing of the Alpine file system (not your host system, this was insice the container!). ### :material-docker: images You can now use the `docker images` command to see a list of all the cached images on your system: ``` $ docker images REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE alpine latest c51f86c28340 4 weeks ago 1.109 MB hello-world latest 690ed74de00f 5 months ago 960 B ``` ??? info "Inspecting your containers" To find out more about a Docker images, run `docker inspect hello-world:latest` ### :material-docker: ps Now it's time to see the `docker ps` command which shows you all containers that are currently running on your machine. ``` docker ps ``` Since no containers are running, you see a blank line. ``` $ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES ``` Let's try a more useful variant: `docker ps --all` ``` $ docker ps --all CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES a5eab9243a15 hello-world "/hello" 5 seconds ago Exited (0) 3 seconds ago loving_mcnulty 3bb4e26d2e0c alpine:latest "/bin/sh" 17 seconds ago Exited (0) 16 seconds ago objective_meninsky 192ffdf0cbae opensearchproject/opensearch-dashboards:latest "./opensearch-dashbo…" 3 days ago Exited (0) 3 days ago opensearch-dashboards a10d47d3b6de opensearchproject/opensearch:latest "./opensearch-docker…" 3 days ago Exited (0) 3 days ago opensearch-node1 ``` What you see above is a list of all containers that you have run. Notice that the `STATUS` column shows the current condition of the container: running, or as shown in the example, when the container was exited. ### :material-docker: stop The `stop` command is used for containers that are actively running, either as a foreground process or as a detached background one. You can find a running container using the `docker ps` command. ### :material-docker: rm You can remove individual stopped containers by using the `rm` command. Use the `ps` command to see all your stopped contiainers: ``` @user ➜ /workspaces $ docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 03542eaac9dc hello-world "/hello" About a minute ago Exited (0) About a minute ago unruffled_nobel ``` Use the first few unique alphanumerics in the CONTAINER ID to remove the stopped container: ``` @user ➜ /workspaces (mkdocs ✗) $ docker rm 0354 0354 ``` Check to see that the container is gone using `ps -a` a second time (`-a` is shorthand for `--all`; the full command is `docker ps -a` or `docker ps --all`). ### :material-docker: rmi The `rmi` command is similar to `rm` but it will remove the cached images. Used in combination with `docker images` or `docker system df` you can clean up a full cache ``` docker rmi ``` ``` @user ➜ /workspaces/ (mkdocs ✗) $ docker images REPOSITORY TAG IMAGE ID CREATED SIZE opendronemap/webodm_webapp latest e075d13aaf35 21 hours ago 1.62GB redis latest a10f849e1540 5 days ago 117MB opendronemap/nodeodm latest b4c50165f838 6 days ago 1.77GB hello-world latest feb5d9fea6a5 7 months ago 13.3kB opendronemap/webodm_db latest e40c0f274bba 8 months ago 695MB @user ➜ /workspaces (mkdocs ✗) $ docker rmi hello-world Untagged: hello-world:latest Untagged: hello-world@sha256:10d7d58d5ebd2a652f4d93fdd86da8f265f5318c6a73cc5b6a9798ff6d2b2e67 Deleted: sha256:feb5d9fea6a5e9606aa995e879d862b825965ba48de054caab5ef356dc6b3412 Deleted: sha256:e07ee1baac5fae6a26f30cabfe54a36d3402f96afda318fe0a96cec4ca393359 @user ➜ /workspaces (mkdocs ✗) $ ``` ### :material-docker: system The `system` command can be used to view information about containers on your cache, you can view your total disk usage, view events or info. You can also use it to `prune` unused data and image layers. To remove all cached layers, images, and data you can use the `-af` flag for `all` and `force` ``` docker system prune -af ``` ### :material-docker: tag By default an image will recieve the tag `latest` when it is not specified during the `docker build` Image names and tags can be created or changed using the `docker tag` command. ``` docker tag imagename:oldtag imagename:newtag ``` You can also change the registry name used in the tag: ``` docker tag docker.io/username/imagename:oldtag harbor.cyverse.org/project/imagename:newtag ``` The cached image laters will not change their `sha256` and both image tags will still be present after the new tag name is generated. ### :material-docker: push By default `docker push` will upload your local container image to the [Docker Hub](https://hub.docker.com){target=_blank} We will cover `push` in more detail at the end of Day 2, but the essential functionality is the same as pull. Also, make sure that your container has the appropriate [tag](https://unm-carc.github.io/container-camp/docker/intro/#tag) First, make sure to log into the Docker Hub, this will allow you to download private limages, to upload private/public images: ``` docker login ``` Alternately, you can [link GitHub / GitLab accounts](https://hub.docker.com/settings/linked-accounts){target=_blank} to the Docker Hub. To push the image to the Docker Hub: ``` docker push username/imagename:tag ``` or ``` docker push docker.io/username/imagename:tag ``` or, to a private registry, here we push to CyVerse private `harbor.cyverse.org` registry which uses "project" sub folders: ``` docker push harbor.cyverse.org/project/imagename:newtag ``` --- ## Commands & Entrypoints We will cover the differences in `CMD` and `ENTRYPOINT` on Day 2 when we build our own images, but it is important to understand that a container can have a command appended to the `docker run` function. When a image has no commands or entrypoints specified in its Dockerfile, it will default to running a `/bin/sh` syntax. In those cases, you can add a command when the congtainer is run: ``` $ docker run alpine echo "Hello world" ``` the Docker client dutifully ran the ``echo`` command in our ``alpine`` container and then exited. If you've noticed, all of that happened pretty quickly. Imagine booting up a virtual machine, running a command and then killing it. Now you know why they say containers are fast! --- ## Interactive Commands with Containers Lets try another command, this time to access the container as a shell: ``` $ docker run alpine:latest sh ``` Wait, nothing happened, right? Is that a bug? Well, no. The container will exit after running any scripted commands such as `sh`, unless they are run in an "interactive" terminal (TTY) - so for this example to not exit, you need to add the `-i` for interactive and `-t` for TTY. You can run them both in a single flag as ``-it``, which is the more common way of adding the flag: ``` $ docker run -it alpine:latest sh ``` The prompt should change to something more like `/ # `. You are now running a shell inside the container! Try out a few commands like `ls -l`, `uname -a` and others. Exit out of the container by giving the `exit` command. ``` / # exit ``` ??? warning "Making sure you've exited the container" If you type `exit` your **container** will exit and is no longer active. To check that, try the following: ``` $ docker ps --latest CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES de4bbc3eeaec alpine "/bin/sh" 3 minutes ago Exited (0) About a minute ago pensive_leavitt ``` If you want to keep the container active, then you can use keys `ctrl +p` `ctrl +q`. To make sure that it is not exited run the same `docker ps --latest` command again: ``` $ docker ps --latest CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 0db38ea51a48 alpine "sh" 3 minutes ago Up 3 minutes elastic_lewin ``` Now if you want to get back into that container, then you can type `docker attach `. This way you can save your container: ``` $ docker attach 0db38ea51a48 ``` --- ## :material-home: House Keeping and :material-broom: Cleaning Up Exited Containers ### Managing Docker Images In the previous example, you pulled the `alpine` image from the registry and asked the Docker client to run a container based on that image. To see the list of images that are available locally on your system, run the `docker images` command. ``` $ docker images REPOSITORY TAG IMAGE ID CREATED SIZE ubuntu bionic 47b19964fb50 4 weeks ago 88.1MB alpine latest caf27325b298 4 weeks ago 5.53MB hello-world latest fce289e99eb9 2 months ago 1.84kB ``` Above is a list of images that I've pulled from the registry and those I've created myself (we'll shortly see how). You will have a different list of images on your machine. The **TAG** refers to a particular snapshot of the image and the **ID** is the corresponding unique identifier for that image. For simplicity, you can think of an image akin to a Git repository - images can be committed with changes and have multiple versions. When you do not provide a specific version number, the client defaults to latest. ### Clutter and Cache Docker images are **cached** on your machine in the location where Docker was installed. These image files are not visible in the same directory where you might have used `docker pull `. Some Docker images can be large. Especially data science images with many scientific programming libraries and packages pre-installed. !!! warning "Checking your system cache" Pulling many images from the Docker Registries may fill up your hard disk! To inspect your system and disk use: ``` $ docker system info $ docker system df ``` To find out how many images are on your machine, type: ``` $ docker images ``` To remove images that you no longer need, type: ``` $ docker system prune ``` This is where it becomes important to differentiate between *images*, *containers*, and *volumes* (which we'll get to more in a bit). You can take care of all of the dangling images and containers on your system. Note, that `prune` will not remove your cached *images* ``` $ docker system prune WARNING! This will remove: - all stopped containers - all networks not used by at least one container - all dangling images - all dangling build cache Are you sure you want to continue? [y/N] ``` If you added the `-af` flag it will remove "all" `-a` dangling images, empty containers, AND ALL CACHED IMAGES with "force" `-f`. --- ## Managing Data in Docker It is possible to store data within the writable layer of a container, but there are some limitations: - The data doesn’t persist when that container is no longer running, and it can be difficult to get the data out of the container if another process needs it. - A container’s writable layer is tightly coupled to the host machine where the container is running. You can’t easily move the data somewhere else. - Its better to put your data into the container **AFTER** it is built - this keeps the container size smaller and easier to move across networks. Docker offers three different ways to mount data into a container from the Docker host: - **Volumes** - **tmpfs mounts** - **Bind mounts** ![bind_mount](https://unm-carc.github.io/container-camp/assets/docker/bind_mount.png) When in doubt, volumes are almost always the right choice. ### Volumes Volumes are often a better choice than persisting data in a container’s writable layer, because using a volume does not increase the size of containers using it, and the volume’s contents exist outside the lifecycle of a given container. While bind mounts (which we will see in the Advanced portion of the Camp) are dependent on the directory structure of the host machine, volumes are completely managed by Docker. Volumes have several advantages over bind mounts: - Volumes are easier to back up or migrate than bind mounts. - You can manage volumes using Docker CLI commands or the Docker API. - Volumes work on both UNIX and Windows containers. - Volumes can be more safely shared among multiple containers. - A new volume’s contents can be pre-populated by a container. ??? tip "When Should I Use the Temporary File System mount?" If your container generates non-persistent state data, consider using a `tmpfs` mount to avoid storing the data anywhere permanently, and to increase the container’s performance by avoiding writing into the container’s writable layer. The data is written to the host's memory instead of a volume; When the container stops, the `tmpfs` mount is removed, and files written there will not be kept. Choose the `-v` flag for mounting volumes `-v` or `--volume`: Consists of three fields, separated by colon characters (:). The fields must be in the correct order, and the meaning of each field is not immediately obvious. - The **first** field is the path on your **local machine** that where the data are. - The **second** field is the path where the file or directory are **mounted in the container**. - The third field is optional, and is a comma-separated list of options, such as `ro` (read only). ``` -v /home/username/your_data_folder:/container_folder ``` ``` $ docker run -v /home/$USER/read_cleanup:/work alpine:latest ls -l /work ``` So what if we wanted to work interactively inside the container? ``` $ docker run -it -v /home/$USER/read_cleanup:/work alpine:latest sh ``` ``` $ ls -l $ ls -l work ``` Once you're in the container, you will see that the `/work` directory is mounted in the working directory. Any data that you add to that folder outside the container will appear INSIDE the container. And any work you do inside the container saved in that folder will be saved OUTSIDE the container as well. --- ## Working with Interactive Containers Let's go ahead and run some Integrated Development Environment images from "trusted" organizations on the Docker Hub Registry. ### Jupyter Lab :material-language-python: or RStudio-Server :material-language-r: IDE In this section, let's find a Docker image which can run a Jupyter Notebook Search for official images on Docker Hub which contain the string 'jupyter' ``` $ docker search jupyter ``` It should return something like: ``` NAME DESCRIPTION STARS OFFICIAL AUTOMATED jupyter/datascience-notebook Jupyter Notebook Data Science Stack from htt… 912 jupyter/all-spark-notebook Jupyter Notebook Python, Scala, R, Spark, Me… 374 jupyter/scipy-notebook Jupyter Notebook Scientific Python Stack fro… 337 jupyterhub/jupyterhub JupyterHub: multi-user Jupyter notebook serv… 307 [OK] jupyter/tensorflow-notebook Jupyter Notebook Scientific Python Stack w/ … 298 jupyter/pyspark-notebook Jupyter Notebook Python, Spark, Mesos Stack … 224 jupyter/base-notebook Small base image for Jupyter Notebook stacks… 168 jupyter/minimal-notebook Minimal Jupyter Notebook Stack from https://… 150 jupyter/r-notebook Jupyter Notebook R Stack from https://github… 44 jupyterhub/singleuser single-user docker images for use with Jupyt… 43 [OK] jupyter/nbviewer Jupyter Notebook Viewer 27 [OK] ``` Search for images on Docker Hub which contain the string 'rstudio' ``` $ docker search rstudio ``` ``` NAME DESCRIPTION STARS OFFICIAL AUTOMATED rocker/rstudio RStudio Server image 389 [OK] rstudio/r-base Docker Images for R 24 rocker/rstudio-stable Build RStudio based on a debian:stable (debi… 16 [OK] rstudio/rstudio-server-pro Deprecated Docker images for RStudio Server … 10 rstudio/r-session-complete Images for sessions and jobs in RStudio Serv… 10 rstudio/plumber 6 rstudio/rstudio-connect Default Docker image for RStudio Connect 4 rstudio/r-builder-images-win 3 rstudio/rstudio-workbench Docker Image for RStudio Workbench (formerly… 2 saagie/rstudio RStudio with sparklyr, Saagie's addin and ab… 2 [OK] ibmcom/rstudio-ppc64le Integrated development environment (IDE) for… 2 rstudio/checkrs-tew Test Environment: Web 1 [OK] rstudio/rstudio-package-manager Default Docker image for RStudio Package Man… 1 rstudio/shinyapps-package-dependencies Docker images used to test the install scrip… 1 rstudio/rstudio-workbench-preview 1 ``` ??? warning "Untrusted community images" An important thing to note: None of these Jupyter or RStudio images are 'official' Docker images, meaning they could be trojans for spyware, malware, or other nasty warez. --- ## Understanding PORTS When we want to run a container that runs on the open internet, we need to add a [TCP or UDP port number](https://en.wikipedia.org/wiki/List_of_TCP_and_UDP_port_numbers){target=_blank} from which we can access the application in a browser using the machine's IP (Internet Protocol) address or DNS (Domain Name Service) location. To do this, we need to access the container over a separate port address on the machine we're working on. Docker uses the flag `--port` or `-p` for short followed by two sets of port numbers. ??? note "Exposing Ports" Docker can in fact expose all ports to a container using the capital `-P` flag For security purposes, it is generally NEVER a good idea to expose all ports. Typically these numbers can be the same, but in some cases your machine may already be running another program (or container) on that open port. The port has two sides `left:right` separated by a colon. The left side port number is the INTERNAL port that the container software thinks its using. The right side number is the EXTERNAL port that you can access on your computer (or virtual machine). Here are some examples to run basic RStudio and Jupyter Lab: ``` $ docker run --rm -p 8787:8787 -e PASSWORD=cc2022 rocker/rstudio ``` **note: on CodeSpaces, the reverse proxy for the DNS requires you to turn off authentication** ``` $ docker run --rm -p 8787:8787 -e DISABLE_AUTH=true rocker/rstudio ``` ``` $ docker run --rm -p 8888:8888 jupyter/base-notebook ``` ??? note "Preempting stale containers from your cache" We've added the `--rm` flag, which means the container will automatically removed from the cache when the container is exited. When you start an IDE in a terminal, the terminal connection must stay active to keep the container alive. ### Detaching your container while it is running If we want to keep our window in the foreground we can use the `-d` - the *detached* flag will run the container as a background process, rather than in the foreground. When you run a container with this flag, it will start, run, telling you the container ID: ``` $ docker run --rm -d -p 8888:8888 jupyter/base-notebook ``` Note, that your terminal is still active and you can use it to launch more containers. To view the running container, use the `docker ps` command. --- ## Docker Commands Here is a compiled list of fundamental Docker Commands: | Command | Usage | Example | |---------|-------|---------| | `pull` | Downloads an image from Docker Hub | `docker pull hello-world:latest` | | `run` | runs a container with entrypoint | `docker run -it user/image:tag` | | `build` | Builds a docker image from a Dockerfile in current working directory | `docker build -t user/image:tag .` | | `images` | List all images on the local machine | `docker images list` | | `tag` | Adds a different tag name to an image | `docker tag hello-world:latest hello-world:new-tag-name` | | `login` | Authenticate to the Docker Hub (requires username and password) | `docker login` | | `push` | Upload your new image to the Docker Hub | `docker push user/image:tag` | | `inspect` | Provide detailed information on constructs controlled by Docker | `docker inspect containerID` | | `ps` | List all containers on your system | `docker ps -a` | | `rm` | Delete a stopped or running container |`docker rm -f ` | | `rmi` | Delete an *image* from your cache | `docker rmi hello-world:latest` | | `stop` | Stop a running container | `docker stop alpine:latest` | | `system` | View system details, remove old images and containers with `prune` |`docker system prune` | | `push` | Uploads an image to the Docker Hub (or other private registry) | `docker push username/image:tag` |

Adapted from [CyVerse Container Camp](https://github.com/CyVerse-learning-materials/container-camp/blob/1746ee8873bcbed2bd3e62500262a9e9571cff16/docs/docker/intro.md){target=_blank} (last source update 2023-08-15), CC BY 4.0. Spotted a problem? [Open an issue](https://github.com/UNM-CARC/container-camp/issues){target=_blank}.

---8<--- https://unm-carc.github.io/container-camp/docker/registry/ --- title: "Finding the right container" description: "Container terminology and the public registries (Docker Hub, Quay.io, BioContainers, NVIDIA NGC) to search before building your own image." type: Lesson tags: - Docker - Container Registries generated: by: "claude/opus-5" at: "2026-09-11T00:00:00Z" sources: - id: cyverse-container-camp resource: "https://github.com/CyVerse-learning-materials/container-camp/blob/1746ee8873bcbed2bd3e62500262a9e9571cff16/docs/docker/registry.md" title: "CyVerse Container Camp: docs/docker/registry.md" author: "team:cyverse" last_modified: "2022-05-06T15:57:18-07:00" --- # Finding the right container There is a high likelihood that the Docker image you need already exists for the application you use in your research. Rather than starting from scratch and creating your own *image* from a `Dockerfile`, you should first go searching for one that already exists. This will save you time writing and compiling your own image. First, you will need to know where to look for existing images. Docker images are hosted across the internet on libraries that are called **Registries**. !!! info "Container Terminology" - **:octicons-container-16: Container**: when an *image* is run it becomes the *container*; A container shares its kernel with other containers using the same image and each runs as an isolated process on the host. - **:material-archive: Registry**: an online library of container images - **:octicons-file-badge-16: Name**: the name of the image - **:material-tag: Tag**: identifies exact version of the image, following a `:` in the name. If no tag name is given, by default Docker will assign the ``latest`` tag name to an image. - **:material-layers: Layer**: an intermediate image, the result of a single set of build commands. An image is built upon layers starting with a base operating system. - **:material-file-edit: Dockerfile**: a text file that contains a list of commands that the Docker daemon calls while creating the layers of an image. - **:material-file: Image**: images are compressed files in a cache on a host and can be built locally or downloaded from a *registry* - **:material-file-code: Base image**: has no parent layers, usually base images are a basic Linux operating system like `alpine`, `debian`, or `ubuntu`. - **:material-file-tree: Child image**: any image built from a base image that has added layers. - **:material-file-check: Official images**: verified (:material-shield-check:) images hosted on a public container registry. Safe to use, built by the professionals who know them best. - **:material-file-star: Publisher image**: certified images that also include support and guarantee compatibility. - **:material-file-question: User image**: images created and shared by users like you. Their contents may be a mystery and therefore are not to be trusted. --- ## Docker Registries Docker uses the concept of **Registries**, online libraries where container **images** are cached for public utilization. ??? question "What *EXACTLY* is a Container Registry?" A Registry is a storage and distribution system for named Docker images Organized by owners into **repositories** with compiled **images** that users can download and run Things you can do with registries: - Search for public images; - Pull official images; - Host and share private images; - Push your images. You must have an account on each registry in order to create repositories and to host your own images. - You can create multiple repositories; - You can create multiple tagged images in a repository; - You can set up your own private registry using a *Docker Trusted Registry*. --- ## Search Image Registries !!! warning Only use images from trusted sources or images for which you can see the `Dockerfile`. Any image from an untrusted source could contain something other than what's indicated. If you can see the Dockerfile you can see exactly what is in the image. Docker looks into the [Docker Hub](https://hub.docker.com/){target=_blank} public registry by default. Examples of public/private registries to consider for your research needs: | Registry Name | Container Types | |---------------|-----------------| | [:material-docker: Docker Hub](https://hub.docker.com/){target=_blank} | Official Images for Docker | | [:material-aws: Amazon Elastic Container Registry](https://aws.amazon.com/ecr/){target=_blank} | run containers on AWS | | [:material-google-cloud: Google Container Registry](https://cloud.google.com/container-registry){target=_blank} | run containers on Google Cloud | | [:material-microsoft-azure: Azure Container Registry](https://docs.microsoft.com/en-us/azure/container-registry/){target=_blank} | run containers on Azure | | [:material-graph: NVIDIA GPU Cloud](https://www.nvidia.com/en-us/gpu-cloud/){target=_blank} | containers for GPU computing | | [:material-github: GitHub Container Registry](https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-container-registry){target=_blank} | managed containers on GitHub | | [:material-gitlab: GitLab Container Registry](https://docs.gitlab.com/ee/user/packages/container_registry/){target=_blank} | managed containers on GitLab | | [:material-redhat: RedHat Quay.io](https://cloud.redhat.com/products/quay){target=_blank} | containers managed by RedHat | | [:material-dna: BioContainers Registry](https://biocontainers.pro/){target=_blank} | bioinformatics containers ### :material-docker: DockerHub Docker Hub is a service provided by Docker for finding and sharing container images with your team. Docker Hub is the most well-known and popular image registry for Docker containers. !!! info - **Registry**: a storage and distribution system for named Docker images - **Repository**: collection of "images" with individual "tags". - **Teams & Organizations:** Manages access to private repositories. - **Builds:** Automatically build container images from GitHub or Bitbucket on the Docker Hub. - **Webhooks:** Trigger actions after a successful push to a repository to integrate Docker Hub with other services. ### :material-dna: BioContainers BioContainers is a community-driven project that provides the infrastructure and basic guidelines to create, manage and distribute bioinformatics containers with **special focus in proteomics, genomics, transcriptomics and metabolomics**. BioContainers is based on the popular frameworks of Docker. Although anyone can create a BioContainer, the majority of BioContainers are created by the Bioconda project. Every Bioconda package has a corresponding BioContainer available at Quay.io. ### :material-redhat: RedHat Quay.io Quay is another general image registry. It works the same way as Docker Hub. However, Quay is home to all BioContainers made by the Bioconda project. Now we will find a BioContainer image at Quay, pull that image and run it on cloud virtual machine. ### :material-graph: NVIDIA GPU Cloud NVIDIA is one of the leading makers of graphic processing units (GPU). GPU were established as a means of handling graphics processing operations for video cards, but have been greatly expanded for use in generalized computing applications, Machine Learning, image processing, and matrix-based linear algebras. NVIDIA have created their own set of Docker containers and Registries for running on CPU-GPU enabled systems. - [NVIDIA-Docker](https://github.com/NVIDIA/nvidia-docker){target=_blank} runs atop the NVIDIA graphics drivers on the host system, the NVIDIA drivers are imported to the container at runtime. - [NVIDIA Docker Hub](https://hub.docker.com/u/nvidia){target=_blank} hosts numerous NVIDIA Docker containers, from which you can build your own images. - [NVIDIA GPU Cloud](https://ngc.nvidia.com){target=_blank} hosts numerous containers for HPC and Cloud applications. You must register an account with them (free) to access these. NVIDIA GPU Cloud hosts three [registry spaces](https://docs.nvidia.com/ngc/ngc-user-guide/ngc-spaces.html#ngc-spaces){target=_blank} - `nvcr.io/nvidia` - catalog of fully integrated and optimized deep learning framework containers. - `nvcr.io/nvidia-hpcvis` - catalog of HPC visualization containers (beta). - `nvcr.io/hpc` - popular third-party GPU ready HPC application containers. NVIDIA Docker can be used as a base-image to create containers running graphical applications remotely. High resolution 3D screens are piped to a remote desktop platform.

Adapted from [CyVerse Container Camp](https://github.com/CyVerse-learning-materials/container-camp/blob/1746ee8873bcbed2bd3e62500262a9e9571cff16/docs/docker/registry.md){target=_blank} (last source update 2022-05-06), CC BY 4.0. Spotted a problem? [Open an issue](https://github.com/UNM-CARC/container-camp/issues){target=_blank}.

---8<--- https://unm-carc.github.io/container-camp/docker/build/ --- title: "Building Docker images" description: "Write a Dockerfile instruction by instruction (FROM, RUN, ENV, COPY, CMD, ENTRYPOINT, USER, EXPOSE), including multi-stage builds." type: Lesson tags: - Docker - Dockerfile generated: by: "claude/opus-5" at: "2026-09-11T00:00:00Z" sources: - id: cyverse-container-camp resource: "https://github.com/CyVerse-learning-materials/container-camp/blob/1746ee8873bcbed2bd3e62500262a9e9571cff16/docs/docker/build.md" title: "CyVerse Container Camp: docs/docker/build.md" author: "team:cyverse" last_modified: "2022-05-12T13:52:06Z" --- # Building Docker images Now that we are relatively comfortable with running Docker, we can look at some advanced Docker topics, such as: - Building our own Docker images from the `Dockerfile` - Modify an existing Dockerfile and create a new image - Push an image to a Registry ## Requirements Clone our example repository with pre-written Dockerfiles From your CodeSpace, we are going to copy a second GitHub repository onto our VM. If you are working locally, make sure that you change directories away from any other Git repository that you may have been working in. ``` $ cd /workspaces $ git clone https://github.com/cyverse-education/intro2docker $ cd intro2docker/ ``` !!! tip "Example Dockerfiles" Copies of the camp's example Dockerfiles are kept in this site's repository under [`examples/`](https://github.com/UNM-CARC/container-camp/tree/main/examples){target=_blank}: a small Python app on [`alpine`](https://github.com/UNM-CARC/container-camp/tree/main/examples/alpine){target=_blank}, the `fortune | cowsay | lolcat` image built in this lesson ([`cowsay`](https://github.com/UNM-CARC/container-camp/tree/main/examples/cowsay){target=_blank}), an iRODS client image on [`ubuntu`](https://github.com/UNM-CARC/container-camp/tree/main/examples/ubuntu){target=_blank}, and a Jupyter image ([`jupyter/minimal-notebook`](https://github.com/UNM-CARC/container-camp/tree/main/examples/jupyter/minimal-notebook){target=_blank}). --- ## Writing a Dockerfile !!! note "Important" `Dockerfile` must be capitalized. It does not have a file extension. Create a file called `Dockerfile`, and add content to it as described below, e.g. ``` $ touch Dockerfile ``` !!! note "Formatting in the `Dockerfile`" We use a code line escape character `\` to allow single line scripts to be written on multiple lines in the Dockerfile. We also use the double characters `&&` which essentially mean “if true, then do this” while executing the code. The `&&` can come at the beginning of a line or the end when used with `\`. The `Dockerfile` contains **Instructions**: a series of commands that Docker executes during the creation and execution of a container. ---- ### ARG The only command that can come before a `FROM` statement is `ARG` `ARG` can be used to set arguments for later in the build, e.g., ``` ARG VERSION=latest FROM ubuntu:$VERSION ``` ### FROM A valid `Dockerfile` must start with a `FROM` statement which initializes a new build stage and sets the **base image** for subsequent layers. We’ll start by specifying our base image, using the FROM statement ``` FROM ubuntu:latest ``` If you are building on an `arm64` or Windows system you can also give the optional `--platform` flag, e.g., ``` FROM --platform=linux/amd64 ubuntu:latest ``` ??? question "When to use a multi-stage build pattern?" Docker has the ability to build container images from one image, and run that "builder" image from a second "base" image, in what is called a "builder pattern". Build patterns are useful if you're compiling code from (proprietary) source code and only want to feature the binary code as an executed function in the container at run time. Build patterns can greatly reduce the size of your container. You can use multiple `FROM` commands as build stages. The `AS` statement follows the `image:tag` as a psuedo argument. ``` # build stage FROM golang:latest AS build-env WORKDIR /go/src/app ADD . /go/src/app RUN go mod init RUN cd /go/src/app && go build -o hello # final stage FROM alpine:latest WORKDIR /app COPY --from=build-env /go/src/app /app/ ENTRYPOINT ./hello ``` A runnable copy of this two-stage build, with a single-stage version for comparison, is in [`examples/builder`](https://github.com/UNM-CARC/container-camp/tree/main/examples/builder){target=_blank}; build both and compare their sizes with `docker images`. ### LABEL You can create labels which are then tagged as JSON metadata to the image ``` LABEL author="your-name" LABEL email="your@email-address" LABEL version="v1.0" LABEL description="This is your first Dockerfile" LABEL date_created="2022-05-13" ``` You can also add labels to a container when it is run: ``` $ docker run --label description="this label came later" ubuntu:latest $ docker ps -a $ docker inspect ### ``` ### RUN Different than the `docker run` command is the `RUN` build function. `RUN` is used to create new layers atop the "base image" Here, we are going to install some games and programs into our base image: ``` RUN apt-get update && apt-get install -y fortune cowsay lolcat ``` Here we've installed `fortune` `cowsay` and `lolcat` as new programs into our base image. !!! warning "Best practices for building new layers" Ever time you use the `RUN` command it is a good idea to use the `apt-get update` or `apt update` command to make sure your layer is up-to-date. This can become a problem though if you have a very large container with a large number of `RUN` layers. ### ENV In our new container, we need to change and update some of the environment flags. We can do this using the `ENV` command ``` ENV PATH=/usr/games:${PATH} ENV LC_ALL=C ``` Here we are adding the `/usr/games` directory to the `PATH` so that when we run the new container it will find our newly installed game commands We are also updating the "[locales](https://www.tecmint.com/set-system-locales-in-linux/){target=_blank}" to set the language of the container. ### COPY The `COPY` command will copy files from the directory where `Dockerfile` is kept into the new image. You must specify where to copy the files or directories ``` COPY . /app ``` ??? question "When to use `COPY` vs `ADD`" `COPY` is more basic and is good for files `ADD` has some extra features like `.tar` extraction and URL support ### CMD The `CMD` command is used to run software in your image. In general use the ["command"] syntax: ``` CMD ["executable", "parameter1", "parameter2"] ``` ### ENTRYPOINT ENTRYPOINT works similarly to `CMD` but is designed to allow you to run your container as an executable. ``` ENTRYPOINT fortune | cowsay | lolcat ``` The default `ENTRYPOINT` of most images is `/bin/sh -c` which executes a `shell` command. `ENTRYPOINT` supports both the `ENTRYPOINT ["command"]` syntax and the `ENTRYPOINT command` syntax ??? question "What is the difference in the `ENTRYPOINT` and `CMD`" The CMD instruction is used to define what is execute when the container is run. The ENTRYPOINT instruction cannot be overridden, instead it is appended to when a new command is given to the `docker run container:tag new-cmd` statement the executable is defined with ENTRYPOINT, while CMD specifies the default parameter ### USER Most containers are run as `root` meaning that they have super-user privileges within themselves Typically, a new user is necessary in a container that is used interactively or may be run on a remote system. During the build of the container, you can create a new user with the `adduser` command and set up a `/home/` directory for them. This new user would have something like 1000:1000 `uid:gid` permissions without `sudo` privileges. As a last step, the container is run as the new `USER`, e.g., ``` ARG VERSION=18.04 FROM ubuntu:$VERSION RUN useradd ubuntu && \ chown -R ubuntu:ubuntu /home/ubuntu USER ubuntu ``` ### EXPOSE You can open [ports](https://unm-carc.github.io/container-camp/docker/intro/#understanding-ports) using the `EXPOSE` command. ``` EXPOSE 8888 ``` The above command will expose port 8888. The [`examples/jupyter/minimal-notebook`](https://github.com/UNM-CARC/container-camp/tree/main/examples/jupyter/minimal-notebook){target=_blank} Dockerfile uses `EXPOSE 8888` for its Jupyter server. !!! note Running multiple containers using the same port is not trivial and would require the usage of a web server such as [NGINX](https://www.nginx.com/){target=_blank}. However, you can have multiple containers interact with each other using [Docker Compose](https://unm-carc.github.io/container-camp/docker/compose/). --- ## Summary of Instructions | Instruction Command | Description | | --- | --- | | `ARG` | Sets environmental variables during image building | | `FROM` | Instructs to use a specific Docker image | | `LABEL` | Adds metadata to the image | | `RUN` | Executes a specific command | | `ENV` | Sets environmental variables | | `COPY` | Copies a file from a specified location to the image | | `CMD` | Sets a command to be executed when running a container | | `ENTRYPOINT` | Configures and run a container as an executable | | `USER` | Used to set User specific information | | `EXPOSE` | exposes a specific port |

Adapted from [CyVerse Container Camp](https://github.com/CyVerse-learning-materials/container-camp/blob/1746ee8873bcbed2bd3e62500262a9e9571cff16/docs/docker/build.md){target=_blank} (last source update 2022-05-12), CC BY 4.0. Spotted a problem? [Open an issue](https://github.com/UNM-CARC/container-camp/issues){target=_blank}.

---8<--- https://unm-carc.github.io/container-camp/docker/advanced/ --- title: "Advanced Docker: volumes and bind mounts" description: "Create, inspect, and populate Docker volumes, and use read-write and read-only bind mounts to share host data with containers." type: Lesson tags: - Docker - Volumes - Data generated: by: "claude/opus-5" at: "2026-09-11T00:00:00Z" sources: - id: cyverse-container-camp resource: "https://github.com/CyVerse-learning-materials/container-camp/blob/1746ee8873bcbed2bd3e62500262a9e9571cff16/docs/docker/advanced.md" title: "CyVerse Container Camp: docs/docker/advanced.md" author: "team:cyverse" last_modified: "2022-04-25T15:46:54-07:00" --- # Advanced Docker: volumes and bind mounts ## Volumes When you run a container, you can bring a directory from the host system into the container, and give it a new name and location using the `-v` or `--volume` flag. ``` bash $ mkdir -p ~/local-data-folder $ echo "some data" >> ~/local-data-folder/data.txt $ docker run -v ${HOME}/local-data-folder:/data $YOUR_DOCKERHUB_USERNAME/mynotebook:latest cat /data/data.txt ``` In the example above, you can mount a folder from your localhost, in your home user directory into the container as a new directory named `/data`. ### Create and manage volumes Unlike a bind mount, you can create and manage volumes outside the scope of any container. A given volume can be mounted into multiple containers simultaneously. When no running container is using a volume, the volume is still available to Docker and is not removed automatically. You can remove unused volumes using `docker volume prune` command. When you create a Docker volume, it is stored within a directory on the Docker Linux host (`/var/lib/docker/`). !!! note File location on Mac OS X is a bit different: Let's create a volume ``` bash $ docker volume create my-vol ``` List volumes: ``` bash $ docker volume ls local my-vol ``` Inspect a volume by looking at the Mount section in the `docker volume inspect` ``` bash $ docker volume inspect my-vol [ { "Driver": "local", "Labels": {}, "Mountpoint": "/var/lib/docker/volumes/my-vol/_data", "Name": "my-vol", "Options": {}, "Scope": "local" } ] ``` Remove a volume ``` bash $ docker volume rm my-vol $ docker volume ls ``` ### Populate a volume using a container This example starts an `alpine` container and populates the new volume `output-vol` with the some output created by the container. ``` bash docker volume create output-vol docker run --name=data-app --mount source=output-vol,target=/data alpine sh -c 'env >> /data/container-env.txt' ``` Use `docker inspect output-vol` to see where the volume data lives on your host, and then use `cat` to confirm that it contains the output created by the container. ``` bash docker volume inspect output-vol sudo cat /var/lib/docker/volumes/output-vol/_data/container-env.txt ``` You should see something like: ``` HOSTNAME=790e13bba28a SHLVL=1 HOME=/root PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PWD=/ ``` After running either of these examples, run the following commands to clean up the container and volume. ``` bash docker rm data-app docker volume rm output-vol ``` --- ## Bind mounts **Bind mounts:** When you use a bind mount, a file or directory on the host machine is mounted into a container. ??? tip If you are developing new Docker applications, consider using named **volumes** instead. You can't use Docker CLI commands to directly manage bind mounts. ![bind_mount](https://unm-carc.github.io/container-camp/assets/docker/bind_mount.png) !!! warning One side effect of using bind mounts, for better or for worse, is that you can change the host filesystem via processes running in a container, including creating, modifying, or deleting important system files or directories. This is a powerful ability which can have security implications, including impacting non-Docker processes on the host system. If you use `--mount` to bind-mount a file or directory that does not yet exist on the Docker host, Docker does not automatically create it for you, but generates an error. ### Start a container with a bind mount Create a `bind-data` directory in your home directory. ``` bash cd ~ mkdir -p ~/bind-data ``` Run a container, mounting this directory inside the container, and the container should create some data in there. ``` bash docker run --mount type=bind,source="$(pwd)"/bind-data,target=/data alpine sh -c 'env >> /data/container-env.txt' ``` Check that the output looks right. ``` bash cat ~/bind-data/container-env.txt ``` ### Use a read-only bind mount For some development applications, the container needs to write into the bind mount, so changes are propagated back to the Docker host. At other times, the container only needs read access. This example modifies the one above but mounts the directory as a read-only bind mount, by adding `ro` to the (empty by default) list of options, after the mount point within the container. Where multiple options are present, separate them by commas. ``` bash docker run --mount type=bind,source="$(pwd)"/bind-data,target=/data,readonly alpine sh -c 'ls -al /data/ && env >> /data/container-env.txt' ``` You should see an error message about not being able to write to a read-only file system. ``` bash sh: can't create /data/container-env.txt: Read-only file system ```

Adapted from [CyVerse Container Camp](https://github.com/CyVerse-learning-materials/container-camp/blob/1746ee8873bcbed2bd3e62500262a9e9571cff16/docs/docker/advanced.md){target=_blank} (last source update 2022-04-25), CC BY 4.0. Spotted a problem? [Open an issue](https://github.com/UNM-CARC/container-camp/issues){target=_blank}.

---8<--- https://unm-carc.github.io/container-camp/docker/actions/ --- title: "GitHub Actions and Docker" description: "Build and push Docker images to Docker Hub automatically with a GitHub Actions workflow and repository secrets." type: Lesson tags: - Docker - GitHub Actions - CI/CD generated: by: "claude/opus-5" at: "2026-09-11T00:00:00Z" sources: - id: cyverse-container-camp resource: "https://github.com/CyVerse-learning-materials/container-camp/blob/1746ee8873bcbed2bd3e62500262a9e9571cff16/docs/docker/actions.md" title: "CyVerse Container Camp: docs/docker/actions.md" author: "team:cyverse" last_modified: "2022-05-16T17:03:29-07:00" --- # GitHub Actions and Docker [GitHub Actions](https://github.com/features/actions?gclid=EAIaIQobChMI6JzCw9bf9wIVhD-tBh11sQ42EAAYASABEgKxrvD_BwE){target=_blank} is a feature that allows automation and execution of workflows invoved in the development of your software and code. Read more on GitHub Actions at the [offical GitHub Docs page](https://docs.github.com/en/actions/learn-github-actions/understanding-github-actions){target=_blank}. Actions can also be used to create and manage your Docker Images. Here you will learn how to can create Actions which build and then push images from GitHub to a Docker Registry of your choice. --- ### Prerequisites - A [GitHub](https://github.com/){target=_blank} account - A [Docker](https://hub.docker.com/){target=_blank} account ## Setting up a Git Repository with Actions for Docker ??? info "CI/CD Terminology" * **:material-infinity: continuous integration** builds, tests, and automatically accepts changes to code in a repository * **:material-infinity: continuous delivery** delivers code changes to production-ready environments * **:material-infinity: continuous deployment** does just that, it deploys changes to the code * **:material-pipe: CI pipeline** compiles code, tests it, and makes sure all of your changes work. It should run whenever there is a change (push) to the repository * **:material-pipe: CD pipeline** goes one step further and deploys new code into production. We are focusing on GitHub, but there are other platforms which you can explore for building and pushing containers to registries These include [GitLab Runners](https://docs.gitlab.com/runner/){target=_blank} Other types of Continuous Integration are used on code repositories to ensure that code stays functional. ### Create a GitHub Repository Navigate to your GitHub Account and select **New**; ![Actions_01](https://unm-carc.github.io/container-camp/assets/docker/Actions_01.png) ??? tip ":material-github: Creating README, LICENSE, & .gitignore" When you create your new git repository you are asked to create a `README`, a `LICENCE`, and a `.gitignore` file. Go ahead and create all three of these, as they are useful and fundamental to making your repository reproducible. README -- we want to use the README to help future-us when we revisit this repository. Make sure to include detailed instructions LICENSE -- pick a license which is useful for your specific software use case. `.gitignore` -- use a file which will keep docker project files isolated ??? info ":material-docker: `.gitignore` example" ``` # Docker project generated files to ignore # if you want to ignore files created by your editor/tools, # please consider a global .gitignore https://help.github.com/articles/ignoring-files .vagrant* bin docker/docker .*.swp a.out *.orig build_src .flymake* .idea .DS_Store docs/_build docs/_static docs/_templates .gopath/ .dotcloud *.test bundles/ .hg/ .git/ vendor/pkg/ pyenv Vagrantfile ``` In the repository, create two nested folders required for a Docker Actions project: - A `.github/workflows` folder: containing required necessary `yml` files that build the containers through GitHub; In the workflows folder, we're going to add a `.yml` In the main repository, along with the `README` and `LICENCE` file, create another folder called `/docker` In the `/docker` folder we're going to put the `Dockerfile` file necessary to build the image. ### Link your GitHub and Docker accounts Ensure you can access [Docker Hub](https://hub.docker.com/){target=_blank} from any workflows you create: 1. Add your Docker ID as a secret to GitHub. Navigate to your GitHub repository and click **Settings > Secrets and variables > Actions > New repository secret**. ![Actions_03](https://unm-carc.github.io/container-camp/assets/docker/Actions_03.png) 2\. Create a new secret with the name `DOCKER_HUB_USERNAME` and your Docker ID as value. 3\. On DockerHub, create a new Personal Access Token (PAT). To create a new token, go to [Docker Hub Settings](https://hub.docker.com/settings/security){target=_blank} and then click **New Access Token**. Name it, and copy the newly generated PAT. ??? tip Name your Docker Hub Access token the same name as your GitHub repository, it will help with keeping track which GitHub repository is related to which Docker image. ![Actions_04](https://unm-carc.github.io/container-camp/assets/docker/Actions_04.png) 4. On GitHub, return to your repository secrets, and add the PAT as a secret with the name `DOCKER_HUB_ACCESS_TOKEN`. ![Actions_05](https://unm-carc.github.io/container-camp/assets/docker/Actions_05.png) --- ## Setting up a GitHub Action Workflow Now that you have connected your GitHub repository with your Docker account, you are ready to add the necessary files to your repo. !!! note In this example, we will use the existing Docker Image Alpine. 1\. In your GitHub repository, create a file and name in `Dockerfile`; In the first line of your `Dockerfile` paste: ``` FROM alpine:3.14 ``` 2\. Click the `Actions` tab and in the search bar, search for `docker`. Select the `docker image workflow` (as shown in the image below) !!! note This will create the `.github/workflows` repository and necessary `yml` file required for the GitHub actions. ![Actions_06](https://unm-carc.github.io/container-camp/assets/docker/Actions_06.png) 3\. You will be prompted to the `docker-image.yml` file; paste the following code, and `commit` your changes. ``` name: Docker Image Small Alpine on: push: branches: [ main ] pull_request: branches: [ main ] jobs: build: runs-on: ubuntu-latest steps: - name: Checkout uses: actions/checkout@v4 - name: Login to Docker Hub uses: docker/login-action@v3 with: username: ${{ secrets.DOCKER_HUB_USERNAME }} password: ${{ secrets.DOCKER_HUB_ACCESS_TOKEN }} - name: Build and push uses: docker/build-push-action@v6 with: context: . file: ./Dockerfile push: true tags: ${{ secrets.DOCKER_HUB_USERNAME }}/simplewhale ``` 4\. Upon committing and pushing your changes, you can check your Workflows under the Actions tab on GitHub. !!! note Github will show you when a workflow is building: - An orange dot next to your commit count means that the workflow is running; - A crossed red circle means that your workflow has failed; - A green check means your workflow ran successfully. ![Actions_07](https://unm-carc.github.io/container-camp/assets/docker/Actions_07.png) 5\. Navigate to your Docker Hub to see your GitHub Actions generated Docker image. ![Actions_08](https://unm-carc.github.io/container-camp/assets/docker/Actions_08.png) ![Actions_09](https://unm-carc.github.io/container-camp/assets/docker/Actions_09.png) --- ## Tagging a Pushing an Image When updating your own Docker Images, you may want to tag the images with a version number. In this example, we update our Alpine Docker Image to a newer version and adding a package. Navigate to your `Dockerfile` and modify it by adding the following lines: ``` FROM alpine:3.15 LABEL author="" LABEL email="" LABEL date_created="" RUN apk update && apk upgrade --available RUN apk add vim fortune ENV PATH=/usr/games:${PATH} ENV LC_ALL=C ENTRYPOINT fortune ``` Save, and navigate to the github action (`.github/workflows/docker-image.yml`). The last line should be `tags`, edit the line to reflect the version you would like to add. ``` tags: ${{ secrets.DOCKER_HUB_USERNAME }}/simplewhale:v0.2 ``` !!! warning - To correctoy add a tag, add a colon followed by a version number such as `:`. - If there is no tag, the version pushed will default to `latest`. - **Remember** to update/change the tag line in future pushes. Add, commit, push your changes; You will see your Docker Image being built with the added tagged version. ![Actions_10](https://unm-carc.github.io/container-camp/assets/docker/Actions_10.png) --- ## Setting up your own [:material-github: GitHub Actions Runner](https://github.com/actions/runner){target=_blank} [:material-file-document-outline: Self-hosted Actions Runner Docs](https://docs.github.com/en/actions/hosting-your-own-runners/about-self-hosted-runners){target=_blank} [:material-gitlab: GitLab Runner](https://docs.gitlab.com/runner/install/){target=_blank}

Adapted from [CyVerse Container Camp](https://github.com/CyVerse-learning-materials/container-camp/blob/1746ee8873bcbed2bd3e62500262a9e9571cff16/docs/docker/actions.md){target=_blank} (last source update 2022-05-16), CC BY 4.0. Spotted a problem? [Open an issue](https://github.com/UNM-CARC/container-camp/issues){target=_blank}.

---8<--- https://unm-carc.github.io/container-camp/docker/compose/ --- title: "Multi-container applications with Docker Compose" description: "Define and run Jupyter and RStudio together from one Compose file, manage the stack with docker compose, and deploy WebODM." type: Lesson tags: - Docker - Docker Compose generated: by: "claude/opus-5" at: "2026-09-11T00:00:00Z" sources: - id: cyverse-container-camp resource: "https://github.com/CyVerse-learning-materials/container-camp/blob/1746ee8873bcbed2bd3e62500262a9e9571cff16/docs/docker/compose.md" title: "CyVerse Container Camp: docs/docker/compose.md" author: "team:cyverse" last_modified: "2022-05-12T19:53:24-07:00" --- # Multi-container applications with Docker Compose [:material-docker: Docker Compose](https://docs.docker.com/compose){target=_blank} is an extension of Docker which allows you to run multiple containers synchronously and in communication with one another. Compose allows you to define and run a multi-container service using a `Dockerfile` and a `docker-compose.yml`. !!! note "Installing Compose" Compose v2 runs as a Docker CLI plugin: you type `docker compose` (with a space). It is included with Docker Desktop on macOS and Windows. On Linux, install the `docker-compose-plugin` package from Docker's package repository, for example on Ubuntu: ``` bash sudo apt-get install docker-compose-plugin docker compose version ``` The older standalone `docker-compose` (v1) command is no longer maintained, so the commands on this page use `docker compose`. Main advantages of Docker compose include: - Your applications can be defined in a YAML file where all the same options required in `docker run` are now defined (reproducibility). - It allows you to manage your application(s) as a single entity rather than dealing with starting individual containers (simplicity). !!! note For the sake of this example, either create your own `Dockerfile` or use the same Jupyter SciPy Notebook as in the [Advanced Section](https://unm-carc.github.io/container-camp/docker/advanced/) --- ## Creating a `docker-compose.yml` Let's now create a Docker Compose `.yml` that calls Jupyter Notebook and RStudio 1\. Create a folder `shared_data` in your current directory: ``` mkdir shared_data ``` 2\. Create an empty `docker-compose.yml` file (e.g., touch docker-compose.yml) and paste the following lines. Compose v2 ignores the top-level `version:` key found in older examples, so it is omitted here. ``` # All available services services: # Computation jupyter: container_name: "jupyter_notebook" image: "jupyter/minimal-notebook" restart: "always" environment: - JUPYTER_TOKEN=mytoken user: root volumes: - ./data:/home/jovyan/work/ ports: - 8888:8888 rstudio: container_name: "rstudio" image: "rocker/rstudio" restart: "always" environment: - DISABLE_AUTH=true volumes: - ./data:/home/rstudio ports: - 8787:8787 ``` 3\. Run both Jupyter Lab and RStudio using `docker compose up` instead of `docker run`. !!! note Handling containers with Docker Compose is fairly simple ``` bash docker compose up ``` attaches the volumes, opens ports, and starts the container ``` bash docker compose down ``` destroys the container A brief explanation of `docker-compose.yml` is as below: - The web service builds from the Dockerfile in the current directory. - Forwards the container's exposed port to port 8888 on the host. - Mounts the project directory on the host to `/work` or `/rstudio` inside the container (allowing you to modify code without having to rebuild the image). - `restart: always` means that it will restart whenever it fails. --- ## Running, shutting down, and restarting with `docker compose` Run the containers with ``` $ docker compose up -d ``` To stop a running `docker compose` session, either press `CTRL + C` or use the command: ``` docker compose down ``` The above command removes containers, networks, volumes and images created by `docker compose up`. To restart a container, use the command ``` docker compose restart ``` `restart` will restart the Compose services *without* taking into account changes one may have made to the `yml` or environment. --- ## Example using Docker Compose: WebODM !!! warning For the purpose of these following examples it is not suggested to use GitHub Codespaces. [:material-quadcopter: Web Open Drone Map (WebODM)](https://github.com/OpenDroneMap/WebODM/#run-it-on-the-cloud-google-compute-amazon-aws){target=_blank} OpenDroneMap is an open source photogrammetry toolkit to process aerial imagery into maps and 3D models running on command line. WebODM (Web OpenDroneMap) is an extension of ODM running on multiple Docker Containers provinding a user friendly web interface for easy visualization. To use WebODM: !!! note "Prerequisites" WebODM requires `docker` and Docker Compose to function. Additionally, if you are on Windows, users will be required to have the [Docker Windows Application](https://docs.docker.com/desktop/windows/install/){target=_blank} installed as well as having the [WSL2](https://docs.microsoft.com/en-us/windows/wsl/install){target=_blank} (Windows Subsystem for Linux) operational. 1. Ensure your machine is up to date: `sudo apt-get update` 2. Clone the WebODM repository: `git clone https://github.com/OpenDroneMap/WebODM --config core.autocrlf=input --depth 1` 3. Move into the WebODM folder: `cd WebODM` 4. Run WebODM: `sudo ./webodm.sh start` 5. The necessary docker images will be downloaded (~2 minutes) and WebODM will be accessible through http://localhost:8000/ !!! note You will be asked to create an account as a formality. Add any *username* and a *password* and select **Create Account**. 6\. Download example data: `git clone https://github.com/OpenDroneMap/odm_data_aukerman.git`. This git repository contains 77 HD images usable for WebODM. For other examples refer to [ODMData](https://www.opendronemap.org/odm/datasets/){target=_blank}. 7\. In the WebODM portal, click on **Select Images and GCP**, navigate to `odm_data_aukerman/images` and select between 20-50 images (16 is the absolute minimum, whilst 32 is the suggested minimum). ![webodm_1](https://unm-carc.github.io/container-camp/assets/docker/WebODM_01.png) 8\. WebODM will process the uploaded images (~5-10 minutes); upon completion, click **View Map**. ![webodm_2](https://unm-carc.github.io/container-camp/assets/docker/WebODM_02.png) 9\. A map will open; you can click on **3D** (bottom right) to see the 3D rendered model generated. ![webodm_3](https://unm-carc.github.io/container-camp/assets/docker/WebODM_03.png)

Adapted from [CyVerse Container Camp](https://github.com/CyVerse-learning-materials/container-camp/blob/1746ee8873bcbed2bd3e62500262a9e9571cff16/docs/docker/compose.md){target=_blank} (last source update 2022-05-12), CC BY 4.0. Spotted a problem? [Open an issue](https://github.com/UNM-CARC/container-camp/issues){target=_blank}.

---8<--- https://unm-carc.github.io/container-camp/singularity/intro/ --- title: "Introduction to Apptainer/Singularity" description: "Install Apptainer or SingularityCE and use the CLI to pull, shell, exec, run, and inspect container images, including images from Docker Hub." type: Lesson tags: - Apptainer - Singularity - Containers - HPC generated: by: "claude/opus-5" at: "2026-09-11T00:00:00Z" sources: - id: cyverse-container-camp resource: "https://github.com/CyVerse-learning-materials/container-camp/blob/1746ee8873bcbed2bd3e62500262a9e9571cff16/docs/singularity/intro.md" title: "CyVerse Container Camp: docs/singularity/intro.md" author: "team:cyverse" last_modified: "2023-03-07T20:36:53-07:00" --- # Introduction to Apptainer/Singularity In this section we work with [Apptainer](https://apptainer.org/){target=_blank} and [Singularity Community Edition (SingularityCE)](https://sylabs.io/guides/3.9/user-guide/index.html){target=_blank}. Apptainer is the Linux Foundation continuation of the original Singularity project; it still installs `singularity` as an alias of the `apptainer` command, so the `singularity ...` commands in these lessons work with either runtime. SingularityCE is the fork maintained by Sylabs. ??? question "Wait, what is "Apptainer", and what is the difference between SingularityCE and Apptainer?" The Singularity project was split into multiple projects managed by different organizations since it was created in 2017. In a nutshell: - Greg Kurtzer founded the Singularity project while at the Lawrence Berkeley National Laboratory - Kurtzer created [Sylabs](https://sylabs.io/){target=_blank}, a private company, around Singularity - Kurtzer left Sylabs to focus on [CIQ](https://ciq.co/){target=_blank}, another private company, and moved Singularity to [HPCng](https://hpcng.org/){target=_blank} (a Community Org) - [Sylabs forked Singularity](https://sylabs.io/singularityce-community-update/){target=_blank} for control and professionially licensed support creating Singularity Community Edition" - HPCng gave the official project to Linux Foundation and renamed it "[Apptainer](https://apptainer.org/){target=_blank}" - Apptainer is being [marketed by CIQ](https://ciq.co/apptainer/){target=_blank} At the present time, Apptainer and SingularityCE have highly similar syntax and will run Singularity `.sif` images interoperably. Apptainer keeps `singularity` as a command alias, so existing scripts keep working. ??? info "Docker vs SingularityCE & Apptainer" **:material-open-source-initiative: Apptainer and SingularityCE are 100% compatible with Docker but they do have some distinct differences** **:material-docker: Docker** :octicons-container-24: Docker containers run as `root` - This privilege is almost never supported by administrators of High Performance Computing (HPC) centers. Meaning Docker is not, and will likely never be, installed natively on your HPC cluster. :octicons-container-24: uses compressed layers to create one image **:material-open-source-initiative: SingularityCE & Apptainer**: :octicons-container-24: Same user and group identity inside as outside the container :octicons-container-24: User only has `root` privileges if elevated with `sudo` when the container is run :octicons-container-24: Can run and modify any existing Docker image - These key differences allow Singularity to be installed on most HPC centers. Because you can run virtually all Docker containers in Singularity, you can effectively run Docker on an HPC. ## Installing Apptainer or SingularityCE Sylabs Singularity Community Edition (CE) homepage: [https://www.sylabs.io/docs/](https://www.sylabs.io/docs/){target=_blank} Apptainer Linux Foundation homepage: [https://apptainer.org/](https://apptainer.org/){target=_blank} ### Conda [SingularityCE](https://anaconda.org/conda-forge/singularityce){target=_blank} or [Apptainer](https://anaconda.org/conda-forge/apptainer){target=_blank} can both be installed from Conda: ``` conda install -c conda-forge singularityce ``` ``` conda install -c conda-forge apptainer ``` ### Install Locally To Install Singularity follow the instructions for your specific OS: [:simple-github: SingularityCE](https://github.com/sylabs/singularity/blob/main/INSTALL.md){target=_blank} [:material-book-open-variant: Apptainer](https://apptainer.org/docs/admin/main/installation.html){target=_blank} ### Module loading on HPC If you are interested in working with SingularityCE on HPC, you may need to contact your systems administrator and request they install [SingularityCE](https://sylabs.io/guides/3.9/user-guide/quick_start.html#installation-request){target=_blank}. Because SingularityCE ideally needs setuid, your admins may have some qualms about giving SingularityCE this privilege. If that is the case, you might consider forwarding [this letter](https://sylabs.io/guides/3.9/user-guide/quick_start.html#Singularity-on-a-shared-resource){target=_blank} to your admins. Most HPC systems are running Environment Modules with the simple command `module`. You can check to see what is available: ``` $ module avail singularity ``` If Singularity is listed as being installed, load a specific version, e.g.: ``` $ module load singularity/3/3.9 ``` On some systems the module is named `apptainer` instead (`module avail apptainer`). !!! tip "UNM CARC users" The CARC documentation has a guide to [running Singularity/Apptainer containers on CARC clusters](https://carc.unm.edu/docs/software/singularity/){target=_blank}. Run containers inside Slurm jobs rather than on the login nodes; see the [CARC introduction to Slurm](https://carc.unm.edu/docs/running-jobs/slurm-intro/){target=_blank}. ### Install in CodeSpaces Let's use Conda (or optionally, Mamba) ``` conda install -c conda-forge singularityce mamba install -c conda-forge singularityce ``` ## Apptainer and Singularity CLI Singularity’s [command line interface](https://sylabs.io/guides/3.9/user-guide/cli.html){target=_blank} allows you to build and interact with containers transparently. You can run programs inside a container as if they were running on your host system. You can easily redirect IO, use pipes, pass arguments, and access files, sockets, and ports on the host system from within a container. ### :octicons-container-24: help The `help` command gives an overview of Singularity options and subcommands as follows: ``` $ singularity help pull Pull an image from a URI Usage: singularity pull [pull options...] [output file] Description: The 'pull' command allows you to download or build a container from a given URI. Supported URIs include: library: Pull an image from the currently configured library library://user/collection/container[:tag] docker: Pull a Docker/OCI image from Docker Hub, or another OCI registry. docker://user/image:tag shub: Pull an image from Singularity Hub shub://user/image:tag oras: Pull a SIF image from an OCI registry that supports ORAS. oras://registry/namespace/image:tag http, https: Pull an image using the http(s?) protocol https://library.sylabs.io/v1/imagefile/library/default/alpine:latest Options: --arch string architecture to pull from library (default "amd64") --dir string download images to the specific directory --disable-cache dont use cached images/blobs and dont create them --docker-login login to a Docker Repository interactively -F, --force overwrite an image file if it exists -h, --help help for pull --library string download images from the provided library --no-cleanup do NOT clean up bundle after failed build, can be helpful for debugging --no-https use http instead of https for docker:// oras:// and library:///... URIs Examples: From Sylabs cloud library $ singularity pull alpine.sif library://alpine:latest From Docker $ singularity pull tensorflow.sif docker://tensorflow/tensorflow:latest From Shub $ singularity pull singularity-images.sif shub://vsoch/singularity-images From supporting OCI registry (e.g. Azure Container Registry) $ singularity pull image.sif oras://.azurecr.io/namespace/image:tag For additional help or support, please visit https://www.sylabs.io/docs/ ``` ### :octicons-container-24: search Just like with Docker, you can `search` the Singularity container registries for images. ``` singularity search tensorflow ``` ### :octicons-container-24: pull The easiest way to use a Singularity is to `pull` an existing container from one of the Registries. ``` singularity pull library://lolcow ``` ``` singularity pull docker:///cowsay ``` #### Downloading pre-built images You can use the `pull` command to download pre-built images from a number of Container Registries, here we'll be focusing on the [Singularity-Hub](https://www.singularity-hub.org){target=_blank} or [DockerHub](https://hub.docker.com/){target=_blank}. Container Registries: - `library://` - images hosted on Sylabs Cloud - `docker://` - images hosted on Docker Hub - `localimage://` - images saved on your machine - `yum://` - yum based systems such as CentOS and Scientific Linux - `debootstrap://` - apt based systems such as Debian and Ubuntu - `arch://` - Arch Linux - `busybox://` - BusyBox - `zypper://` - zypper based systems such as Suse and OpenSuse - `shub://` - (archived) images hosted on Singularity Hub, no longer maintained ### Pulling an image from Singularity Hub Similar to previous example, in this example I am pulling a base Ubuntu container from Singularity-Hub: ``` $ singularity pull shub://singularityhub/ubuntu WARNING: Authentication token file not found : Only pulls of public images will succeed 88.58 MiB / 88.58 MiB [===============================================================================================] 100.00% 31.86 MiB/s 2s ``` You can rename the container using the --name flag: ``` $ singularity pull --name ubuntu_test.simg shub://singularityhub/ubuntu WARNING: Authentication token file not found : Only pulls of public images will succeed 88.58 MiB / 88.58 MiB [===============================================================================================] 100.00% 35.12 MiB/s 2s ``` The above command will save the alpine image from the Container Library as `alpine.sif` ### Pulling an image from Docker Hub This example pulls an `ubuntu:16.04` image from DockerHub and saves it to the working directory. ``` $ singularity pull docker://ubuntu:20.04 INFO: Converting OCI blobs to SIF format INFO: Starting build... Getting image source signatures Copying blob 8f6b7df711c8 done Copying blob 0703c52b8763 done Copying blob 07304348ce1b done Copying blob 4795dceb8869 done Copying config 05ac933964 done Writing manifest to image destination Storing signatures 2020/03/09 16:14:12 info unpack layer: sha256:8f6b7df711c8a4733138390ff2aba1bfeb755bf4736c39c6e4858076c40fb5eb 2020/03/09 16:14:13 info unpack layer: sha256:0703c52b8763604318dcbb1730c82ad276a487335ecabde2f43f69a6222e8090 2020/03/09 16:14:13 info unpack layer: sha256:07304348ce1b6d24f136a3c4ebaa800297b804937a6942ce9e9fe0dac0b0ca74 2020/03/09 16:14:13 info unpack layer: sha256:4795dceb8869bdfa64f3742e1df492e6f31baf9cfc36f1a042a8f981607e99a2 INFO: Creating SIF file... INFO: Build complete: ubuntu_20.04.sif ``` !!! warning Pulling Docker images reduces reproducibility. If you were to pull a Docker image today and then wait six months and pull again, you are not guaranteed to get the same image. If any of the source layers has changed the image will be altered. If reproducibility is a priority for you, try building your images from the Container Library. ### Pulling an image from Sylabs cloud library Let’s use an easy example of `alpine.sif` image from the [container library](https://cloud.sylabs.io/library/) !!! tip You can use `singularity search ` command to locate groups, collections, and containers of interest on the Container Library ## Interact with images You can interact with images in several ways such as `shell`, `exec` and `run`. For these examples we will use a `cowsay_latest.sif` image that can be pulled from the Docker Hub. ``` $ singularity pull docker://tswetnam/cowsay INFO: Downloading library image 67.00 MiB / 67.00 MiB [=====================================================================================================] 100.00% 5.45 MiB/s 12s WARNING: unable to verify container: cowsay_latest.sif WARNING: Skipping container verification $ sudo singularity run cowsay_latest.sif ________________________________________ / Expect a letter from a friend who will \ \ ask a favor of you. / ---------------------------------------- \ ^__^ \ (oo)\_______ (__)\ )\/\ ||----w | || || ``` ### :octicons-container-24: shell The `shell` command allows you to spawn a new shell within your container and interact with it as though it were a small virtual machine. ``` $ singularity shell cowsay_latest.sif Singularity cowsay_latest.sif:~> ``` The change in prompt indicates that you have entered the container (though you should not rely on that to determine whether you are in container or not). Once inside of a Singularity container, you are the same user as you are on the host system. ``` $ Singularity cowsay_latest.sif:~> whoami tswetnam ``` !!! warning `shell` also works with the library://, docker://, and shub:// URIs. This creates an ephemeral container that disappears when the shell is exited. ### :octicons-container-24: exec The exec command allows you to execute a custom command within a container by specifying the image file. For instance, to execute the `cowsay` program within the cowsay\_latest.sif container: ``` $ singularity exec cowsay_latest.sif cowsay container camp rocks ______________________ < container camp rocks > ---------------------- \ ^__^ \ (oo)\_______ (__)\ )\/\ ||----w | || || ``` `exec` also works with the library://, docker://, and shub:// URIs. ``` singularity exec library://lolcow container camp 2022 ``` This creates an ephemeral container that executes a command and disappears. ### :octicons-container-24: run Singularity containers contain [runscripts](https://www.sylabs.io/guides/3.0/user-guide/definition_files.html#runscript){target=_blank}. These are user defined scripts that define the actions a container should perform when someone runs it. The runscript can be triggered with the `run` command, or simply by calling the container as though it were an executable. ``` singularity run lolcow_latest.sif _________________________________________ / You will remember, Watson, how the \ | dreadful business of the Abernetty | | family was first brought to my notice | | by the depth which the parsley had sunk | | into the butter upon a hot day. | | | \ -- Sherlock Holmes / ----------------------------------------- \ ^__^ \ (oo)\_______ (__)\ )\/\ ||----w | || || ``` ### :octicons-container-24: inspect The `inspect` command will provide information about labels, metadata, and environmental variables. ``` singularity inspect lolcow.sif ``` ``` singularity inspect library://lolcow ``` ### :octicons-container-24: build See [Next Section](https://unm-carc.github.io/container-camp/singularity/advanced/) for details of `build`

Adapted from [CyVerse Container Camp](https://github.com/CyVerse-learning-materials/container-camp/blob/1746ee8873bcbed2bd3e62500262a9e9571cff16/docs/singularity/intro.md){target=_blank} (last source update 2023-03-07), CC BY 4.0. Spotted a problem? [Open an issue](https://github.com/UNM-CARC/container-camp/issues){target=_blank}.

---8<--- https://unm-carc.github.io/container-camp/singularity/advanced/ --- title: "Building Apptainer/Singularity images" description: "Build SIF images from Docker images, sandboxes, and definition files, with the %post, %environment, %runscript, and %labels sections explained." type: Lesson tags: - Apptainer - Singularity - Definition Files generated: by: "claude/opus-5" at: "2026-09-11T00:00:00Z" sources: - id: cyverse-container-camp resource: "https://github.com/CyVerse-learning-materials/container-camp/blob/1746ee8873bcbed2bd3e62500262a9e9571cff16/docs/singularity/advanced.md" title: "CyVerse Container Camp: docs/singularity/advanced.md" author: "team:cyverse" last_modified: "2022-05-16T18:42:04-07:00" --- # Building Apptainer/Singularity images ## Apptainer and Singularity CLI continued ### Build Singularity `.sif` images Similar to Docker which uses a `Dockerfile` to build its images, Singularity uses a file called `Singularity`. Apptainer and SingularityCE call this file a *definition file*; it is often named `Singularity` or given a `.def` extension. **Important** * Singularity images use the `.sif` extension and appear as a single compressed file, versus Docker which uses many cached file layers to create a single image. * `.sif` files also use layers, but these are not apparent. * `.sif` images are cached in the folder where you build them, or designate them. * When building from `docker://` the Docker image layers are downloaded and cached by Singularity in a `/.Singularity` folder on your build environment. ### :octicons-container-24: build ### Create a `.sif` image using a Docker image as the template As we've learned from the HPC and HTC groups, building Singularity images is not necessarily the most accessible and reproducible method for managing containers. Most groups suggest that you build your containers with Docker, and host them on a Docker Registry first. The market dominance of Docker and its wide acceptance as a container format, has led us to use Singularity with Docker in most cases. We've already covered how you can pull an existing image from Docker Hub, but we can also build a Singularity image from the Docker Hub using the build command: ``` $ sudo singularity build --sandbox ubuntu-latest/ docker://ubuntu ``` Test the new `.sif` image: ``` $ singularity shell --writable ubuntu-latest/ Singularity ubuntu-latest.sif:~> apt-get update ``` Does it work? ``` $ sudo singularity shell ubuntu-latest.sif Singularity: Invoking an interactive shell within container... Singularity ubuntu-latest.sif:~> apt-get update ``` When I try to install software to the image without `sudo` it is denied, because `root` is the owner inside the container. When I use `sudo` I can install software into the container. The software remains in the sandbox container after closing the container and restart. In order to make these changes permanant, I need to rebuild the sandbox as a `.sif` image ``` $ sudo singularity build ubuntu-latest.sif ubuntu-latest/ ``` ## Creating Singularity `.sif` from scratch The contents of the `Singularity` file differ from `Dockerfile` - `%help` - create text for a help menu associated with your container - `%setup` - executed on the host system outside of the container, after the base OS has been installed. - `%files` - copy files from your host system into the container - `%labels` - store metadata in the container - `%environment` - loads environment variables at the time the container is run (not built) - `%post` - set environment variables during the build - `%runscript` - executes a script when the container runs - `%test` - runs a test on the build of the container | Dockerfile | FROM | ARG | COPY | RUN | ENV | LABEL | CMD | |--------|------|-----|------|-----|-----|-------|-----| | Singularity | `Bootstrap:` | `%post` | `%files` | `%post` | `%environment` | `%label` | `%runscript` | ### Writing the `Singularity` file [SyLabs User-Guide](https://sylabs.io/guides/3.9/user-guide/){target=_blank} A `Singularity` file can be hosted on GitHub next to the code it packages. (Singularity Hub, which once built these files automatically, has been retired; build images yourself or in a CI service such as [GitHub Actions](https://unm-carc.github.io/container-camp/docker/actions/).) When you are building locally, you can name the `Singularity` file whatever you wish, but a better practice is to put it in a specified directory and name it `Singularity`. Building your own containers requires that you have sudo privileges - therefore you'll need to develop these on your local machine or on a VM that you can gain root access on. #### :material-layers-edit: **Header** The top of the file, selects the base OS for the container, just like `FROM` in Docker. `Bootstrap:` references another registry (e.g. `docker` for DockerHub, `debootstrap`, or `library` for Sylabs [Container Library](https://cloud.sylabs.io/library){target=_blank}). `From:` selects the tag name. Using `debootstrap` with a build that uses a mirror: ``` BootStrap: debootstrap OSVersion: jammy MirrorURL: http://us.archive.ubuntu.com/ubuntu/ ``` Using CentOS-like container: ``` Bootstrap: yum OSVersion: 7 MirrorURL: http://mirror.centos.org/centos-7/7/os/x86_64/ Include:yum ``` Note: to use yum to build a container you should be operating on a RHEL system, or an Ubuntu system with yum installed. Using a localimage to build: ``` Bootstrap: localimage From: /path/to/container/file/or/directory ``` The container registries which Singularity uses are listed in the [introduction](https://unm-carc.github.io/container-camp/singularity/intro/#downloading-pre-built-images). - The Singularity file uses sections to specify the dependencies, environmental settings, and runscripts when it builds. #### :material-layers-edit: **help** `%help` section can be as verbose as you want ``` Bootstrap: docker From: ubuntu %help This is the container help section. ``` #### :material-layers-edit: **setup** `%setup` commands are executed on the localhost system outside of the container - these files could include necessary build dependencies. We can copy files to the `$SINGULARITY_ROOTFS` file system can be done during setup #### :material-layers-edit: **files** `%files` include any files that you want to copy from your localhost into the container. #### :material-layers-edit: **post** `%post` includes all of the environment variables and dependencies that you want to see installed into the container at build time. ``` %post apt-get -y update apt-get -y install fortune cowsay lolcat ``` #### :material-layers-edit: **environment** `%environment` includes the environment variables which we want to be run when we start the container ``` %environment export LC_ALL=C export PATH=/usr/games:$PATH ``` #### :material-layers-edit: **runscript** `%runscript` does what it says, it executes a set of commands when the container is run. ``` %runscript fortune | cowsay | lolcat ``` #### :material-layers-edit: **labels** `%labels` are used similar to Dockerfile which allow you to add metadata to the image file in key-value pairs ``` %labels Author Your Name Email email@address.org Version v2022 CustomLabel statement here ``` Labels can be read using the `inspect` command #### Example File Example `Singularity` file bootstrapping an Ubuntu (22.04) image. ``` Bootstrap: docker From: ubuntu:22.04 %post apt-get -y update apt-get -y install fortune cowsay lolcat %environment export LC_ALL=C export PATH=/usr/games:$PATH %runscript fortune | cowsay | lolcat %labels Maintainer Your Name Version v2022 ``` Build the container: ``` singularity build cowsay.sif Singularity ``` Run the container: ``` singularity run cowsay.sif ``` ## Sandbox Sandboxing is another approach to building up a container image.

Adapted from [CyVerse Container Camp](https://github.com/CyVerse-learning-materials/container-camp/blob/1746ee8873bcbed2bd3e62500262a9e9571cff16/docs/singularity/advanced.md){target=_blank} (last source update 2022-05-16), CC BY 4.0. Spotted a problem? [Open an issue](https://github.com/UNM-CARC/container-camp/issues){target=_blank}.

---8<--- https://unm-carc.github.io/container-camp/singularity/hpc/ --- title: "Apptainer/Singularity on HPC" description: "Run containers through a Slurm batch scheduler, match MPI versions between host and container, and use GPUs with the --nv flag." type: Lesson tags: - Apptainer - HPC - Slurm - MPI - GPU generated: by: "claude/opus-5" at: "2026-09-11T00:00:00Z" sources: - id: cyverse-container-camp resource: "https://github.com/CyVerse-learning-materials/container-camp/blob/1746ee8873bcbed2bd3e62500262a9e9571cff16/docs/singularity/hpc.md" title: "CyVerse Container Camp: docs/singularity/hpc.md" author: "team:cyverse" last_modified: "2022-04-12T17:50:10-07:00" --- # Apptainer/Singularity on HPC High Performance Computing resources fill an important role in research computing and can support container execution through runtimes such as Apptainer/Singularity, among other options. Conducting analyses on HPC clusters happens through different patterns of interaction than running analyses on a cloud VM. When you login, you are on a node that is shared with lots of people, typically called the "login node". Trying to run jobs on the login node is not "high performance" at all (and will likely get you an admonishing email from the system administrator). Login nodes are intended to be used for moving files, editing files, and launching jobs. Importantly, most jobs run on an HPC cluster are neither **interactive**, nor **realtime**. When you submit a job to the scheduler, you must tell it what resources you need (e.g. how many nodes, how much RAM, what type of nodes, and for how long) in addition to what you want to run. Then the scheduler finally has resources matching your requirements, it runs the job for you. If your request is very large, or very long, you may never make it out of the queue. For example, on a VM if you run the command: ``` singularity exec docker://python:latest /usr/local/bin/python ``` The container will immediately start. On an HPC system, your job submission script would look something like: ``` #!/bin/bash # #SBATCH -J myjob # Job name #SBATCH -o output.%j # Name of stdout output file (%j expands to jobId) #SBATCH -p development # Queue name #SBATCH -N 1 # Total number of nodes requested (68 cores/node) #SBATCH -n 17 # Total number of mpi tasks requested #SBATCH -t 02:00:00 # Run time (hh:mm:ss) - 4 hours module load singularity/3/3.1 singularity exec docker://python:latest /usr/local/bin/python ``` This example is for the Slurm scheduler. Each of the \#SBATCH lines looks like a comment to the bash kernel, but the scheduler reads all those lines to know what resources to reserve for you. It is usually possible to get an interactive session as well, by using an interactive flag, -i. !!! tip "UNM CARC users" On CARC clusters, run container workloads through Slurm as shown above. The [CARC Singularity guide](https://carc.unm.edu/docs/software/singularity/){target=_blank} covers the container runtime on CARC systems, and the [CARC introduction to Slurm](https://carc.unm.edu/docs/running-jobs/slurm-intro/){target=_blank} explains CARC partitions and resource requests. !!! warning Every HPC cluster is a little different, but they almost universally have a "User's Guide" that serves both as a quick reference for helpful commands and contains guidelines for how to be a "good citizen" while using the system. For TACC's Stampede2 system, see the [user guide](https://portal.tacc.utexas.edu/user-guides/stampede2){target=_blank}. For The University of Arizona, see the [user guide](https://docs.hpc.arizona.edu/){target=_blank}. ## How do HPC systems fit into the development workflow? A few things to consider when using HPC systems: 1. Using `sudo` is not allowed on HPC systems, and building a Singularity container from scratch requires sudo. That means you have to build your containers on a different development system. You can pull a docker image on HPC systems 2. If you need to edit text files, command line text editors don't support using a mouse, so working efficiently has a learning curve. There are text editors that support editing files over SSH. This lets you use a local text editor and just save the changes to the HPC system. These constraints make HPC systems perfectly suitable for execution environments, but currently a limiting choice for a development environment. We usually recommend your local laptop or a VM as a development environment where you can iterate on your code rapidly and test container building and execution. ## Singularity and MPI Singularity supports MPI fairly well. Since (by default) the network is the same insde and outside the container, the communication between containers usually just works. The more complicated bit is making sure that the container has the right set of MPI libraries. MPI is an open specification, but there are several implementations (OpenMPI, MVAPICH2, and Intel MPI to name three) with some non-overlapping feature sets. If the host and container are running different MPI implementations, or even different versions of the same implementation, hilarity may ensue. The general rule is that you want the version of MPI inside the container to be the same version or newer than the host. You may be thinking that this is not good for the portability of your container, and you are right. Containerizing MPI applications is not terribly difficult with Singularity, but it comes at the cost of additional requirements for the host system. !!! warning Many HPC Systems, like Stampede2 at TACC and Ocelote at UAHPC, have high-speed, low-latency networks that have special drivers. Infiniband, Ares, and OmniPath are three different specs for these types of networks. When running MPI jobs, if the container doesn't have the right libraries, it won't be able to use those special interconnects to communicate between nodes. ## Base Docker images Depending on the system you will use, you may have to build your own MPI enabled Singularity images (to get the versions to match). When running at TACC, there is a set of curated Docker images for use in the FROM line of your own containers. You can see a list of availabe images at Specifically, you can use the `tacc/tacc-ubuntu18-mvapich2.3-psm2` image to satisfy the MPI architecture and version requirements for running on Stampede2. Because you may have to build your own MPI enabled Singularity images (to get the versions to match), here is a 3.1 compatible example of what it may look like: ``` BootStrap: debootstrap OSVersion: xenial MirrorURL: http://us.archive.ubuntu.com/ubuntu/ %runscript echo "This is what happens when you run the container..." %post echo "Hello from inside the container" sed -i 's/$/ universe/' /etc/apt/sources.list apt update apt -y --allow-unauthenticated install vim build-essential wget gfortran bison libibverbs-dev libibmad-dev libibumad-dev librdmacm-dev libmlx5-dev libmlx4-dev wget http://mvapich.cse.ohio-state.edu/download/mvapich/mv2/mvapich2-2.1.tar.gz tar xvf mvapich2-2.1.tar.gz cd mvapich2-2.1 ./configure --prefix=/usr/local make -j4 make install /usr/local/bin/mpicc examples/hellow.c -o /usr/bin/hellow ``` You could also build in everything in a Dockerfile and convert the image to Singularity at the end. Once you have a working MPI container, invoking it would look something like: ``` mpirun -np 4 singularity exec ./mycontainer.sif /app.py arg1 arg2 ``` This will use the **host MPI** libraries to run in parallel, and assuming the image has what it needs, can work across many nodes. For a single node, you can also use the **container MPI** to run in parallel (usually you don't want this) ``` singularity exec ./mycontainer.sif mpirun -np 4 /app.py arg1 arg2 ``` ## Example Containerized MPI App In your Docker development environment, make a new directory in which to build up a new image and download (or copy and paste) two files in that directory: Take a look at both files. `pi-mpi.py` is a simple MPI Python script that approximates pi (very inefficiently) by random sampling. `Dockerfile.mpi` is an updated Dockerfile that uses the TACC base image to satisfy all the MPI requirements on Stampede2. Next, try building the new container. ``` $ docker build -t USERNAME/pi-estimator:0.1-mpi -f Dockerfile.mpi . ``` Don't forget to change USERNAME to your DockerHub username. Once you have successfully built an image, push it up to DockerHub with the `docker push` command so that we can pull it back down on Stampede2. ## Running an MPI Container on Stampede2 To test, we can grab an interactive session that has two nodes. That way we can see if we can make the two nodes work together. On TACC systems, the "idev" command will start an interactive session on a compute node: ``` $ idev -m 60 -p normal -N 2 -n 128 ``` Once you have nodes at your disposal and a container on DockerHub, invoking it would look something like: ``` module load tacc-singularity cd $WORK singularity pull docker://USERNAME/pi-estimator:0.1-mpi time singularity exec pi-estimator_0.1-mpi.sif pi-mpi.py 10000000 time ibrun singularity exec pi-estimator_0.1-mpi.sif pi-mpi.py 10000000 ``` !!! warning TACC uses a command called `ibrun` on all of its systems that configures MPI to use the high-speed, low-latency network. If you are familiar with MPI, this is the functional equivalent to `mpirun` The first `singularity exec pi-estimator_0.1-mpi.sif pi-mpi.py 10000000` command will use 1 CPU core to sample ten million times. The second command, using `ibrun` will run 128 processes that sample ten million times each and pass their results back to the "rank 0" MPI process to merge the results. This will use the **host MPI** libraries to run in parallel, and assuming the image has what it needs, can work across many nodes. As an aside, for a single node you can also use the **container MPI** to run in parallel (but usually you don't want this). When you are don with your interactive session, don't forget to `exit` to end the session and go back to the login node. ## Singularity and GPU Computing GPU support in Singularity is very good. Since Singularity supported docker containers, it has been fairly simple to utilize GPUs for machine learning code like TensorFlow. We will not do this as a hands-on exercise, but in general the procedule is as follows. ``` # Load the singularity module module load singularity/3/3.1 # Pull your image singularity pull docker://nvidia/caffe:latest singularity exec --nv caffe-latest.sif caffe device_query -gpu 0 ``` Please note that the --nv flag specifically passes the GPU drivers into the container. If you leave it out, the GPU will not be detected. ``` # this is missing the --nv flag and will not work singularity exec caffe-latest.sif caffe device_query -gpu 0 ``` The main requirement for GPU containers to work is that the version of the host drivers matches the major version of the library inside the container. So, for example, if CUDA 10 is on the host, the container needs to use CUDA 10 internally. For TensorFlow, you can directly pull their latest GPU image and utilize it as follows. ``` # Change to your $WORK directory cd $WORK #Get the software git clone https://github.com/tensorflow/models.git ~/models # Pull the image singularity pull docker://tensorflow/tensorflow:latest-gpu # Run the code singularity exec --nv tensorflow-latest-gpu.sif python $HOME/models/tutorials/image/mnist/convolutional.py ``` The University of Arizona HPS [Singularity examples](https://docs.hpc.arizona.edu/display/UAHPC/Containers){target=_blank}.

Adapted from [CyVerse Container Camp](https://github.com/CyVerse-learning-materials/container-camp/blob/1746ee8873bcbed2bd3e62500262a9e9571cff16/docs/singularity/hpc.md){target=_blank} (last source update 2022-04-12), CC BY 4.0. Spotted a problem? [Open an issue](https://github.com/UNM-CARC/container-camp/issues){target=_blank}.

---8<--- https://unm-carc.github.io/container-camp/orchestration/kubernetes/ --- title: "Introduction to Kubernetes" description: "Kubernetes concepts and terminology, connecting to an existing K3s cluster with kubectl, and launching pods, deployments, and the dashboard." type: Lesson tags: - Kubernetes - kubectl - K3s generated: by: "claude/opus-5" at: "2026-09-11T00:00:00Z" sources: - id: cyverse-container-camp resource: "https://github.com/CyVerse-learning-materials/container-camp/blob/1746ee8873bcbed2bd3e62500262a9e9571cff16/docs/orchestration/k8s.md" title: "CyVerse Container Camp: docs/orchestration/k8s.md" author: "team:cyverse" last_modified: "2023-08-15T20:54:48-07:00" --- # Introduction to Kubernetes !!! info "Learning Objectives" * Understand what Kubernetes is and its role in managing containerized applications * Recognize the importance and benefits of using Kubernetes in cloud computing * Identify reasons why you may need to use Kubernetes for your own research or projects * Learn about the existence of Kubernetes clusters and how to leverage them using `kubectl` [:material-kubernetes: Kubernetes](https://kubernetes.io/docs/concepts/overview/what-is-kubernetes/){target=_blank}, or `K8s` for short, is an open-source system designed to automate the deployment, scaling, and management of containerized applications. K8s is the most prevalent platform for managing containerized applications at scale in the realm of cloud computing. If you're considering incorporating containers into your research or project portfolio, it's highly likely you have already interacted with or will interact with a K8s cluster in the future. In this introductory lesson, we'll focus on how to leverage *existing* Kubernetes Clusters using `kubectl`. ??? question ":simple-kubernetes: K8s vs :simple-k3s: K3s" We are going to be using a platform developed by a project named [:simple-rancher: Rancher](https://www.rancher.com/){target=_blank}, called [:simple-k3s: K3s](https://k3s.io/){target=_blank}. K3s is a lightweight certified Kubernetes distribution designed for edge and production workloads. K3s uses the same `kubectl` and `helm` commands as full K8s. We prefer K3s to K8s for our virtual machines and for workshops for the same reason you might prefer to use [Alpine Linux](https://www.alpinelinux.org/){target=_blank} to [Ubuntu Linux](https://ubuntu.com/){target=_blank} in a simple container deployment. ??? question "What is container orchestration?" [Container orchestration](https://www.redhat.com/en/topics/containers/what-is-container-orchestration){target=_blank} is the process of automating the deployment, scaling, management, and coordination of containerized applications. In the context of Kubernetes, container orchestration involves managing the lifecycle of containers within a cluster. This includes tasks such as deploying containers, ensuring high availability, distributing network traffic, scaling applications up or down based on demand, and handling updates seamlessly. ??? question ":material-scale-balance: Why would you want to build your own K8s Cluster? " You're here to learn about containerized applications and container orchestration. We aim to familiarize you with these technologies and help you understand their place in the roadmap of Cloud Native Computing. However, it's crucial to note, ⚠️ __Designing, deploying, and maintaining your own K8s cluster can be a complex and challenging task__ ⚠️ - This is partly because there are numerous existing platforms out there, such as managed K8s services. Additionally, managing K8s requires a dedicated DevOps or software engineering team to ensure your platform operates smoothly. Still interested in building your own cluster? Here are some reasons why you might need to use K8s for your research or project: (1) Your applications consist of multiple services. The K8s API automates the tasks of managing numerous containers and allocating resources. (2) Your work scales dynamically - if your computing needs fluctuate based on workloads, K8s can be useful. Using containers to scale your applications is more efficient than manually launching VMs. (3) You have too many containers to manage - K8s excels at its primary function: managing and maintaining containers. (4) Your domain is transitioning to the cloud. If your field is moving towards being cloud-native, it's crucial to develop workflows in anticipation of this shift. (5) Consistency is key. K8s' declarative state provides a clear description of how everything is managed. !!! info "Kubernetes Terminology" **Kubernetes Cluster**: A collection of nodes that run containerized applications. This is the primary unit of organization in Kubernetes. **kubectl**: The command-line interface used for interacting with Kubernetes clusters. It enables users to manage various aspects of the cluster. **Pod**: The smallest deployable unit in Kubernetes, which can consist of one or more containers that communicate with each other. Pods are run on nodes. **Node**: Typically a virtual machine (VM) that runs Kubernetes components, including the kubelet, kube-proxy, and container runtime. **Kubelet**: The agent that runs on each node, responsible for managing the pods on its respective VM. **Kube-proxy**: A network proxy that runs on each node, used by a Service to handle network communication. **Container runtime**: The software that runs containers within pods. Examples include `Docker`, `containerd` (native to Kubernetes), and `CRI-O`. **Service**: Assigns a network address to an application, including a persistent IP address and a DNS entry within the cluster. A Service also manages load balancing across pods and can dynamically add or remove them. **ReplicaSet**: Allows for the creation of multiple pods simultaneously, ensuring that a specified number of identical pods are running at any given time. **Deployment**: Provides declarative updates for Pods and ReplicaSets, allowing for rolling updates and rollbacks. **Control Plane**: A node that manages worker nodes. It consists of the API server, cluster store, controller manager, and scheduler. **Namespace**: A way to divide cluster resources between multiple users. **Ingress**: An API object that manages external access to the services in a cluster, typically HTTP. **Persistent Volume (PV)**: A piece of storage in the cluster that has been provisioned by an administrator. **Persistent Volume Claim (PVC)**: A request for storage by a user. **ConfigMap**: An API object used to store non-confidential data in key-value pairs. **Secret**: An API object used to store sensitive data, like passwords and keys. !!! question "So, what does a K8s Cluster look like?" In short, this: [![k8s cluster](https://d33wubrfki0l68.cloudfront.net/2475489eaf20163ec0f54ddc1d92aa8d4c87c96b/e7c81/images/docs/components-of-kubernetes.svg){target=_blank}](https://kubernetes.io/docs/concepts/overview/components/) The image above is taken from the official [K8s documentation](https://kubernetes.io/docs/concepts/overview/components/){target=_blank} and depics the relation between each component. A K8s cluster comprises of multiple elemets essentially groupable in 2 subsets: - The **Control Plane** components: - **api** (K8s API): the front end of the control plane, it exposes the Kubernentes API. Command line tool: [`kube-apiserver`](https://kubernetes.io/docs/reference/command-line-tools-reference/kube-apiserver/). - **etcd**: a store used for backing cluster data (e.g., cluster cofiguration, state information). Controlled by the [`kube-apiserver`](https://kubernetes.io/docs/reference/command-line-tools-reference/kube-apiserver/) tool. - **sched** (Scheduler): the component that watches for newly created Pods with no assigned node. Once a new Pod is detected, a node is assigned. Controlled by [`kube-scheduler`](https://kubernetes.io/docs/reference/command-line-tools-reference/kube-scheduler/). - **c-m** (Controller Manager): controls the [controllers](https://kubernetes.io/docs/concepts/architecture/controller/){target=_blank} processes. Accessible through [`kube-controller-manager`](https://kubernetes.io/docs/reference/command-line-tools-reference/kube-controller-manager/) - **c-c-m** (Cloud Controller Manager): lets you to link your cluster to your provider's API, allowing you to choose what components interact with the external platform and which components interact with the internal cluster. Read more on the cloud controller manager [here](https://kubernetes.io/docs/concepts/architecture/cloud-controller/){target=_blank}. - The **Node** components: - **kubelet**: An agent that runs on each node in the cluster. It makes sure that containers are running in a Pod. [`kubelet`](https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/) - **k-proxy** (kube-proxy): maintains network rules on nodes. These network rules allow network communication to your Pods from network sessions inside or outside of your cluster. [`kube-proxy`](https://kubernetes.io/docs/reference/command-line-tools-reference/kube-proxy/) !!! question "We keep talking about Pods and Nodes, *what are Pods and Nodes?*" [**Pods**](https://kubernetes.io/docs/concepts/workloads/pods/){target=_blank} [![pods](https://d33wubrfki0l68.cloudfront.net/fe03f68d8ede9815184852ca2a4fd30325e5d15a/98064/docs/tutorials/kubernetes-basics/public/images/module_03_pods.svg){target=_blank}](https://kubernetes.io/docs/tutorials/kubernetes-basics/explore/explore-intro/) A Pod is the smallest deployable unit in Kubernetes. It represents a single instance of a running process in a cluster and encapsulates one or more closely related containers. Containers within the same Pod share the same network namespace, IP address, and storage volumes, making them suitable for co-located and tightly coupled applications. In the image above, we see 4 different Pods, each having at least one containerized app. Notice how each Pod has its own IP address, and apps within the same Pods share volumes for storage and IP address. [**Nodes**](https://kubernetes.io/docs/concepts/architecture/nodes/){target=_blank} [![nodes](https://d33wubrfki0l68.cloudfront.net/5cb72d407cbe2755e581b6de757e0d81760d5b86/a9df9/docs/tutorials/kubernetes-basics/public/images/module_03_nodes.svg){target=_blank}](https://kubernetes.io/docs/tutorials/kubernetes-basics/explore/explore-intro/) A Node is a physical or virtual machine that runs containers. Nodes are the worker machines in a Kubernetes cluster where Pods are scheduled and executed. Nodes collectively form the computational resources of the cluster, where containers are scheduled and executed. The interaction between Pods and Nodes forms the core of how Kubernetes manages and distributes workloads within a cluster. In the image, the Node contains different Pods. Notice [`kubelet`](https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/) and Docker: - [`kubelet`](https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/) is a component that runs on each Node in a Kubernetes cluster and manages its life cycle, ensuring that the Node is healthy. - Docker provides the runtime environment for containers, whilst K8s manages the orchestration. ## K8s CLI `kubectl` The Kubernetes API uses a command-line tool called `kubectl`. Using K8s does not require you to own or maintain your own cluster. You can use the `kubectl` tool to connect to running clusters and start your containers. ### Install `kubectl` [Official Documentation](https://kubernetes.io/docs/tasks/tools/){target=_blank} `kubectl` is already installed in CodeSpaces. To install on Linux: ``` curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl" chmod +x kubectl sudo mv ./kubectl /usr/local/bin/kubectl ``` ### :material-kubernetes: kubeconfig K8s uses YAML files for configuring a cluster. The `config` file is required to make the handshake between the K8s Cluster and external requests, like the one you're making from your local computer or CodeSpace. #### `config` To connect to a running K8s cluster, you need to create a `config` (.yaml) and place it in the `~/.kube/` folder. In our demo [K3s](https://k3s.io/){target=_blank} cluster, the `config` file is maintained in the `/etc/rancher/k3s/k3s.yaml`. You can copy the file to `~/.kube/config` or create a symlink. ??? info " example `config` file" Example of the `~/.kube/config` file which is used for K3s ```{yaml} apiVersion: v1 clusters: - cluster: certificate-authority-data: server: https://127.0.0.1:6443 name: default contexts: - context: cluster: default user: default name: default current-context: default kind: Config preferences: {} users: - name: default user: client-certificate-data: client-key-data: ``` 1. Copy the `~/.kube/config` file from your cluster or use our example here over to your localhost or VM and put it into a temporary directory. 2. Make a second copy and put it into your own `~/.kube/config` folder By default, the `config` file is in the `~/.kube/` directory, but it can be put anywhere or given any name, use the `--kubeconfig` flag to tell `kubectl` where to get the config: ``` kubectl --kubeconfig /custom/path/kube.config get pods ``` #### Setting up the `namespace` `kubectl` needs to know the `namespace` of the cluster `config` you're working with. In this first example today, we're going to be working locally on a K3s with a new `namespace` for each of the students (so we can differentiate whose pods are whose) To set the context and namespace of the cluster: ``` kubectl config set-context student##-cc23 --namespace=student##-cloudcamp23 ``` where ## is your assigned student ID. If you're doing this lesson on your own time, you can choose your own unique context and namespace. Once the `config` namespace is set, you should be ready to launch "pods" i.e. containers on the cluster ### :material-kubernetes: pods Each pod is managed using its own `.yaml` file. This way K8s / K3s can declaratively set the resources, container, networking, volumes, and permissions for each running pod. Create a new directory called `examples` on the JS2 workshop instance you have connected to. Create a new file called `alpine.yaml` (copies of this and the camp's other pod manifests, `ubuntu.yaml` and `jupyterlab.yaml`, are in [`examples/k3s`](https://github.com/UNM-CARC/container-camp/tree/main/examples/k3s){target=_blank}) ``` apiVersion: v1 kind: Pod metadata: name: alpine- spec: containers: - name: mypod image: alpine:3.14 resources: limits: memory: 100Mi cpu: 100m requests: memory: 100Mi cpu: 100m command: ["sh", "-c", "sleep infinity"] ``` #### Launching a Pod Try launching a pod on the cluster ``` kubectl create -f alpine.yaml ``` Check to see that it is running ``` kubectl get pods ``` Do you see any other pods? If the pod hasn't started yet, you can check the timestamp to see if it was created: ``` kubectl get events --sort-by=.metadata.creationTimestamp ``` Try to connect to your running pod (container) ``` kubectl exec -it alpine- -- /bin/sh ``` #### Pod networking Let's check the networking inside the pod ``` # uncomment if ifconfig is not installed # apk add net-tools ifconfig -a ``` Exit the pod (`ctrl`^`D`) Check the IP with `kubectl` ``` kubectl get pod -o wide alpine- ``` #### Taking down a Pod Once you've exited the pod, delete it ``` kubectl delete -f alpine.yaml ``` double check that its gone ``` kubectl get pods ``` ## Create a Deployment While we can create and delete pods on our own, what we realy want is to make our containers have "high availability". High availiability means that when a node dies or is restarted, the pod will "come back up" on its own. First, we can create a `deployment.yaml` ```{yaml} apiVersion: apps/v1 kind: Deployment metadata: name: alpine-instructor-deployment spec: replicas: 3 # Number of Pod replicas selector: matchLabels: app: alpine-instructor template: metadata: labels: app: alpine-instructor spec: containers: - name: mypod image: alpine:3.14 resources: limits: memory: 100Mi cpu: 100m requests: memory: 100Mi cpu: 100m command: ["sh", "-c", "sleep infinity"] ``` This `deplyoment.yaml` will create three replicas of the given Pod, each with the specified `alpine` image, resource limits, and command. You can adjust the number of replicas or other parameters as needed for your workshop demonstration. To create the deployment using the provided YAML file, you can save the file to your local system (e.g., deployment.yaml) and then run the following kubectl command: ``` kubectl apply -f deployment.yaml ``` ??? question "Why bother with Deployments?" **High Availability**: By running three replicas of the Pod, the application becomes more resilient to failures. If one Pod fails, the other two can continue to handle requests, ensuring that the application remains available. **Load Balancing**: Kubernetes automatically distributes incoming traffic across the replicas. This helps in evenly spreading the load, leading to better utilization of resources and potentially improved response times. **Fault Tolerance**: If a node in the cluster fails, the Pods running on that node will be lost. Having multiple replicas ensures that the application continues to run on the remaining nodes. Kubernetes will also work to reschedule the lost Pods on other available nodes. **Scalability**: Having multiple replicas allows the application to handle more simultaneous requests. If the load increases further, the number of replicas can be easily adjusted, either manually or through autoscaling. **Rolling Updates and Rollbacks**: Deployments in Kubernetes facilitate rolling updates, allowing you to update the application with zero downtime. If something goes wrong, you can also easily roll back to a previous version. The multiple replicas ensure that some instances of the application are always available during this process. **Observability and Monitoring**: With multiple replicas, you can monitor the behavior of the application across different instances and nodes, providing insights into the system's performance and potential issues. **Consistent Environment**: Each replica runs in an identical environment, ensuring consistency in the application's behavior and facilitating testing and debugging. In summary, a deployment with three replicas enhances the availability, reliability, scalability, and manageability of the application within a Kubernetes cluster. It aligns well with best practices for running production-grade, distributed systems. ## Run a Dashboard [UpCloud instructions](https://upcloud.com/community/tutorials/deploy-kubernetes-dashboard/){target=_blank} Once the cluster is up and running, you may choose to create a Dashboard for it The Dashboard can only run from a `localhost` so we have to do an ssh tunnel to connect to it ``` ssh -L localhost:8001:127.0.0.1:8001 @ ``` start the dashboard ``` kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.0/aio/deploy/recommended.yaml ``` check to see the dashboard pods are running ``` kubectl get pods -A ``` #### create an `admin` dashboard user ``` mkdir ~/dashboard && cd ~/dashboard ``` create a `dashboard-admin.yaml` ``` apiVersion: v1 kind: ServiceAccount metadata: name: admin-user namespace: kubernetes-dashboard --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: admin-user roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: cluster-admin subjects: - kind: ServiceAccount name: admin-user namespace: kubernetes-dashboard ``` deploy ``` kubectl apply -f dashboard-admin.yaml ``` print the `admin` token so you can log into the dashboard ``` kubectl get secret -n kubernetes-dashboard $(kubectl get serviceaccount admin-user -n kubernetes-dashboard -o jsonpath="{.secrets[0].name}") -o jsonpath="{.data.token}" | base64 --decode ``` #### create a read-only `dashboard-read-only.yaml` user ``` apiVersion: v1 kind: ServiceAccount metadata: name: read-only-user namespace: kubernetes-dashboard --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: annotations: rbac.authorization.kubernetes.io/autoupdate: "true" labels: name: read-only-clusterrole namespace: default rules: - apiGroups: - "" resources: ["*"] verbs: - get - list - watch - apiGroups: - extensions resources: ["*"] verbs: - get - list - watch - apiGroups: - apps resources: ["*"] verbs: - get - list - watch --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: read-only-binding roleRef: kind: ClusterRole name: read-only-clusterrole apiGroup: rbac.authorization.k8s.io subjects: - kind: ServiceAccount name: read-only-user namespace: kubernetes-dashboard ``` deploy it ``` kubectl apply -f dashboard-read-only.yaml ``` print read-only token ``` kubectl get secret -n kubernetes-dashboard $(kubectl get serviceaccount read-only-user -n kubernetes-dashboard -o jsonpath="{.secrets[0].name}") -o jsonpath="{.data.token}" | base64 --decode ``` #### Start the dashboard ``` kubectl proxy ``` ``` http://localhost:8001/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:/proxy/ ``` Copy/Paste your admin or read-only token ``` kubectl -n kubernetes-dashboard create token admin-user ``` #### delete the dashboard ``` kubectl delete -f dashboard-admin.yaml kubectl delete -f dashboard-read-only.yaml kubectl delete -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.0/aio/deploy/recommended.yaml ```

Adapted from [CyVerse Container Camp](https://github.com/CyVerse-learning-materials/container-camp/blob/1746ee8873bcbed2bd3e62500262a9e9571cff16/docs/orchestration/k8s.md){target=_blank} (last source update 2023-08-15), CC BY 4.0. Spotted a problem? [Open an issue](https://github.com/UNM-CARC/container-camp/issues){target=_blank}.

---8<--- https://unm-carc.github.io/container-camp/orchestration/kubernetes-advanced/ --- title: "Deploying your own Kubernetes clusters" description: "Deploy Zero to JupyterHub with Helm, and compare lightweight Kubernetes distributions such as minikube, MicroK8s, and K3s." type: Lesson tags: - Kubernetes - Helm - JupyterHub generated: by: "claude/opus-5" at: "2026-09-11T00:00:00Z" sources: - id: cyverse-container-camp resource: "https://github.com/CyVerse-learning-materials/container-camp/blob/1746ee8873bcbed2bd3e62500262a9e9571cff16/docs/orchestration/advk8s.md" title: "CyVerse Container Camp: docs/orchestration/advk8s.md" author: "team:cyverse" last_modified: "2023-08-16T03:36:46Z" --- # Deploying your own Kubernetes clusters ## Deploying your own Kubernetes clusters We are going to be using Jetstream-2 for this section and will rely on their [documentation for Kubernetes deployment](https://docs.jetstream-cloud.org/general/kubernetes/){target=_blank} ### :material-ship-wheel: Helm [Helm](https://helm.sh/){target=_blank} is the package manager that allows users to define, install, and manage K8s applications using [Helm Charts](https://helm.sh/docs/topics/charts/){target=_blank}, which are packages of pre-configured Kubernetes resources. Similarly to how Docker users can create their own containers and share them collaborators through DockerHub, Helm Charts are built by users and shared through repositories such as [artifacthub](https://artifacthub.io/){target=_blank}. This allows for streamlining deployment and management for complex K8s orchestrations. #### Helm example: [Zero2JupyterHub](https://z2jh.jupyter.org/en/latest/index.html){target=_blank} 1. Install Helm - Users can install Helm on their own machines by following the [official documentation](https://helm.sh/docs/intro/install/){target=_blank}. For this example, we are going to assume that students are using a Unix OS. In that case, the following commands should work: ``` curl https://baltocdn.com/helm/signing.asc | gpg --dearmor | sudo tee /usr/share/keyrings/helm.gpg > /dev/null sudo apt-get install apt-transport-https --yes echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/helm.gpg] https://baltocdn.com/helm/stable/debian/ all main" | sudo tee /etc/apt/sources.list.d/helm-stable-debian.list sudo apt-get update sudo apt-get install helm ``` !!! info If you are using the Virtual Machines provided by the CyVerse team, Helm is already installed. 2. Add the JupyterHub Helm repository, which contains the charts needed to deploy JupyterLab. ``` helm repo add jupyterhub https://jupyterhub.github.io/helm-chart/ helm repo update ``` You can check the Helm repository list with `helm repo list`. You should see something like ``` NAME URL jupyterhub https://jupyterhub.github.io/helm-chart/ ``` 3. Obtain the `config.yaml` - One can write their own config file, but these take extensive work and deep understanding of charts. What we can do instead, is to *pull* the values that we need from Jupyterhub and save its output to a file. We can do that with the following command: ``` helm show values jupyterhub/jupyterhub > jupyter.yaml ``` - If you were to look inside this yaml file, you will see a long list of configurable lines. These values need not to be changed to execute the jupyterhub, apart from `secretToken`. - Generate the secret token with `openssl rand -hex 32`, copy the hex string and replace the `secretToken` line from the `jupyter.yml` file using a text editor (e.g., nano or vim). 4. From within the folder, deploy the jupyterhub using the following command (installation should only take a minute): ``` helm install jupyterhub jupyterhub/jupyterhub --values jupyter.yaml ``` 5. Access the JupyterHub through your VM IP address! ??? tip "Uninstalling the chart" To uninstall the chart simply do ``` helm uninstall jupyterhub ``` If you also want to remove the repository do ``` helm repo remove jupyterhub ``` The names can get pretty confusing, please understand that the first is uninstalling the chart named `jupyterhub` and the second the repository named `jupyterhub` (and this is why you want to keep names unique :) ). ## Miniaturized versions of Kubernetes ??? question "Why use mini versions of Kubernetes?" There are multiple projects developing "light-weight" Kubernetes. The justification for these projects being that full Kubernetes deployments with all of its functionality increases the cognitive load and the number of configurations and parameters that you must work with when running containers and clusters, particularly at the edge. Projects that are working on miniaturized versions of K8s: | name | functionality | use cases | |------|---------------|-----------| | [minikube](https://minikube.sigs.k8s.io/docs/start/){target=_blank} | | | | [microK8s](https://microk8s.io/){target=_blank} | | runs fast, self-healing, and highly available Kubernetes clusters | | [K3s](https://k3s.io/){target=_blank} | | runs production-level Kubernetes workloads on low resourced and remotely located IoT and Edge devices | | K3d | lightweight wrapper that runs K3s in a docker container | | ### Install `K3s` Follow the [K3s quick-start guide](https://docs.k3s.io/quick-start){target=_blank}. The camp's K3s pod manifests are in [`examples/k3s`](https://github.com/UNM-CARC/container-camp/tree/main/examples/k3s){target=_blank}, and [`examples/terraform-k3s`](https://github.com/UNM-CARC/container-camp/tree/main/examples/terraform-k3s){target=_blank} provisions a K3s cluster on Jetstream2 with Terraform. ### Install `minikube` [Minikube](https://minikube.sigs.k8s.io/docs/start/){target=_blank} is useful for running K8s on a single node or locally -- its primary use is to teach you how to use K8s. ``` curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64 sudo install minikube-linux-amd64 /usr/local/bin/minikube sudo minikube config set vm-driver none ``` The pod manifest from the camp's minikube demo is in [`examples/minikube`](https://github.com/UNM-CARC/container-camp/tree/main/examples/minikube){target=_blank}. ### Installing `microK8s` See the [MicroK8s getting started guide](https://microk8s.io/docs/getting-started){target=_blank}.

Adapted from [CyVerse Container Camp](https://github.com/CyVerse-learning-materials/container-camp/blob/1746ee8873bcbed2bd3e62500262a9e9571cff16/docs/orchestration/advk8s.md){target=_blank} (last source update 2023-08-16), CC BY 4.0. Spotted a problem? [Open an issue](https://github.com/UNM-CARC/container-camp/issues){target=_blank}.

---8<--- https://unm-carc.github.io/container-camp/orchestration/terraform/ --- title: "Introduction to Terraform" description: "Install Terraform, create OpenStack application credentials and an SSH keypair, and init, apply, and destroy VMs on Jetstream2." type: Lesson tags: - Terraform - OpenStack - Jetstream2 - Infrastructure as Code generated: by: "claude/opus-5" at: "2026-09-11T00:00:00Z" sources: - id: cyverse-container-camp resource: "https://github.com/CyVerse-learning-materials/container-camp/blob/1746ee8873bcbed2bd3e62500262a9e9571cff16/docs/orchestration/terra.md" title: "CyVerse Container Camp: docs/orchestration/terra.md" author: "team:cyverse" last_modified: "2023-08-10T14:17:31-07:00" --- # Introduction to Terraform [:simple-terraform: Terraform](https://www.terraform.io/){target=_blank} is an open-source infrastructure-as-code (IaC) software tool created by [HashiCorp](https://www.hashicorp.com/){target=_blank}. ??? info "What is Infrastructure-as-Code (IaC)?" "Infrastructure as code (IaC) is the process of managing and provisioning computer data centers through machine-readable definition files, rather than physical hardware configuration or interactive configuration tools." - [:simple-wikipedia: Wikipedia](https://en.wikipedia.org/wiki/Infrastructure_as_code){target=_blank} IaC tools allow you to manage infrastructure with configuration files rather than through a graphical user interface. IaC allows you to build, change, and manage your infrastructure in a safe, consistent, and repeatable way by defining resource configurations that you can version, reuse, and share. -- [:simple-terraform: Terraform Documentation](https://developer.hashicorp.com/terraform/tutorials/aws-get-started/infrastructure-as-code){target=_blank} ## Overview This basic tutorial will guide you through setting up a Terraform project and deploying virtual machines (VMs) as infrastructure on OpenStack Cloud. !!! success "Goals" :material-play: Understand orchestration for deployment to OpenStack cloud (Jetstream2) :material-play: Understand the benefits of Terraform :material-play: Ability to perform basic deployments on OpenStack using Terraform :material-play: Ability to perform provisioning of deployed OpenStack resources through Terraform ??? failure "Things we won't cover" :material-play: OpenStack API :material-play: All of Terraform's features ## Prerequisites * Basic understanding of [:simple-openstack: OpenStack](https://www.openstack.org/){target=_blank} and VMs * Access to an OpenStack cloud (we will use [Jetstream2](https://docs.jetstream-cloud.org/){target=_blank}) * [:simple-terraform: Terraform](https://developer.hashicorp.com/terraform/downloads){target=_blank} installed on your local machine * Optional: request your own allocation to Jetstream2 on [NSF ACCESS-CI](https://allocations.access-ci.org/){target=_blank} !!! success "Outcomes" By the end of this tutorial, you will :material-play: have created SSH keypair :material-play: generated an `*-openrc.sh` file from OpenStack :material-play: started, stopped, and destroyed a Terraform deployment on an OpenStack Cloud ??? info "Terminology" **:material-play: Ansible** - is a suite of software tools that enables infrastructure as code **:material-play: Deploy** - to create a cloud resource or software **:material-play: Infrastructure** - is the collection of hardware and software elements such as computing power, networking, storage, and virtualization resources needed to enable cloud computing **:material-play: Orchestration** - is the automated configuring, coordinating, and managing of computer systems and software **:material-play: Playbook** - are a list of tasks that automatically execute against a host **:material-play: Provision** - making changes to a VM including updating the operating system, installing software, adding configurations **:material-play: Terraform** - is an infrastructure as code tool that lets you build, change, and version cloud and on-prem resources safely and efficiently ### Getting onto :simple-openstack: OpenStack Cloud ??? info "What is OpenStack?" [:simple-openstack: OpenStack](https://www.openstack.org/){target=_blank} is an open source cloud computing infrastructure software project and is one of the three most active open source projects in the world. OpenStack clouds are managed by individuals and institutions on their own bare-metal hardware. [![ACCESS](https://unm-carc.github.io/container-camp/assets/terraform/access-logo.svg){width=200}](https://allocations.access-ci.org/){target=_blank} If you do not have an account, go to [https://allocations.access-ci.org/](https://allocations.access-ci.org/){target=_blank} and begin the process by requesting an "Explore" start-up allocation. ACCESS is the NSF's management layer for their research computing network (formerly called TerraGrid and XSEDE) which includes high performance computing, high throughput computing, and research clouds like Jetstream2. [![JS2](https://unm-carc.github.io/container-camp/assets/terraform/js2-logo.png){width=200}](https://docs.jetstream-cloud.org/general/access/){target=_blank} Jetstream2 is a public research cloud which uses OpenStack as its management layer. CyVerse is developing a User Interface for Jetstream2 called [CACAO (Cloud Automation & Continuous Analysis Orchestration)](https://cacao.jetstream-cloud.org/help){target=_blank}. Beneath its hood is Terraform. CACAO can also be used from the CLI (which we will [show in a later lesson](https://unm-carc.github.io/container-camp/orchestration/cacao-terraform/)). ## :simple-terraform: Terraform installation [:simple-terraform: Official Documentation](https://developer.hashicorp.com/terraform/tutorials/aws-get-started/install-cli#install-terraform){target=_blank} ??? tip "Windows Installation" Download [Terraform](https://developer.hashicorp.com/terraform/downloads){target=_blank} using the appropriate distribution for your OS ??? tip "Mac OS X Installation" Instructions for Mac OS X installation If you're on OS X, you can use [brew](https://brew.sh/){target=_blank} to install with the following commands: ```bash # install terraform -- taken from https://learn.hashicorp.com/tutorials/terraform/install-cli brew tap hashicorp/tap && brew install hashicorp/tap/terraform # install ansible and jq (for processing terraform's statefile into an ansible inventory) brew install ansible jq ``` ??? tip "Linux Installation" Instructions for Ubuntu 22.04 installation ```bash wget -O- https://apt.releases.hashicorp.com/gpg | gpg --dearmor | sudo tee /usr/share/keyrings/hashicorp-archive-keyring.gpg echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list sudo apt update && sudo apt install terraform ``` Install Ansible & J Query ```bash sudo apt-add-repository ppa:ansible/ansible sudo apt update & sudo apt install -y ansible jq ``` Confirm installation: ```bash terraform ``` Should output: ```bash Usage: terraform [global options] [args] The available commands for execution are listed below. The primary workflow commands are given first, followed by less common or more advanced commands. Main commands: init Prepare your working directory for other commands validate Check whether the configuration is valid plan Show changes required by the current configuration apply Create or update infrastructure destroy Destroy previously-created infrastructure All other commands: console Try Terraform expressions at an interactive command prompt fmt Reformat your configuration in the standard style force-unlock Release a stuck lock on the current workspace get Install or upgrade remote Terraform modules graph Generate a Graphviz graph of the steps in an operation import Associate existing infrastructure with a Terraform resource login Obtain and save credentials for a remote host logout Remove locally-stored credentials for a remote host metadata Metadata related commands output Show output values from your root module providers Show the providers required for this configuration refresh Update the state to match remote systems show Show the current state or a saved plan state Advanced state management taint Mark a resource instance as not fully functional test Experimental support for module integration testing untaint Remove the 'tainted' state from a resource instance version Show the current Terraform version workspace Workspace management Global options (use these before the subcommand, if any): -chdir=DIR Switch to a different working directory before executing the given subcommand. -help Show this help output, or the help for a specified subcommand. -version An alias for the "version" subcommand. ``` ## Generate an OpenStack Credential for Terraform Log into OpenStack's Horizon Interface **Step 1** Log into OpenStack's Horizon Interface and create application credentials Generate an `openrc.sh` file in Jetstream2 Horizon Interface ([https://js2.jetstream-cloud.org](https://js2.jetstream-cloud.org){target=_blank}), Select the "Identity" then "Application Credentials" option in the menu (left side) Select "+ Create Application Credential" button on right ![create application credential](https://unm-carc.github.io/container-camp/assets/terraform/create_app_cred.png) Give your new credentials a name and description, leave most of the fields blank ![create description application credential](https://unm-carc.github.io/container-camp/assets/terraform/description_app_cred.png) Download the new crededential `openrc.sh` file to your local !!! tip "**Important**" Do not close the Application Credentials window without copying the `secret` or downloading the `openrc.sh` file. ![download credential](https://unm-carc.github.io/container-camp/assets/terraform/download_app_cred.png) ### Create an SSH keypair with OpenStack ??? tip "Creating a SSH key" To create an SSH key on an Ubuntu 22.04 terminal, you can follow these steps: **Step 1:** Open your terminal and type the following command to generate a new SSH key pair: ```bash ssh-keygen -t rsa -b 4096 ``` **Step 2:** When prompted, press "Enter" to save the key in the default location, or enter a different filename and location to save the key. Enter a passphrase to secure your key. This passphrase will be required to use the key later. Once the key is generated, you can add it to your SSH agent by running the following command: ```bash eval "$(ssh-agent -s)" ssh-add ~/.ssh/id_rsa ``` **Step 3:** Copy the public key to your remote server by running the following command, replacing "user" and "server" with your username and server address: ```bash ssh-copy-id @ ``` **create_ssh_script.sh:** ```bash #!/bin/bash echo "Generating SSH key pair..." ssh-keygen -t rsa -b 4096 echo "Adding key to SSH agent..." eval "$(ssh-agent -s)" ssh-add ~/.ssh/id_rsa read -p "Enter your remote server username: " username read -p "Enter your remote server IP address: " server echo "Ready to copy your new public key to a remote server:" ssh-copy-id username@server echo "SSH key setup complete!" ``` Save the script to a file, make it executable with the following command: ```bash chmod +x create_ssh_script.sh ``` run it with the following command: ```bash ./create_ssh_script.sh ``` Check to make sure you have a public key in the `~/.ssh/` directory, it should have the extension `.pub` ```bash ls ~/.ssh/ ``` Create the keypair to OpenStack ```bash openstack keypair create --public-key ~/.ssh/id_rsa.pub tswetnam-terraform-key ``` You can now check in OpenStack for the new keypair here: [https://js2.jetstream-cloud.org/project/key_pairs](https://js2.jetstream-cloud.org/project/key_pairs){target=_blank} ## Initialize your Terraform project Create a new project folder for our configuration files. ```bash mkdir ~/terraform ``` Copy the `openrc.sh` file you downloaded from OpenStack into the new folder. ```bash cp *-openrc.sh ~/terraform/ ``` Change Directory into your new `terraform/` folder and `source` the `openrc.sh` file to create its environmental variables locally. By sourcing this file, you avoid placing sensitive information about yourself into your code. ```bash source *-openrc.sh ``` ### Configuration files Terraform code is written in HCL (Hashicorp Configuration Language), and its configuration files typically end in the `.tf` file extension. Configuration `.tf` files can either be split into multiple files or maintained in a single file. When using multiple files, it is up to your discretion what the file names are, or how many you decide to split it into. #### File Organization An example for file organization of a terraform project might involve: ```css terraform-project/ ├── main.tf ├── variables.tf ├── outputs.tf ├── instances.tf ├── security.tf └── modules/ ├── network/ │ ├── main.tf │ ├── variables.tf │ └── outputs.tf └── compute/ ├── main.tf ├── variables.tf └── outputs.tf ``` **:simple-terraform: Main Configuration File (`main.tf`):** - contains the primary infrastructure resources and configurations for virtual machines, networks, and storage. **:simple-terraform: Security File (`security.tf`):** - contains optional security group details for instances. **:simple-terraform: Variables File (`variables.tf`):** - defines all the input variables used in the configuration. Declare variables with default values or leave them empty for required user input. Include descriptions for each variable to provide context. **:simple-terraform: Instances File (`instances.tf`):** - provisions the Instance flavor and IP networking of the VMs **:simple-terraform: Outputs File (`outputs.tf`):** - defines the outputs Terraform displays after applying the Main and Variables configuration. Includes: IP addresses, DNS names, or information for resources. ##### Other optional files **:simple-terraform: Provider Configuration File (`provider.tf`):** - includes the provider(s) used in the configuration, such as OpenStack (on commerical cloud: Amazon Web Services (AWS), Azure, or Google Cloud Platform(GCP)) along with their authentication and regional settings. **:simple-terraform: Modules and Reusable Configurations:** - create separate `.tf` files for reusable modules and configurations. Reuse across multiple projects or within the same project on multiple VMs. ## Terraform configuration files. !!! tip "Complete example files" The finished configuration from this lesson (`main.tf`, `variables.tf`, `security.tf`, `instances.tf`, and `output.tf`) is in [`examples/terraform-openstack`](https://github.com/UNM-CARC/container-camp/tree/main/examples/terraform-openstack){target=_blank}. [`examples/terraform-k3s`](https://github.com/UNM-CARC/container-camp/tree/main/examples/terraform-k3s){target=_blank} extends the same pattern to provision a small K3s Kubernetes cluster on Jetstream2. Create the `main.tf` file in the `~/terraform/` directory #### :simple-terraform: `main.tf` ```bash terraform { required_version = ">= 0.14.0" required_providers { openstack = { source = "terraform-provider-openstack/openstack" version = ">=1.47.0" } } } provider "openstack" { auth_url = "https://js2.jetstream-cloud.org:5000/v3/" region = "IU" } ``` The `main.tf` file has just the basics for calling out to an OpenStack provider - we created the necessary configurations for this in the prior steps by sourcing the `*-openrc.sh` file and running `terraform init`. #### :simple-terraform: `variables.tf` Create `variables.tf`, it can also be called `inputs.tf` Here you need to go back to OpenStack and get a couple of additional variables: `vm_number` - defines the number of VMs you wish to launch `public_key` - you need the name of your paired SSH key that you generated in the [prior step](#create-an-ssh-keypair-with-openstack) `image_name` - select the name of a featured or unique image you wish to launch. ```bash variable "vm_number" { # creates a single VM # replace with a larger number to launch more than one VM default = "1" } variable "public_key" { # replace this with the name of the public ssh key you uploaded to Jetstream 2 # https://docs.jetstream-cloud.org/ui/cli/managing-ssh-keys/ default = "tswetnam-terraform-key" } variable "image_name" { # replace this with the image name of the ubuntu iso you want to use # https://js2.jetstream-cloud.org/project/images default = "Featured-Ubuntu20" } variable "network_name" { # replace this with the id of the public interface on JS2 in Project / Network / Networks / public # https://js2.jetstream-cloud.org/project/networks/ default = "auto_allocated_network" } ``` #### :simple-terraform: `security.tf` This file produces two distinct security groups, `terraform_ssh_ping` creates a group with `ssh` access and `ping`, `terraform_tcp_1` creates a group which opens the HTTP (80, 8080), HTTPS (443) ports for connecting a browser based service. ```bash ################ #Security section ################ # Creating Compute Security group resource "openstack_compute_secgroup_v2" "terraform_ssh_ping" { name = "terraform_ssh_ping" description = "Security group with SSH and PING open to 0.0.0.0/0" #ssh rule rule{ ip_protocol = "tcp" from_port = "22" to_port = "22" cidr = "0.0.0.0/0" } rule { from_port = -1 to_port = -1 ip_protocol = "icmp" cidr = "0.0.0.0/0" } } # Create a Netowrking Security group resource "openstack_networking_secgroup_v2" "terraform_tcp_1" { name = "terraform_tcp_1" description = "Security group with TCP open to 0.0.0.0/0" } # Allow HTTP (port 80) traffic resource "openstack_networking_secgroup_rule_v2" "http_rule" { direction = "ingress" ethertype = "IPv4" protocol = "tcp" port_range_min = 80 port_range_max = 80 remote_ip_prefix = "0.0.0.0/0" security_group_id = "${openstack_networking_secgroup_v2.terraform_tcp_1.id}" } # Allow HTTPS (port 443) traffic resource "openstack_networking_secgroup_rule_v2" "https_rule" { direction = "ingress" ethertype = "IPv4" protocol = "tcp" port_range_min = 443 port_range_max = 443 remote_ip_prefix = "0.0.0.0/0" security_group_id = "${openstack_networking_secgroup_v2.terraform_tcp_1.id}" } # Allow Service (port 8080) traffic resource "openstack_networking_secgroup_rule_v2" "service_rule" { direction = "ingress" ethertype = "IPv4" protocol = "tcp" port_range_min = 8080 port_range_max = 8080 remote_ip_prefix = "0.0.0.0/0" security_group_id = "${openstack_networking_secgroup_v2.terraform_tcp_1.id}" } ``` #### :simple-terraform: `instances.tf` ```bash ################ # Instance OS ################ # create each Ubuntu20 instance resource "openstack_compute_instance_v2" "Ubuntu20" { name = "container_camp_Ubuntu20_${count.index}" # ID of Featured-Ubuntu20 image_name = var.image_name # flavor_id is the size of the VM # https://docs.jetstream-cloud.org/general/vmsizes/ flavor_name = "m3.tiny" # this public key is set above in security section key_pair = var.public_key security_groups = ["terraform_ssh_ping", "default"] count = var.vm_number metadata = { terraform_controlled = "yes" } network { name = var.network_name } #depends_on = [openstack_networking_network_v2.terraform_network] } # creating floating ips from the public ip pool resource "openstack_networking_floatingip_v2" "terraform_floatip_ubuntu20" { pool = "public" count = var.vm_number } # assign floating ip to each Ubuntu20 VM resource "openstack_compute_floatingip_associate_v2" "terraform_floatip_ubuntu20" { floating_ip = "${openstack_networking_floatingip_v2.terraform_floatip_ubuntu20[count.index].address}" instance_id = "${openstack_compute_instance_v2.Ubuntu20[count.index].id}" count = var.vm_number } ``` #### :simple-terraform: `output.tf` ```bash ################ #Output ################ output "floating_ip_ubuntu20" { value = openstack_networking_floatingip_v2.terraform_floatip_ubuntu20.*.address description = "Public IP for Ubuntu 20" } ``` #### :simple-terraform: `terraform.tfvars` A `terraform.tfvars` file is used to define the values of input variables. It can also be renamed `*.auto.tfvars`. It serves as a convenient way to store and manage variable values that you don't want to hardcode in your `.tf` files or provide via command-line arguments. By using a `terraform.tfvars` file, you can easily customize and update the variable values for different environments or scenarios. The file should contain key-value pairs, where the key is the variable name and the value is the corresponding variable value. The syntax for defining variables in the `terraform.tfvars` file can be either HCL or JSON. Add your OpenStack credentials and other required information to `terraform.tfvars`: In HCL: ```bash openstack_user_name = "your-openstack-username" openstack_password = "your-openstack-password" openstack_tenant_name = "your-openstack-tenant-name" openstack_auth_url = "your-openstack-auth-url" ``` In JSON: ```json { "openstack_user_name": "your-openstack-username", "openstack_password": "your-openstack-password", "openstack_tenant_name": "your-openstack-tenant-name", "openstack_auth_url": "your-openstack-auth-url" } ``` When you run `terraform apply`, Terraform will automatically load the values from the `terraform.tfvars` file if it exists in the working directory. You can also create multiple `.tfvars` files and specify which one to use by passing the `-var-file` flag when executing Terraform commands: ```bash terraform apply -var-file="custom.tfvars" ``` Variables can also be passed directly into the terraform command-line using the `-var`. You can also combine the use of terraform.tfvars, *.auto.tfvars, and command line flags to set input variables; however, you should understand the [rules for variable precedence before doing so](https://developer.hashicorp.com/terraform/language/values/variables#variable-definition-precedence){target=_blank}. ### :simple-terraform: Intermediate directories and files When `terraform apply` are executed, Terraform generates some new project files, notably `.terraform/` - this directory will contain the `terraform-provider-openstack_version` and a `README.md`, `LICENCE`, and `CHANGELOG.md` `terraform.lock.hcl` `terraform.tfstate` `terraform.tfstate.backup` ## :simple-terraform: Terraform Commands ### :simple-terraform: `init` Now, you are ready to initialize the Terraform project ```bash cd ~/terraform terraform init ``` ??? success "Expected Response" ```bash Initializing the backend... Initializing provider plugins... - Reusing previous version of terraform-provider-openstack/openstack from the dependency lock file - Using previously-installed terraform-provider-openstack/openstack v1.51.1 Terraform has been successfully initialized! You may now begin working with Terraform. Try running "terraform plan" to see any changes that are required for your infrastructure. All Terraform commands should now work. If you ever set or change modules or backend configuration for Terraform, rerun this command to reinitialize your working directory. If you forget, other commands will detect it and remind you to do so if necessary. ``` ### :simple-terraform: `validate` `validate` - scans your terraform directory and reports any syntax errors in your terraform ```bash terraform validate ``` ??? success "Expected Response" ```bash Success! The configuration is valid. ``` ### :simple-terraform: apply Using `terraform apply` will output the changes that will occur to your cloud. You can review the changes and decide to continue. Using the flag `-auto-approve` will also output the changes that will occur but will continue with the execution of the terraform as though you entered `yes`. ```bash terraform apply ``` ??? success "Expected response" ```bash Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols: + create Terraform will perform the following actions: # openstack_compute_floatingip_associate_v2.terraform_floatip_ubuntu20[0] will be created + resource "openstack_compute_floatingip_associate_v2" "terraform_floatip_ubuntu20" { + floating_ip = (known after apply) + id = (known after apply) + instance_id = (known after apply) + region = (known after apply) } # openstack_compute_instance_v2.Ubuntu20[0] will be created + resource "openstack_compute_instance_v2" "Ubuntu20" { + access_ip_v4 = (known after apply) + access_ip_v6 = (known after apply) + all_metadata = (known after apply) + all_tags = (known after apply) + availability_zone = (known after apply) + created = (known after apply) + flavor_id = (known after apply) + flavor_name = "m3.tiny" + force_delete = false + id = (known after apply) + image_id = (known after apply) + image_name = "Featured-Ubuntu20" + key_pair = "tswetnam-terraform-key" + metadata = { + "terraform_controlled" = "yes" } + name = "container_camp_Ubuntu20_0" + power_state = "active" + region = (known after apply) + security_groups = [ + "default", + "terraform_ssh_ping", ] + stop_before_destroy = false + updated = (known after apply) + network { + access_network = false + fixed_ip_v4 = (known after apply) + fixed_ip_v6 = (known after apply) + floating_ip = (known after apply) + mac = (known after apply) + name = "auto_allocated_network" + port = (known after apply) + uuid = (known after apply) } } # openstack_compute_secgroup_v2.terraform_ssh_ping will be created + resource "openstack_compute_secgroup_v2" "terraform_ssh_ping" { + description = "Security group with SSH and PING open to 0.0.0.0/0" + id = (known after apply) + name = "terraform_ssh_ping" + region = (known after apply) + rule { + cidr = "0.0.0.0/0" + from_port = -1 + id = (known after apply) + ip_protocol = "icmp" + self = false + to_port = -1 } + rule { + cidr = "0.0.0.0/0" + from_port = 22 + id = (known after apply) + ip_protocol = "tcp" + self = false + to_port = 22 } } # openstack_networking_floatingip_v2.terraform_floatip_ubuntu20[0] will be created + resource "openstack_networking_floatingip_v2" "terraform_floatip_ubuntu20" { + address = (known after apply) + all_tags = (known after apply) + dns_domain = (known after apply) + dns_name = (known after apply) + fixed_ip = (known after apply) + id = (known after apply) + pool = "public" + port_id = (known after apply) + region = (known after apply) + subnet_id = (known after apply) + tenant_id = (known after apply) } # openstack_networking_secgroup_rule_v2.http_rule will be created + resource "openstack_networking_secgroup_rule_v2" "http_rule" { + direction = "ingress" + ethertype = "IPv4" + id = (known after apply) + port_range_max = 80 + port_range_min = 80 + protocol = "tcp" + region = (known after apply) + remote_group_id = (known after apply) + remote_ip_prefix = "0.0.0.0/0" + security_group_id = (known after apply) + tenant_id = (known after apply) } # openstack_networking_secgroup_rule_v2.https_rule will be created + resource "openstack_networking_secgroup_rule_v2" "https_rule" { + direction = "ingress" + ethertype = "IPv4" + id = (known after apply) + port_range_max = 443 + port_range_min = 443 + protocol = "tcp" + region = (known after apply) + remote_group_id = (known after apply) + remote_ip_prefix = "0.0.0.0/0" + security_group_id = (known after apply) + tenant_id = (known after apply) } # openstack_networking_secgroup_rule_v2.service_rule will be created + resource "openstack_networking_secgroup_rule_v2" "service_rule" { + direction = "ingress" + ethertype = "IPv4" + id = (known after apply) + port_range_max = 8080 + port_range_min = 8080 + protocol = "tcp" + region = (known after apply) + remote_group_id = (known after apply) + remote_ip_prefix = "0.0.0.0/0" + security_group_id = (known after apply) + tenant_id = (known after apply) } # openstack_networking_secgroup_v2.terraform_tcp_1 will be created + resource "openstack_networking_secgroup_v2" "terraform_tcp_1" { + all_tags = (known after apply) + description = "Security group with TCP open to 0.0.0.0/0" + id = (known after apply) + name = "terraform_tcp_1" + region = (known after apply) + tenant_id = (known after apply) } Plan: 8 to add, 0 to change, 0 to destroy. Changes to Outputs: + floating_ip_ubuntu20 = [ + null, ] Do you want to perform these actions? Terraform will perform the actions described above. Only 'yes' will be accepted to approve. ``` You should be prompted ```bash Do you want to perform these actions? Terraform will perform the actions described above. Only 'yes' will be accepted to approve. ``` ??? success "Expected Response after choosing 'yes' " ```bash openstack_networking_secgroup_v2.terraform_tcp_1: Creating... openstack_networking_floatingip_v2.terraform_floatip_ubuntu20[0]: Creating... openstack_compute_secgroup_v2.terraform_ssh_ping: Creating... openstack_compute_instance_v2.Ubuntu20[0]: Creating... openstack_networking_secgroup_v2.terraform_tcp_1: Creation complete after 1s [id=4f0ab1d5-ca29-4f60-9a65-c38d2380719c] openstack_networking_secgroup_rule_v2.service_rule: Creating... openstack_networking_secgroup_rule_v2.https_rule: Creating... openstack_networking_secgroup_rule_v2.http_rule: Creating... openstack_networking_secgroup_rule_v2.http_rule: Creation complete after 1s [id=2d7d6f4c-a3db-48ca-996a-1fe0b18f4f41] openstack_networking_secgroup_rule_v2.service_rule: Creation complete after 2s [id=ead48696-4aae-436f-a12b-8bf3ceac253a] openstack_networking_secgroup_rule_v2.https_rule: Creation complete after 2s [id=20225b99-fd2a-4a05-912c-61056fa21e3b] openstack_compute_secgroup_v2.terraform_ssh_ping: Creation complete after 4s [id=1af38fb4-1d41-4ac2-9c9f-91fc6bbaa72f] openstack_networking_floatingip_v2.terraform_floatip_ubuntu20[0]: Creation complete after 7s [id=6af630bc-9c16-450c-b16f-0483294d8d75] openstack_compute_instance_v2.Ubuntu20[0]: Still creating... [10s elapsed] openstack_compute_instance_v2.Ubuntu20[0]: Creation complete after 15s [id=afc53214-d2bd-45d5-b8b5-b6886976df8c] openstack_compute_floatingip_associate_v2.terraform_floatip_ubuntu20[0]: Creating... openstack_compute_floatingip_associate_v2.terraform_floatip_ubuntu20[0]: Creation complete after 2s [id=149.165.168.217/afc53214-d2bd-45d5-b8b5-b6886976df8c/] Apply complete! Resources: 8 added, 0 changed, 0 destroyed. Outputs: floating_ip_ubuntu20 = [ "149.165.168.217", ] ``` **Congratulations!** You now have a running VM, you can check its status on Horizon, or try to connect to it over `ssh` The `floating_ip_ubuntu20` should be the new VM's IP address (yours will be different than this example). `ssh ubuntu@` ### :simple-terraform: `refresh` `refresh` - will update (refresh) the current terraform state This command is sometimes needed if haven't touched your terraform code in a while or there's a chance that your or other individuals manage your resources using horizon, exosphere, or the cli. ```bash terraform refresh ``` ??? success "Expected Response" ```bash ``` ### :simple-terraform: `show` `show` - will show the current terraform state, as stored in your state file ```bash terraform show ``` ??? success "Expected Response" ```bash ``` ### :simple-terraform: destroy When you are ready to complete the use of your VMs or your deployment, you can `destroy` the project. ```bash terraform destroy ``` ??? success "Expected Response" ```bash openstack_compute_secgroup_v2.terraform_ssh_ping: Refreshing state... [id=1af38fb4-1d41-4ac2-9c9f-91fc6bbaa72f] openstack_compute_instance_v2.Ubuntu20[0]: Refreshing state... [id=afc53214-d2bd-45d5-b8b5-b6886976df8c] openstack_networking_secgroup_v2.terraform_tcp_1: Refreshing state... [id=4f0ab1d5-ca29-4f60-9a65-c38d2380719c] openstack_networking_floatingip_v2.terraform_floatip_ubuntu20[0]: Refreshing state... [id=6af630bc-9c16-450c-b16f-0483294d8d75] openstack_networking_secgroup_rule_v2.service_rule: Refreshing state... [id=ead48696-4aae-436f-a12b-8bf3ceac253a] openstack_networking_secgroup_rule_v2.https_rule: Refreshing state... [id=20225b99-fd2a-4a05-912c-61056fa21e3b] openstack_networking_secgroup_rule_v2.http_rule: Refreshing state... [id=2d7d6f4c-a3db-48ca-996a-1fe0b18f4f41] openstack_compute_floatingip_associate_v2.terraform_floatip_ubuntu20[0]: Refreshing state... [id=149.165.168.217/afc53214-d2bd-45d5-b8b5-b6886976df8c/] Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols: - destroy Terraform will perform the following actions: # openstack_compute_floatingip_associate_v2.terraform_floatip_ubuntu20[0] will be destroyed - resource "openstack_compute_floatingip_associate_v2" "terraform_floatip_ubuntu20" { - floating_ip = "149.165.168.217" -> null - id = "149.165.168.217/afc53214-d2bd-45d5-b8b5-b6886976df8c/" -> null - instance_id = "afc53214-d2bd-45d5-b8b5-b6886976df8c" -> null - region = "IU" -> null } # openstack_compute_instance_v2.Ubuntu20[0] will be destroyed - resource "openstack_compute_instance_v2" "Ubuntu20" { - access_ip_v4 = "10.0.24.205" -> null - all_metadata = { - "terraform_controlled" = "yes" } -> null - all_tags = [] -> null - availability_zone = "nova" -> null - created = "2023-03-23 23:09:42 +0000 UTC" -> null - flavor_id = "1" -> null - flavor_name = "m3.tiny" -> null - force_delete = false -> null - id = "afc53214-d2bd-45d5-b8b5-b6886976df8c" -> null - image_id = "27424df1-c3ea-4c4a-ad8b-6ea9a476f6f8" -> null - image_name = "Featured-Ubuntu20" -> null - key_pair = "tswetnam-terraform-key" -> null - metadata = { - "terraform_controlled" = "yes" } -> null - name = "container_camp_Ubuntu20_0" -> null - power_state = "active" -> null - region = "IU" -> null - security_groups = [ - "default", - "terraform_ssh_ping", ] -> null - stop_before_destroy = false -> null - tags = [] -> null - updated = "2023-03-23 23:09:52 +0000 UTC" -> null - network { - access_network = false -> null - fixed_ip_v4 = "10.0.24.205" -> null - mac = "fa:16:3e:16:6e:8f" -> null - name = "auto_allocated_network" -> null - uuid = "e4ff98ff-d29a-48f4-a95b-88ebd6b0662f" -> null } } # openstack_compute_secgroup_v2.terraform_ssh_ping will be destroyed - resource "openstack_compute_secgroup_v2" "terraform_ssh_ping" { - description = "Security group with SSH and PING open to 0.0.0.0/0" -> null - id = "1af38fb4-1d41-4ac2-9c9f-91fc6bbaa72f" -> null - name = "terraform_ssh_ping" -> null - region = "IU" -> null - rule { - cidr = "0.0.0.0/0" -> null - from_port = -1 -> null - id = "611e1c0a-edc1-462d-8e1a-5f8a72c4d968" -> null - ip_protocol = "icmp" -> null - self = false -> null - to_port = -1 -> null } - rule { - cidr = "0.0.0.0/0" -> null - from_port = 22 -> null - id = "ce99a035-5df2-4131-ba90-268d045d0ff3" -> null - ip_protocol = "tcp" -> null - self = false -> null - to_port = 22 -> null } } # openstack_networking_floatingip_v2.terraform_floatip_ubuntu20[0] will be destroyed - resource "openstack_networking_floatingip_v2" "terraform_floatip_ubuntu20" { - address = "149.165.168.217" -> null - all_tags = [] -> null - fixed_ip = "10.0.24.205" -> null - id = "6af630bc-9c16-450c-b16f-0483294d8d75" -> null - pool = "public" -> null - port_id = "71e9ae50-e281-4f98-afab-f1b6c1932806" -> null - region = "IU" -> null - tags = [] -> null - tenant_id = "db016a81886b4f918705e5dee2b24298" -> null } # openstack_networking_secgroup_rule_v2.http_rule will be destroyed - resource "openstack_networking_secgroup_rule_v2" "http_rule" { - direction = "ingress" -> null - ethertype = "IPv4" -> null - id = "2d7d6f4c-a3db-48ca-996a-1fe0b18f4f41" -> null - port_range_max = 80 -> null - port_range_min = 80 -> null - protocol = "tcp" -> null - region = "IU" -> null - remote_ip_prefix = "0.0.0.0/0" -> null - security_group_id = "4f0ab1d5-ca29-4f60-9a65-c38d2380719c" -> null - tenant_id = "db016a81886b4f918705e5dee2b24298" -> null } # openstack_networking_secgroup_rule_v2.https_rule will be destroyed - resource "openstack_networking_secgroup_rule_v2" "https_rule" { - direction = "ingress" -> null - ethertype = "IPv4" -> null - id = "20225b99-fd2a-4a05-912c-61056fa21e3b" -> null - port_range_max = 443 -> null - port_range_min = 443 -> null - protocol = "tcp" -> null - region = "IU" -> null - remote_ip_prefix = "0.0.0.0/0" -> null - security_group_id = "4f0ab1d5-ca29-4f60-9a65-c38d2380719c" -> null - tenant_id = "db016a81886b4f918705e5dee2b24298" -> null } # openstack_networking_secgroup_rule_v2.service_rule will be destroyed - resource "openstack_networking_secgroup_rule_v2" "service_rule" { - direction = "ingress" -> null - ethertype = "IPv4" -> null - id = "ead48696-4aae-436f-a12b-8bf3ceac253a" -> null - port_range_max = 8080 -> null - port_range_min = 8080 -> null - protocol = "tcp" -> null - region = "IU" -> null - remote_ip_prefix = "0.0.0.0/0" -> null - security_group_id = "4f0ab1d5-ca29-4f60-9a65-c38d2380719c" -> null - tenant_id = "db016a81886b4f918705e5dee2b24298" -> null } # openstack_networking_secgroup_v2.terraform_tcp_1 will be destroyed - resource "openstack_networking_secgroup_v2" "terraform_tcp_1" { - all_tags = [] -> null - description = "Security group with TCP open to 0.0.0.0/0" -> null - id = "4f0ab1d5-ca29-4f60-9a65-c38d2380719c" -> null - name = "terraform_tcp_1" -> null - region = "IU" -> null - tags = [] -> null - tenant_id = "db016a81886b4f918705e5dee2b24298" -> null } Plan: 0 to add, 0 to change, 8 to destroy. Changes to Outputs: - floating_ip_ubuntu20 = [ - "149.165.168.217", ] -> null Do you really want to destroy all resources? Terraform will destroy all your managed infrastructure, as shown above. There is no undo. Only 'yes' will be accepted to confirm. Enter a value: yes ``` ??? success "Expected Response after choosing 'yes' " ```bash openstack_networking_secgroup_rule_v2.service_rule: Destroying... [id=ead48696-4aae-436f-a12b-8bf3ceac253a] openstack_networking_secgroup_rule_v2.https_rule: Destroying... [id=20225b99-fd2a-4a05-912c-61056fa21e3b] openstack_compute_secgroup_v2.terraform_ssh_ping: Destroying... [id=1af38fb4-1d41-4ac2-9c9f-91fc6bbaa72f] openstack_networking_secgroup_rule_v2.http_rule: Destroying... [id=2d7d6f4c-a3db-48ca-996a-1fe0b18f4f41] openstack_compute_floatingip_associate_v2.terraform_floatip_ubuntu20[0]: Destroying... [id=149.165.168.217/afc53214-d2bd-45d5-b8b5-b6886976df8c/] openstack_compute_floatingip_associate_v2.terraform_floatip_ubuntu20[0]: Destruction complete after 3s openstack_networking_floatingip_v2.terraform_floatip_ubuntu20[0]: Destroying... [id=6af630bc-9c16-450c-b16f-0483294d8d75] openstack_compute_instance_v2.Ubuntu20[0]: Destroying... [id=afc53214-d2bd-45d5-b8b5-b6886976df8c] openstack_networking_secgroup_rule_v2.https_rule: Destruction complete after 6s openstack_networking_floatingip_v2.terraform_floatip_ubuntu20[0]: Destruction complete after 6s openstack_networking_secgroup_rule_v2.service_rule: Still destroying... [id=ead48696-4aae-436f-a12b-8bf3ceac253a, 10s elapsed] openstack_networking_secgroup_rule_v2.http_rule: Still destroying... [id=2d7d6f4c-a3db-48ca-996a-1fe0b18f4f41, 10s elapsed] openstack_networking_secgroup_rule_v2.service_rule: Destruction complete after 11s openstack_compute_instance_v2.Ubuntu20[0]: Still destroying... [id=afc53214-d2bd-45d5-b8b5-b6886976df8c, 10s elapsed] openstack_compute_instance_v2.Ubuntu20[0]: Destruction complete after 10s openstack_networking_secgroup_rule_v2.http_rule: Destruction complete after 16s openstack_networking_secgroup_v2.terraform_tcp_1: Destroying... [id=4f0ab1d5-ca29-4f60-9a65-c38d2380719c] openstack_networking_secgroup_v2.terraform_tcp_1: Destruction complete after 9s openstack_compute_secgroup_v2.terraform_ssh_ping: Destroying... [id=1af38fb4-1d41-4ac2-9c9f-91fc6bbaa72f] openstack_compute_secgroup_v2.terraform_ssh_ping: Destruction complete after 2s Destroy complete! Resources: 8 destroyed. ``` ## Troubleshooting ??? failure "My deployment cannot authenticate to the provider" Make sure that you run `source *-openrc.sh` to provide your OpenStack credentials. Also make sure that you are using the correct `~/.ssh/id_rsa.pub` key and that it has been [injected to OpenStack](#create-an-ssh-keypair-with-openstack) ??? failure "`terraform destroy` did not complete" there was an error like: ```bash Error: Error deleting openstack_compute_secgroup_v2 1af38fb4-1d41-4ac2-9c9f-91fc6bbaa72f: Bad request with: [DELETE https://js2.jetstream-cloud.org:8774/v2.1/os-security-groups/1af38fb4-1d41-4ac2-9c9f-91fc6bbaa72f], error message: {"badRequest": {"code": 400, "message": "Security Group 1af38fb4-1d41-4ac2-9c9f-91fc6bbaa72f in use.\nNeutron server returns request_ids: ['req-e3d1548c-9d4c-4445-9642-952977a44853']"}} ``` Try running `terraform destroy` one more time, occasionally Horizon times out while destroying deployments.

Adapted from [CyVerse Container Camp](https://github.com/CyVerse-learning-materials/container-camp/blob/1746ee8873bcbed2bd3e62500262a9e9571cff16/docs/orchestration/terra.md){target=_blank} (last source update 2023-08-10), CC BY 4.0. Spotted a problem? [Open an issue](https://github.com/UNM-CARC/container-camp/issues){target=_blank}.

---8<--- https://unm-carc.github.io/container-camp/orchestration/terraform-advanced/ --- title: "Advanced Terraform: managing Docker containers" description: "Practice count, for_each, input validation, preconditions, and dynamic blocks by managing Docker containers with the Terraform Docker provider." type: Lesson tags: - Terraform - Docker - Infrastructure as Code generated: by: "claude/opus-5" at: "2026-09-11T00:00:00Z" sources: - id: cyverse-container-camp resource: "https://github.com/CyVerse-learning-materials/container-camp/blob/1746ee8873bcbed2bd3e62500262a9e9571cff16/docs/orchestration/advterra.md" title: "CyVerse Container Camp: docs/orchestration/advterra.md" author: "team:cyverse" last_modified: "2023-08-17T12:07:39-07:00" --- # Advanced Terraform: managing Docker containers ## Overview This advanced tutorial will guide you through setting up a Terraform project with [Docker](https://www.docker.com/){target=_blank}. !!! success "Goals" :material-play: Understand how to use registry.terraform.io :material-play: Understand practical terraform advanced language concepts by orchestrating Docker containers :material-play: Understand how to install software and provision multiple VMs :material-play: Understand how Terraform is used to orchestrate VMs together :material-play: Undertand how Terraform is used to create a multi-node JupyterHub ??? failure "Things we won't cover" :material-play: Basic management of Kubernetes clusters (that is a different lesson) :material-play: All of Terraform's features ## Using registry.terraform.io Terraform maintains an active registry site located at [https://registry.terraform.io](https://registry.terraform.io){target=_blank}. This is the Terraform's defacto catalog for - providers - modules - policies (enforcement rules via Terraform Cloud) - runtasks (integrations with other services via Terraform Cloud) Examples of providers: - https://registry.terraform.io/providers/terraform-provider-openstack/openstack/latest - https://registry.terraform.io/providers/kreuzwerker/docker/latest - https://registry.terraform.io/providers/hashicorp/kubernetes/latest ## Prerequisites in doing Terraform exercises * Basic understanding of [:simple-openstack: OpenStack](https://www.openstack.org/){target=_blank} and [:simple-terraform: Terraform](https://terraform-docs.io/){target=_blank} as was covered in the [prior lesson](https://unm-carc.github.io/container-camp/orchestration/terraform/) * Access to an OpenStack cloud (we will use [Jetstream2](https://docs.jetstream-cloud.org/){target=_blank}) * [:simple-terraform: Terraform](https://developer.hashicorp.com/terraform/downloads){target=_blank} installed on your workstation or local machine !!! success "Outcomes" By the end of this tutorial, you will :material-play: have used Terraform to manage Docker containers while learning advanced Terraform concepts ## Using a Terraform to simply manage Docker We will use VMs with Docker installed and learn how to launch containers in a declarative way, rather than using the `docker` command. Once you gain access to your VM, perform the following steps 1. ssh into your VM (alternatively, you can declare use the Docker Terraform provider using ssh access) 2. `git clone https://gitlab.com/stack0/terraform-docker-helloworld.git` 3. `cd terraform-docker-helloworld` 4. Review `input.tf`, `main.tf`, and `terraform.tfvars.example` to get a sense of how to manage a Docker container using Terraform 1. Review the concept of a Terraform `resource` 2. How many resources are created? 3. What is the relationship between the resources? 4. Is there any question that comes to mind about the port property of `docker_container` resource 5. `cp terraform.tfvars.example terraform.tfvars` 6. edit `terraform.tfvars` 1. feel free to edit the `image` or `container_name` with a container you prefer, but if you do, please select a container with a port that you can access 2. if you are using local docker access, use the default value; if you are accessing the docker host remotely, use the string "ssh://myuser@1.2.3.4", where `myuser` is replaced with your vm username and `1.2.3.4` is replaced with your vm's ip address 7. `terraform apply -auto-approve` ??? success "Expected Response" ```bash Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols: + create Terraform will perform the following actions: # docker_container.mycontainer will be created + resource "docker_container" "mycontainer" { + attach = false + bridge = (known after apply) + command = (known after apply) + container_logs = (known after apply) + container_read_refresh_timeout_milliseconds = 15000 + entrypoint = (known after apply) + env = (known after apply) + exit_code = (known after apply) + hostname = (known after apply) + id = (known after apply) + image = (known after apply) + init = (known after apply) + ipc_mode = (known after apply) + log_driver = (known after apply) + logs = false + must_run = true + name = "edwins-container" + network_data = (known after apply) + read_only = false + remove_volumes = true + restart = "no" + rm = false + runtime = (known after apply) + security_opts = (known after apply) + shm_size = (known after apply) + start = true + stdin_open = false + stop_signal = (known after apply) + stop_timeout = (known after apply) + tty = false + wait = false + wait_timeout = 60 } # docker_image.mydocker will be created + resource "docker_image" "mydocker" { + id = (known after apply) + image_id = (known after apply) + name = "nginx:latest" + repo_digest = (known after apply) } Plan: 2 to add, 0 to change, 0 to destroy. docker_image.mydocker: Creating... docker_image.mydocker: Creation complete after 3s [id=sha256:89da1fb6dcb964dd35c3f41b7b93ffc35eaf20bc61f2e1335fea710a18424287nginx:latest] docker_container.mycontainer: Creating... docker_container.mycontainer: Creation complete after 1s [id=54f9987cde7fe5b474348ca3e89955c24eb076d6d4ea49aee7e44928f3f8e711] Apply complete! Resources: 2 added, 0 changed, 0 destroyed. ``` 8. `terraform destroy -auto-approve` ## Increasing the number of containers using count We will next update Terraform to create multiple containers of the same image. 1. Modify the `input.tf` with a new variable ``` variable "num_containers" { type = number description = "number, number of containers to launch" default = 1 } ``` 2. Modify the `main.tf` with the following changes (highlighted lines): ```hcl hl_lines="2 4 7 10-14" resource "docker_container" "mycontainer" { count = var.num_containers image = docker_image.mydocker.image_id name = "${format("%s%02d", var.container_name, count.index)}" ports { internal = 80 external = "${8080 + count.index}" # illustrates using count.index to indicate which port to use } } output "docker_containers" { value = keys({ for index, d in docker_container.mycontainer.* : "${format("%s,%s,%s", index, d.name,d.id)}" => d }) } ``` 3. Edit your `terraform.tfvars` to include a new `num_containers` input variable with a value `5` 4. `terraform apply -auto-approve` ??? success "Expected Response" ```bash $ terraform apply -auto-approve Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols: + create Terraform will perform the following actions: # docker_container.mycontainer[0] will be created + resource "docker_container" "mycontainer" { + attach = false + bridge = (known after apply) + command = (known after apply) + container_logs = (known after apply) + container_read_refresh_timeout_milliseconds = 15000 + entrypoint = (known after apply) + env = (known after apply) + exit_code = (known after apply) + hostname = (known after apply) + id = (known after apply) + image = (known after apply) + init = (known after apply) + ipc_mode = (known after apply) + log_driver = (known after apply) + logs = false + must_run = true + name = "mycontainer00" + network_data = (known after apply) + read_only = false + remove_volumes = true + restart = "no" + rm = false + runtime = (known after apply) + security_opts = (known after apply) + shm_size = (known after apply) + start = true + stdin_open = false + stop_signal = (known after apply) + stop_timeout = (known after apply) + tty = false + wait = false + wait_timeout = 60 + ports { + external = 8080 + internal = 80 + ip = "0.0.0.0" + protocol = "tcp" } } # docker_container.mycontainer[1] will be created + resource "docker_container" "mycontainer" { + attach = false + bridge = (known after apply) + command = (known after apply) + container_logs = (known after apply) + container_read_refresh_timeout_milliseconds = 15000 + entrypoint = (known after apply) + env = (known after apply) + exit_code = (known after apply) + hostname = (known after apply) + id = (known after apply) + image = (known after apply) + init = (known after apply) + ipc_mode = (known after apply) + log_driver = (known after apply) + logs = false + must_run = true + name = "mycontainer01" + network_data = (known after apply) + read_only = false + remove_volumes = true + restart = "no" + rm = false + runtime = (known after apply) + security_opts = (known after apply) + shm_size = (known after apply) + start = true + stdin_open = false + stop_signal = (known after apply) + stop_timeout = (known after apply) + tty = false + wait = false + wait_timeout = 60 + ports { + external = 8081 + internal = 80 + ip = "0.0.0.0" + protocol = "tcp" } } # docker_container.mycontainer[2] will be created + resource "docker_container" "mycontainer" { + attach = false + bridge = (known after apply) + command = (known after apply) + container_logs = (known after apply) + container_read_refresh_timeout_milliseconds = 15000 + entrypoint = (known after apply) + env = (known after apply) + exit_code = (known after apply) + hostname = (known after apply) + id = (known after apply) + image = (known after apply) + init = (known after apply) + ipc_mode = (known after apply) + log_driver = (known after apply) + logs = false + must_run = true + name = "mycontainer02" + network_data = (known after apply) + read_only = false + remove_volumes = true + restart = "no" + rm = false + runtime = (known after apply) + security_opts = (known after apply) + shm_size = (known after apply) + start = true + stdin_open = false + stop_signal = (known after apply) + stop_timeout = (known after apply) + tty = false + wait = false + wait_timeout = 60 + ports { + external = 8082 + internal = 80 + ip = "0.0.0.0" + protocol = "tcp" } } # docker_container.mycontainer[3] will be created + resource "docker_container" "mycontainer" { + attach = false + bridge = (known after apply) + command = (known after apply) + container_logs = (known after apply) + container_read_refresh_timeout_milliseconds = 15000 + entrypoint = (known after apply) + env = (known after apply) + exit_code = (known after apply) + hostname = (known after apply) + id = (known after apply) + image = (known after apply) + init = (known after apply) + ipc_mode = (known after apply) + log_driver = (known after apply) + logs = false + must_run = true + name = "mycontainer03" + network_data = (known after apply) + read_only = false + remove_volumes = true + restart = "no" + rm = false + runtime = (known after apply) + security_opts = (known after apply) + shm_size = (known after apply) + start = true + stdin_open = false + stop_signal = (known after apply) + stop_timeout = (known after apply) + tty = false + wait = false + wait_timeout = 60 + ports { + external = 8083 + internal = 80 + ip = "0.0.0.0" + protocol = "tcp" } } # docker_container.mycontainer[4] will be created + resource "docker_container" "mycontainer" { + attach = false + bridge = (known after apply) + command = (known after apply) + container_logs = (known after apply) + container_read_refresh_timeout_milliseconds = 15000 + entrypoint = (known after apply) + env = (known after apply) + exit_code = (known after apply) + hostname = (known after apply) + id = (known after apply) + image = (known after apply) + init = (known after apply) + ipc_mode = (known after apply) + log_driver = (known after apply) + logs = false + must_run = true + name = "mycontainer04" + network_data = (known after apply) + read_only = false + remove_volumes = true + restart = "no" + rm = false + runtime = (known after apply) + security_opts = (known after apply) + shm_size = (known after apply) + start = true + stdin_open = false + stop_signal = (known after apply) + stop_timeout = (known after apply) + tty = false + wait = false + wait_timeout = 60 + ports { + external = 8084 + internal = 80 + ip = "0.0.0.0" + protocol = "tcp" } } # docker_image.mydocker will be created + resource "docker_image" "mydocker" { + id = (known after apply) + image_id = (known after apply) + name = "nginx:latest" + repo_digest = (known after apply) } Plan: 6 to add, 0 to change, 0 to destroy. Changes to Outputs: + docker_containers = (known after apply) docker_image.mydocker: Creating... docker_image.mydocker: Creation complete after 3s [id=sha256:89da1fb6dcb964dd35c3f41b7b93ffc35eaf20bc61f2e1335fea710a18424287nginx:latest] docker_container.mycontainer[1]: Creating... docker_container.mycontainer[4]: Creating... docker_container.mycontainer[2]: Creating... docker_container.mycontainer[0]: Creating... docker_container.mycontainer[3]: Creating... docker_container.mycontainer[1]: Creation complete after 1s [id=8c80b558394f62fe004ff156fa1fee06ce723f2b0721cfdfd0f4e64fb6fb28f8] docker_container.mycontainer[3]: Creation complete after 1s [id=c0bc456f847a4e403ee4bcd133f79a21f22b4ec8f94db8793160b28807789e06] docker_container.mycontainer[0]: Creation complete after 1s [id=4ce1d4e0617994ba225701e5e0ef243daa8762968c8d7202be2612ecaeaca064] docker_container.mycontainer[2]: Creation complete after 1s [id=36cf1496a418395408a9d5630c8e1cf8b12fa948ab73fe95a3b1a3d8ce767e99] docker_container.mycontainer[4]: Creation complete after 1s [id=46ab03a8cd2616f20d34058d20b298df8ecc84bf5e8cfaf8013086f21ba1f501] Apply complete! Resources: 6 added, 0 changed, 0 destroyed. Outputs: docker_containers = [ "0,mycontainer00,4ce1d4e0617994ba225701e5e0ef243daa8762968c8d7202be2612ecaeaca064", "1,mycontainer01,8c80b558394f62fe004ff156fa1fee06ce723f2b0721cfdfd0f4e64fb6fb28f8", "2,mycontainer02,36cf1496a418395408a9d5630c8e1cf8b12fa948ab73fe95a3b1a3d8ce767e99", "3,mycontainer03,c0bc456f847a4e403ee4bcd133f79a21f22b4ec8f94db8793160b28807789e06", "4,mycontainer04,46ab03a8cd2616f20d34058d20b298df8ecc84bf5e8cfaf8013086f21ba1f501", ] ``` 5. Notice some pieces of the codes that were introduced 1. What is the `count.index`? 2. What is the format function and why use it? 3. What will index start at? 4. Terraform supports math operations 5. An example of using the keys() method and ad hoc dictionary construction 6. `terraform destroy -auto-approve` ## When resources change outside of Terraform In this exercise we'll discover how to use Terraform to handle change. 1. Use `docker stop` and `docker rm` to stop and delete docker containers 2 and 3. ??? success "Expected Response" ```bash $ docker stop 36cf1496a418395408a9d5630c8e1cf8b12fa948ab73fe95a3b1a3d8ce767e99 c0bc456f847a4e403ee4bcd133f79a21f22b4ec8f94db8793160b28807789e06 36cf1496a418395408a9d5630c8e1cf8b12fa948ab73fe95a3b1a3d8ce767e99 c0bc456f847a4e403ee4bcd133f79a21f22b4ec8f94db8793160b28807789e06 $ docker rm 36cf1496a418395408a9d5630c8e1cf8b12fa948ab73fe95a3b1a3d8ce767e99 c0bc456f847a4e403ee4bcd133f79a21f22b4ec8f94db8793160b28807789e06 36cf1496a418395408a9d5630c8e1cf8b12fa948ab73fe95a3b1a3d8ce767e99 c0bc456f847a4e403ee4bcd133f79a21f22b4ec8f94db8793160b28807789e06 ``` 2. Execute a `terraform show` and count the number of instances in the state 3. Execute a `terraform refresh` ??? success "Expected Response" ```bash $ terraform refresh docker_image.mydocker: Refreshing state... [id=sha256:89da1fb6dcb964dd35c3f41b7b93ffc35eaf20bc61f2e1335fea710a18424287nginx:latest] docker_container.mycontainer[3]: Refreshing state... [id=c0bc456f847a4e403ee4bcd133f79a21f22b4ec8f94db8793160b28807789e06] docker_container.mycontainer[2]: Refreshing state... [id=36cf1496a418395408a9d5630c8e1cf8b12fa948ab73fe95a3b1a3d8ce767e99] docker_container.mycontainer[1]: Refreshing state... [id=8c80b558394f62fe004ff156fa1fee06ce723f2b0721cfdfd0f4e64fb6fb28f8] docker_container.mycontainer[4]: Refreshing state... [id=46ab03a8cd2616f20d34058d20b298df8ecc84bf5e8cfaf8013086f21ba1f501] docker_container.mycontainer[0]: Refreshing state... [id=4ce1d4e0617994ba225701e5e0ef243daa8762968c8d7202be2612ecaeaca064] Outputs: docker_containers = [ "0,mycontainer00,4ce1d4e0617994ba225701e5e0ef243daa8762968c8d7202be2612ecaeaca064", "1,mycontainer01,8c80b558394f62fe004ff156fa1fee06ce723f2b0721cfdfd0f4e64fb6fb28f8", "2,mycontainer02,36cf1496a418395408a9d5630c8e1cf8b12fa948ab73fe95a3b1a3d8ce767e99", "3,mycontainer03,c0bc456f847a4e403ee4bcd133f79a21f22b4ec8f94db8793160b28807789e06", "4,mycontainer04,46ab03a8cd2616f20d34058d20b298df8ecc84bf5e8cfaf8013086f21ba1f501", ] ``` 4. Execute a `terraform show` again and recount the instance in the state. Why doesn't the output variable change? 5. Execute a `terraform apply` (without the `-auto-approve`) and review what will be updated. Once you are satisfied with the changes that will happen, enter `yes` at the prompt. ??? success "Expected Response" ```bash $ terraform apply docker_image.mydocker: Refreshing state... [id=sha256:89da1fb6dcb964dd35c3f41b7b93ffc35eaf20bc61f2e1335fea710a18424287nginx:latest] docker_container.mycontainer[0]: Refreshing state... [id=4ce1d4e0617994ba225701e5e0ef243daa8762968c8d7202be2612ecaeaca064] docker_container.mycontainer[1]: Refreshing state... [id=8c80b558394f62fe004ff156fa1fee06ce723f2b0721cfdfd0f4e64fb6fb28f8] docker_container.mycontainer[4]: Refreshing state... [id=46ab03a8cd2616f20d34058d20b298df8ecc84bf5e8cfaf8013086f21ba1f501] Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols: + create Terraform will perform the following actions: # docker_container.mycontainer[2] will be created + resource "docker_container" "mycontainer" { + attach = false + bridge = (known after apply) + command = (known after apply) + container_logs = (known after apply) + container_read_refresh_timeout_milliseconds = 15000 + entrypoint = (known after apply) + env = (known after apply) + exit_code = (known after apply) + hostname = (known after apply) + id = (known after apply) + image = "sha256:89da1fb6dcb964dd35c3f41b7b93ffc35eaf20bc61f2e1335fea710a18424287" + init = (known after apply) + ipc_mode = (known after apply) + log_driver = (known after apply) + logs = false + must_run = true + name = "mycontainer02" + network_data = (known after apply) + read_only = false + remove_volumes = true + restart = "no" + rm = false + runtime = (known after apply) + security_opts = (known after apply) + shm_size = (known after apply) + start = true + stdin_open = false + stop_signal = (known after apply) + stop_timeout = (known after apply) + tty = false + wait = false + wait_timeout = 60 + ports { + external = 8082 + internal = 80 + ip = "0.0.0.0" + protocol = "tcp" } } # docker_container.mycontainer[3] will be created + resource "docker_container" "mycontainer" { + attach = false + bridge = (known after apply) + command = (known after apply) + container_logs = (known after apply) + container_read_refresh_timeout_milliseconds = 15000 + entrypoint = (known after apply) + env = (known after apply) + exit_code = (known after apply) + hostname = (known after apply) + id = (known after apply) + image = "sha256:89da1fb6dcb964dd35c3f41b7b93ffc35eaf20bc61f2e1335fea710a18424287" + init = (known after apply) + ipc_mode = (known after apply) + log_driver = (known after apply) + logs = false + must_run = true + name = "mycontainer03" + network_data = (known after apply) + read_only = false + remove_volumes = true + restart = "no" + rm = false + runtime = (known after apply) + security_opts = (known after apply) + shm_size = (known after apply) + start = true + stdin_open = false + stop_signal = (known after apply) + stop_timeout = (known after apply) + tty = false + wait = false + wait_timeout = 60 + ports { + external = 8083 + internal = 80 + ip = "0.0.0.0" + protocol = "tcp" } } Plan: 2 to add, 0 to change, 0 to destroy. Changes to Outputs: ~ docker_containers = [ - "0,mycontainer00,4ce1d4e0617994ba225701e5e0ef243daa8762968c8d7202be2612ecaeaca064", - "1,mycontainer01,8c80b558394f62fe004ff156fa1fee06ce723f2b0721cfdfd0f4e64fb6fb28f8", - "2,mycontainer02,36cf1496a418395408a9d5630c8e1cf8b12fa948ab73fe95a3b1a3d8ce767e99", - "3,mycontainer03,c0bc456f847a4e403ee4bcd133f79a21f22b4ec8f94db8793160b28807789e06", - "4,mycontainer04,46ab03a8cd2616f20d34058d20b298df8ecc84bf5e8cfaf8013086f21ba1f501", ] -> (known after apply) Do you want to perform these actions? Terraform will perform the actions described above. Only 'yes' will be accepted to approve. Enter a value: ``` 6. `terraform destroy -auto-approve` ## Validating your input data Next we will show how to add validation to your inputs 1. In your `input.tf`, add the following variable ```bash variable "port_assignment_list" { type = list(number) description = "list of port assignments, size should = num_containers" default = [] validation { condition = length(var.port_assignment_list) > 0 error_message = "Port assignment not > 0" } } ``` 2. In your `main.tf`, replace your docker_container resource to the following: ```bash resource "docker_container" "mycontainer" { count = var.num_containers image = docker_image.mydocker.image_id name = "${format("%s%02d", var.container_name, count.index)}" ports { internal = 80 external = var.port_assignment_list[count.index] # illustrates using count.index to indicate which port to use } } ``` 3. Add/update the following variables in your `terraform.tfvars` ```bash num_containers=5 port_assignment_list=[] ``` 4. `terraform apply -auto-approve` 5. What do you expect? - Try updating the condition such that length must equal `num_containers` Now, let's figure out how to make our validation slightly more useful 1. If necessary, restore your `input.tf` to include the original validation block ```bash variable "port_assignment_list" { type = list(number) description = "list of port assignments, size should = num_containers" default = [] validation { condition = length(var.port_assignment_list) > 0 error_message = "Port assignment not > 0" } } ``` 2. Update your `main.tf` to include a validation block like the following: ```bash resource "docker_container" "mycontainer" { count = var.num_containers image = docker_image.mydocker.image_id name = "${format("%s%02d", var.container_name, count.index)}" ports { internal = 80 external = var.port_assignment_list[count.index] # illustrates using count.index to indicate which port to use } lifecycle { precondition { condition = length(var.port_assignment_list) == num_containers error_message = "length != num_containers" } } } ``` 3. Update your `terraform.tfvars` with the following settings ```bash num_containers=5 port_assignment_list=[4040,5050,6060,7070] ``` 4. `terraform apply -auto-approve` 5. What do you expect? - Try adding another port (or remove a port) so that `num_containers` and the length of the list is equal 6. `terraform destroy -auto-approve` ## Increasing the number of containers using for_each Next we will create multiple docker containers, but using a different method, `for_each`. 1. copy `input.tf`, `main.tf` from `01c-multiple-containers` (overwrite your existing files) 1. review the differences in `input.tf` 2. review the differences in `main.tf` 2. remove the `num_containers` from your `terraform.tfvars` and add a new variable called `containers_map`, something like ``` containers_map={ cat = 8080, dog = 8090, bird = 9090, cow = 9080, horse = 9070 } ``` 3. `terraform apply -auto-approve` 4. Notice some pieces of the codes that were introduced 1. Notice the declaration of a `map`, the default value, and how to set it in `terraform.tfvars` 2. Map keys do not need quoting. What about values? 3. How would you compare the how ports are configured in between the use of `count` and `for_each` 4. Why might you use `count` and `for_each` 5. `terraform destroy -auto-approve` ## Using dynamic blocks create repeatable nestable blocks in resources Next we will see an example of resource properties that can be repeated 1. Copy `input.tf`, `main.tf` from `02a-using-ports` (overwrite your existing files) 2. Review the differences in `main.tf` 2. Edit `input.tf` with a new variable ```bash variable "container_ports" { type = list(object({ internal=string, external=string })) description = "map(object), port object as defined by https://registry.terraform.io/providers/kreuzwerker/docker/latest/docs/resources/container#nestedblock--ports" default = [] } ``` 3. edit the main.tf with the following `docker_container` resource definition ```bash resource "docker_container" "mycontainer" { image = docker_image.mydocker.image_id name = var.container_name dynamic "ports" { for_each = var.container_ports content { internal = ports.value.internal external = ports.value.external } } } ``` 4. Add the following variable in your `terraform.tfvars` ```bash ports = [ {internal="80", external="8080"}, {internal="90", external="9090"} ] ``` 4. `terraform apply -auto-approve` 5. Verify the ports were added when you use `docker ps` 6. `terraform destroy -auto-approve`

Adapted from [CyVerse Container Camp](https://github.com/CyVerse-learning-materials/container-camp/blob/1746ee8873bcbed2bd3e62500262a9e9571cff16/docs/orchestration/advterra.md){target=_blank} (last source update 2023-08-17), CC BY 4.0. Spotted a problem? [Open an issue](https://github.com/UNM-CARC/container-camp/issues){target=_blank}.

---8<--- https://unm-carc.github.io/container-camp/orchestration/cacao/ --- title: "CyVerse CACAO interface" description: "Log in to CACAO on Jetstream2 with ACCESS, add a cloud credential and SSH key, and launch and open a VM deployment." type: Lesson tags: - CACAO - Jetstream2 - Cloud generated: by: "claude/opus-5" at: "2026-09-11T00:00:00Z" sources: - id: cyverse-container-camp resource: "https://github.com/CyVerse-learning-materials/container-camp/blob/1746ee8873bcbed2bd3e62500262a9e9571cff16/docs/orchestration/cacao.md" title: "CyVerse Container Camp: docs/orchestration/cacao.md" author: "team:cyverse" last_modified: "2023-08-18T14:54:06-07:00" --- # CyVerse CACAO interface ## Log into CACAO (JS2) To start, you will need an ACCESS user account and have access to a project. Go to [https://cacao.jetstream-cloud.org](https://cacao.jetstream-cloud.org){target=_blank} to log in to CACAO. You will be prompted to authenticate with ACCESS. !!! info You should select "ACCESS CI (XSEDE)" as your identity provider unless you linked your other identities through CILogon. If in doubt, select "ACCESS CI (XSEDE)". ![](https://unm-carc.github.io/container-camp/assets/cacao/access-ci-select-identity-provider2.png) ### Add your Cloud Credential Next, you will need to add cloud computing credentials. For now, the only option is Jetstream 2, but in the future, other providers like Google and AWS will be supported. Click on Credentials on the lefthand menu bar. From the Credentials page, click the "+ Add Credential" button and select "Cloud Credential". Select Jetstream 2 and the project you would like to add. These correspond to Jetstream 2 projects you have access to. ![](https://unm-carc.github.io/container-camp/assets/cacao/js2_new_credential.png) Once your credential is added, it should show up on the Credentials page. ### Add your SSH Key Next, you will need to add a public SSH key. This will allow you to access your VMs via SSH. If you already have an SSH key, you can use that. Otherwise, you can generate a new one. Again, click on the "+ Add Credential" button and this time select "Public SSH Key". Enter a name and paste in your public key. ![](https://unm-carc.github.io/container-camp/assets/cacao/new_ssh_key.png) ### Start a Deployment Next, we will start a deployment onto Jetstream 2. Click on the [:material-rocket-launch: Deployments](https://cacao.jetstream-cloud.org/deployments){target=_blank} tab on the lefthand menu bar. You will see your cloud providers and projects, and if you have multiple providers or projects, you can select them here. You should have Jetstream 2 and a project selected. Next, click the "+ Add Deployment" button. You will see several options for default templates to launch VMs, containers, or whole clusters. We will use the first template to launch a single VM. ![](https://unm-carc.github.io/container-camp/assets/cacao/cacao_deployments_select_template.png) You will then name the deployment, select the number of instances and size of instances. For now, stick with the default Featured-AlmaLinux8 image and 1 instance of m3.tiny. You can name it whatever you want. ### Access Deployment Once you have submitted your deployment, you can see it on the Deployments page. It will be in the "Starting" status for a few minutes. Once it is running, you can click on it to see details about it and to access it. ![](https://unm-carc.github.io/container-camp/assets/cacao/cacao-deployment-running.png) From here, you can click on the icons on the right to access a Web Shell or a Web Desktop (for certain images). You can also pause, shelve, or delete the deployment from here. Try opening the Web Shell, which will bring you to a command line inside your running VM.

Adapted from [CyVerse Container Camp](https://github.com/CyVerse-learning-materials/container-camp/blob/1746ee8873bcbed2bd3e62500262a9e9571cff16/docs/orchestration/cacao.md){target=_blank} (last source update 2023-08-18), CC BY 4.0. Spotted a problem? [Open an issue](https://github.com/UNM-CARC/container-camp/issues){target=_blank}.

---8<--- https://unm-carc.github.io/container-camp/orchestration/cacao-terraform/ --- title: "CACAO Terraform templates on the CLI" description: "Install the CACAO CLI, log in with an ACCESS token, import a Terraform template from Git, and deploy it through the CACAO UI." type: Lesson tags: - CACAO - Terraform - Command Line generated: by: "claude/opus-5" at: "2026-09-11T00:00:00Z" sources: - id: cyverse-container-camp resource: "https://github.com/CyVerse-learning-materials/container-camp/blob/1746ee8873bcbed2bd3e62500262a9e9571cff16/docs/orchestration/cacao_terra.md" title: "CyVerse Container Camp: docs/orchestration/cacao_terra.md" author: "team:cyverse" last_modified: "2023-08-18T14:54:06-07:00" --- # CACAO Terraform templates on the CLI ## Overview This tutorial will guide you through the process importing a basic terraform template into CACAO. A more detailed version of this tutorial can be found here [https://docs.jetstream-cloud.org/ui/cacao/import_terraform_template/](https://docs.jetstream-cloud.org/ui/cacao/import_terraform_template/){target=_blank} By the end of this tutorial, you will have imported a terraform template into CACAO and deployed it using the UI to confirm that it works. ## Prerequisites * Basic understanding of Terraform, git, and the command line. * An OpenStack environment with access to the API (if you need to validate any terraform changes). * Terraform installed on your local machine or VM server (if you need to validate any terraform changes). * An SSH key pair to access the VM server running Terraform (if you need to validate any terraform changes). ## CyVerse CACAO Browser UI One of CyVerse's test deployments for CACAO will be used for this exercise so that we can import a template using the newer metadata schemas, which will be deployed to production in the very near future. The url for the CACAO site that we will use today is [https://cacao.jetstream-cloud.org](https://cacao.jetstream-cloud.org){target=_blank} Please login to verify that your ACCESS credentials work with the CACAO test site. !!! warning For the workshop, use the "ACCESS CI (XSEDE)" identity provider when you login. ![](https://unm-carc.github.io/container-camp/assets/cacao/access-ci-select-identity-provider2.png) ## Installation of the CyVerse CACAO CLI !!! info If you are using a VM provided by the workshop, the CACAO CLI is already installed. You can skip this installation section. The CyVerse CACAO CLI is a command line tool interact with the CyVerse CACAO API. The first step is the install the cli on your local machine or to a vm. - [Linux download for CACAO CLI](https://gitlab.com/cyverse/cacao/-/package_files/88735155/download){target=_blank} - [Windows download for CACAO CLI](https://gitlab.com/cyverse/cacao/-/package_files/88735261/download){target=_blank} - [MacOS (Intel) download for CACAO CLI](https://gitlab.com/cyverse/cacao/-/package_files/88735323/download){target=_blank} - [MacOS (ARM) download for CACAO CLI](https://gitlab.com/cyverse/cacao/-/package_files/88735401/download){target=_blank} If you're using a VM provided by the workshop or a Linux system, you can use these instructions to install the CACAO CLI. 1. If necessary, obtain a shell or terminal on the Linux system you wish to install the CACAO CLI. 2. Copy link for "Linux download for CACAO CLI" from above. 3. `curl https://gitlab.com/cyverse/cacao/-/package_files/88735155/download --output cacao.zip` 4. `unzip cacao.zip # this will create a file cacao_linux_amd64` 5. `sudo mv cacao_linux_amd64 /usr/local/bin/cacao # optional; otherwise, add it into your path` 6. `chmod +x /usr/local/bin/cacao` ## Reviewing the Terraform template we will use to import We will be using [https://github.com/cyverse/cacao-terraform-hello-world](https://github.com/cyverse/cacao-terraform-hello-world){target=_blank} for this tutorial. You will find a directory called `.cacao` at the root of the template and two files: - `metadata.json`, which contains the CACAO-specific metadata for the template - `ui.json`, which contains the UI-specific hints for the template The `ui.json` file is optional, and if not present, then the CACAO UI will present a default naive rendering of the template. Details about the metadata and ui schemas can be found here: [https://docs.jetstream-cloud.org/ui/cacao/import_terraform_template/](https://docs.jetstream-cloud.org/ui/cacao/import_terraform_template/){target=_blank} !!! warning The UI metadata specification is currently in flux and will likely change in the near future, especially as we get feedback from the community. ### Stop! If you want to fork the template, let's do it now. !!! info This is an optional step. You do not need to fork a template for this part of the tutorial, but you may if you wish. ## Using the CACAO CLI ### Login to CACAO CLI 1. `cacao login --browser` ??? tip "What to do if you encounter a login issue" Sometimes login using the command line will fail -- a typo happens, a copy-n-paste of a token happens, using the wrong api url happens, etc -- and you need to reset your login. To reset your login, you can use the following command: `cacao logout` ???+ success "Expected Response" ```bash linenums="1" hl_lines="4" Please provide address of Cacao API. Format Should be: http://: or : (Developers: this should match the value of API_DOMAIN in install/config.yaml followed by "/api", e.g. ca.cyverse.local/api) Cacao API address (http://ca.cyverse.local/api): https://cacao.jetstream-cloud.org/api ``` 2. Enter the CACAO API url: `https://cacao.jetstream-cloud.org/api` ???+ success "Expected Response" ```bash linenums="4" hl_lines="2" Cacao API address (http://ca.cyverse.local/api): https://cacao.jetstream-cloud.org/api Please go to this URL in the browser: https://cacao.jetstream-cloud.org/api/user/login After login, you should get a JSON response, the auth token could be the value of following properties: - "IDToken" or "id_token" if keycloak - "access_token" if other auth provider Enter the auth token you get from browser: ``` 3. In your browser open the following URL (this is also echoed in the terminal, line #5): https://cacao.jetstream-cloud.org/api/user/login !!! warning For the workshop, use the "ACCESS CI (XSEDE)" identity provider when you login. ![](https://unm-carc.github.io/container-camp/assets/cacao/access-ci-select-identity-provider2.png) 4. You will need to grab the the "access_token" that is returned from the browser and paste the following text into the terminal: "Bearer" followed by a space and then the access token. For example, if the access_token were GOOSE, the the string you enter will be: `Bearer GOOSE` !!! tip - Do not include the quotes around the token. - `Bearer` is case sensitive. - There is one space between Bearer and the token. Below are screenshots of what you might expect to see from Chrome and Firefox. Chrome: ![](https://unm-carc.github.io/container-camp/assets/cacao/access-token-chrome.png) Firefox: ![](https://unm-carc.github.io/container-camp/assets/cacao/access-token-firefox.png) After pasting the "Bearer " + access_token, you should return to the shell prompt. ???+ success "Expected Response" ```bash linenums="11" Enter the auth token you get from browser: Bearer NB2HI4DTHIXASDFUINASDKLFASDHFKASDFJASDFA23FNYTHI4Z5GE3DSMRTGMYDGNJWHE3TIJTWMVZHG2LPNY6XMMROGATGY2LGMV2GS3LFHU4TAMBQGAYA $ ``` 5. To test a successful login, you can execute a cacao command, such as: `cacao provider get` ### Importing a Terraform template into CACAO `cacao template create git