# CARC Documentation — full corpus

Each page below begins with its canonical URL followed by its original Markdown, OKF frontmatter included.

---8<--- https://unm-carc.github.io/docs/getting-started/good-neighbor-policy/

---
title: "Good Neighbor Use Policy"
description: "Acceptable-use rules that all CARC users agree to: account sharing, data restrictions, job monitoring, and security."
type: Policy
tags:
  - Policy
  - New users
generated:
  by: "claude/fable-5"
  at: "2026-08-29T00:00:00Z"
sources:
  - id: carc-web
    resource: "https://carc.unm.edu/new-users/good-neighbor-policy.html"
    title: "Good Neighbor Use Policy (carc.unm.edu)"
    author: "team:unm-carc"
---

# Good Neighbor Use Policy

These guidelines for the appropriate use of CARC computing facilities apply to
all users. They ensure the security of user accounts and equitable access to
computing facilities for everyone.

!!! warning "Compliance is required"

    Failure to comply with these rules will result in removal of your accounts
    and/or projects, and possible disciplinary or legal action.

## Acceptable use

By accepting a CARC account, I agree that:

* My CARC account is for my **sole use** and will not be shared with others,
  including supervising faculty. CARC ensures equitable access using account
  limits; sharing accounts prevents fair allocation, and shared accounts will
  be locked. CARC can help solve the problems that often lead people to want
  to share accounts — just ask!
* I am responsible for all use of this computer account.
* I will use the account for **research computing purposes** and not for
  commercial purposes.
* I will not engage in any form of illegal software copying or other
  copyright infringement.
* I will not attempt to access accounts, files, or information belonging to
  other users without their knowledge and consent.
* I understand that CARC personnel may monitor user activity on CARC machines.
* **HIPAA, PHI, PCI, FERPA, and Controlled Unclassified Information (CUI)
  data may not be stored on or transferred via CARC systems.**
* All requests to install **export-controlled software** on CARC systems
  require prior written approval or a written exception from UNM Export
  Control.
* I will not store individually identifiable health information on CARC
  systems, and will follow de-identification procedures in accordance with
  U.S. Department of Health and Human Services regulations when working with
  health data.

## Job monitoring

* Monitor your compute jobs to ensure they are making progress and are not
  stuck in a static state.
* Request resources suitable for the software you are running — do not leave
  hardware idle.
* Use `seff <jobid>` after your job completes to check whether the resources
  you requested were used effectively. CARC personnel can help via
  [office hours](../support/help.md) or <help@carc.unm.edu>.

## Security

* No passwords — for CARC or any other machines — may be stored in
  unencrypted form on CARC file systems. Violation may lead to immediate and
  permanent expulsion from CARC systems.
* Minimum complexity checks are enforced on passwords.
* Only secure connection protocols (e.g., SSH, SSL) may be used to connect to
  CARC machines.

## Use of servers

CARC provides high-performance computing access. We do not provide enterprise
services such as website or email hosting — contact UNM IT for those services.

## Acknowledging CARC

Publications, dissertations, theses, posters, and talks that used CARC
resources should [acknowledge CARC](../support/acknowledging-carc.md), and
should be added to your project's publication list in
[ColdFront](https://coldfront.alliance.unm.edu){ target=_blank }.

---8<--- https://unm-carc.github.io/docs/getting-started/learning-linux/

---
title: "Learning Linux resources"
description: "Curated external resources for learning the Linux command line."
type: Reference
tags:
  - Linux
  - New users
generated:
  by: "claude/fable-5"
  at: "2026-08-29T00:00:00Z"
sources:
  - id: quickbytes
    resource: "https://github.com/UNM-CARC/QuickBytes/blob/master/learning_linux.md"
    title: "UNM-CARC QuickBytes: learning_linux.md"
    author: "team:unm-carc"
    last_modified: "2020-01-28T10:56:14-07:00"
---

# Learning Linux resources

All CARC systems run Linux as their operating system and are interacted with in a command-line interface (CLI) as opposed to a graphical user interface (GUI) where you can point at and click on things with a mouse.In order to effectively navigate and efficiently utilize CARC systems we recommend you become familiar with basic Linux commands and navigation in a CLI.
Although the command line environment can look intimidating at first, you will soon learn that because Linux/Unix were written by scientists for scientists, these tools are extremely powerful and you can become more productive using them than you thought possible. There are many resources freely available for learning this material, and since they have been written better than CARC could hope to do, we will now simply direct you to some of our favorite learning resources.

### Learning Linux
Below are some great resources for learning the basics of operating in a Linux CLI environment:

1. [Software Carpentry](https://software-carpentry.org/lessons/){target=_blank} \- This website has lessons that help develop a strong foundation in Linux, especially lesson 1, but the other lessons are helpful in learning some very powerful tools found in Linux.
2. [Linux Journey](https://linuxjourney.com/){target=_blank} \- There is a lot of overlap between Linux Journey and Software Carpentry, but some may find the approach of Linux Journey more accessible.
3. [Greg's Bash Wiki](https://mywiki.wooledge.org/){target=_blank} \- One of the classic resources on utilizing Bash, the default shell for interacting with a Linux system.

### Learning programming languages
Once you are confident operating within Linux you may want to dabble in writing some of your own programs. One of the great things about computers is that we can automate them to perform tedious and menial tasks that would take us hours or days to complete, and a computer can do it almost instantaneously and without error. Below are resources for learning some of the more common programming languages.

1. [Python](https://www.python.org/){target=_blank} \- Python is one of the most popular languages for scientific computing and is a great skill to learn, even if you are just trying to debug someone else's code. [Rosalind](http://rosalind.info/problems/locations/){target=_blank} has some really great lessons focused on solving actual problems in order to learn to program in python. Lessons range in difficulty from very simple to rather advanced.
2. [Julia](https://julialang.org/){target=_blank} \- Julie is another powerful programing language popular for scientific computing. 
	* [Learn X in Y Minutes](https://learnxinyminutes.com/docs/julia/){target=_blank}
	* [Julia by Example](http://samuelcolvin.github.io/JuliaByExample/){target=_blank}
3. [Go](https://golang.org/){target=_blank} \- Go is a programming language developed by Google. 
	* [Learn X in Y minutes](https://learnxinyminutes.com/docs/go/){target=_blank}
	* [Go by example](https://gobyexample.com/){target=_blank}

In addition to these more modern languages, a decent foundation in the classics like Perl, C, C++, and Fortran can be beneficial to learn as a lot of software, both old and current, are written in these languages, and a basic understanding of how to read the code can help when troubleshooting issues.

<p class="carc-provenance" markdown>Migrated from [UNM-CARC QuickBytes](https://github.com/UNM-CARC/QuickBytes/blob/master/learning_linux.md){target=_blank} (last source update 2020-01-28). Spotted a problem? [Open an issue or pull request](https://github.com/UNM-CARC/QuickBytes){target=_blank}.</p>

---8<--- https://unm-carc.github.io/docs/getting-started/linux-intro/

---
title: "Introduction to Linux"
description: "A first tour of the Linux command line for new HPC users."
type: Tutorial
tags:
  - Linux
  - New users
generated:
  by: "claude/fable-5"
  at: "2026-08-29T00:00:00Z"
sources:
  - id: quickbytes
    resource: "https://github.com/UNM-CARC/QuickBytes/blob/master/linux_intro.md"
    title: "UNM-CARC QuickBytes: linux_intro.md"
    author: "team:unm-carc"
    last_modified: "2026-06-17T13:42:20-06:00"
---

# Introduction to Linux

If you are new to the Unix/Linux command line, you are in the right place. Rather than duplicating the excellent training materials that already exist, this page points you to the best resources for getting started, then covers the CARC-specific context you need to use any of our clusters effectively.

## Start Here: Software Carpentry

[Software Carpentry](https://software-carpentry.org/lessons/){target=_blank} is a volunteer organization that develops and maintains free, peer-reviewed tutorials for computing skills used in research. Their materials are used by universities and research institutions worldwide and are specifically designed for researchers who are new to programming and the command line.

We strongly recommend working through their **Unix Shell** lesson before using Easley:

**[The Unix Shell — Software Carpentry](https://swcarpentry.github.io/shell-novice/){target=_blank}**

This lesson covers everything you need to get started, including navigating the filesystem, creating and editing files, working with directories, redirecting output, and writing shell scripts. It takes approximately 3–4 hours to complete and requires no prior experience.

Their full lesson catalog is available at [software-carpentry.org/lessons](https://software-carpentry.org/lessons/){target=_blank} and includes tutorials on Python, R, Git, and more — all highly relevant to HPC research workflows.

## Other Useful Resources for New Users

- **[explainshell.com](https://explainshell.com/){target=_blank}** — paste any shell command and get a plain-English explanation of each part. Extremely useful when you encounter an unfamiliar command.
- **[The Linux Command Line (free book)](https://linuxcommand.org/tlcl.php){target=_blank}** — a comprehensive introduction to the Linux shell, freely available online.
- **[Git and Version Control — Software Carpentry](https://swcarpentry.github.io/git-novice/){target=_blank}** — managing your code and scripts with Git is strongly recommended for any research computing work.
- **[Programming with Python — Software Carpentry](https://swcarpentry.github.io/python-novice-inflammation/){target=_blank}** — if you plan to use Python on Easley, this is a good starting point.
- **`man <command>`** — every CARC system has built-in manual pages for every command. For example, `man ls` explains every option available for the `ls` command.

## Logging In to Easley

Once you are comfortable with the basics, log in to our main computing cluster, Easley, via SSH from your terminal:

```bash
ssh username@easley.alliance.unm.edu
```

On Mac, use the built-in Terminal app (found in Applications → Utilities). On Windows, use [MobaXterm](https://mobaxterm.mobatek.net){target=_blank}, which provides an SSH client and terminal in one.

## CARC-Specific Notes

A few things about the Easley environment that differ from a typical desktop Linux system:

**Your home directory** is at `/users/yourusername`. This is where you land when you log in. Storage here is backed up but has a quota — use it for scripts and important files, not large datasets.

**Scratch storage** for large working data is at `/easley/scratch/users/yourusername`. This is not backed up, so copy important results to your home or project directory when your job is done.

**Software is managed with modules**, not installed globally. Before using most software, you need to load it with `module load`. See the [Managing Software Modules](../running-jobs/modules.md) QuickByte for details.

**Jobs run through SLURM**, not directly on the login node. Do not run computationally intensive work on the login node — submit it as a job. See the [Intro to SLURM](../running-jobs/slurm-intro.md) QuickByte to get started.

If you have any trouble please reach out to us at help@carc.unm.edu.

*This quickbyte was validated on 6/17/2026*

<p class="carc-provenance" markdown>Migrated from [UNM-CARC QuickBytes](https://github.com/UNM-CARC/QuickBytes/blob/master/linux_intro.md){target=_blank} (last source update 2026-06-17). Spotted a problem? [Open an issue or pull request](https://github.com/UNM-CARC/QuickBytes){target=_blank}.</p>

---8<--- https://unm-carc.github.io/docs/getting-started/logging-in/

---
title: "Logging in to CARC systems"
description: "Connect to CARC clusters with SSH from Linux, macOS, or Windows."
type: Guide
tags:
  - SSH
  - New users
generated:
  by: "claude/fable-5"
  at: "2026-08-29T00:00:00Z"
sources:
  - id: quickbytes
    resource: "https://github.com/UNM-CARC/QuickBytes/blob/master/logging_in.md"
    title: "UNM-CARC QuickBytes: logging_in.md"
    author: "team:unm-carc"
    last_modified: "2026-06-25T13:09:57-06:00"
---

# Logging in to CARC systems

To log in to the CARC systems and start computing you will need a terminal that can log in to remote systems using Secure Shell (SSH). If you are using a Linux or Mac machine you are in luck as they come bundled with a terminal that is ready to use. The terminal application packaged with your OS is great, but if you would like something with a bit more customization, utility, and flexibility there are other free options available. Below are a couple of options that CARC recommends. Note that iterm2 is only available on Mac, and MobaXterm is only available on Windows.


* iTerm2
* MobaXterm


If you are on windows, you can start with using Powershell, however it is recommended to install MobaXterm for a better overall experience.


Now that you have your terminal open you can log in. To do this type the following from the terminal prompt:

```bash
ssh <CARC-USERNAME>@<MACHINE-NAME>.alliance.unm.edu
```

Where `CARC-USERNAME` is the username you were assigned once your account was approved. `MACHINE-NAME` will be one of our carc systems; `hopper` or `easley`, for example. If this is your first time logging in, you will get a prompt asking you to accept your computer as a new authorized host. You can accept this first prompt, and you will then be prompted for your password. 

If you are unsure of your current password, please reference [the password reset quickbyte.](password-reset.md)


*This quickbyte was validated on 5/21/2024*

## Video walkthrough

**Logging into CARC Systems** — from the [CARC video tutorials](../training/videos.md):

<iframe class="carc-video" src="https://www.youtube-nocookie.com/embed/Puhaf6tCNO0" title="Logging into CARC Systems" loading="lazy" allow="accelerometer; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen></iframe>

<p class="carc-provenance" markdown>Migrated from [UNM-CARC QuickBytes](https://github.com/UNM-CARC/QuickBytes/blob/master/logging_in.md){target=_blank} (last source update 2026-06-25). Spotted a problem? [Open an issue or pull request](https://github.com/UNM-CARC/QuickBytes){target=_blank}.</p>

---8<--- https://unm-carc.github.io/docs/getting-started/overview/

---
title: "Getting started at CARC"
description: "Create a CARC account, join a project in ColdFront, and find support."
type: Guide
tags:
  - Accounts
  - New users
generated:
  by: "claude/fable-5"
  at: "2026-08-29T00:00:00Z"
sources:
  - id: carc-web
    resource: "https://carc.unm.edu/new-users/getting-started-at-carc1.html"
    title: "Getting started at CARC (carc.unm.edu)"
    author: "team:unm-carc"
---

# Getting started at CARC

The UNM Center for Advanced Research Computing provides high-performance
computing resources to facilitate UNM research — free of charge to faculty,
staff, and student researchers, with support from the UNM Office of the Vice
President for Research.

Access is organized around **CARC accounts** (people) and **CARC projects**
(resources). Every user needs an active account, and that account must belong
to a project that holds the compute and storage resources you want to use.

!!! note "Good Neighbor Use Policy"

    By creating a CARC account you agree to abide by the
    [Good Neighbor Use Policy](good-neighbor-policy.md). Please read it first.

## 1. Create your account

=== "UNM affiliates"

    If you have a UNM email address, use the self-service account portal:

    [:material-account-plus: Request an account](https://mokey.alliance.unm.edu){ .md-button .md-button--primary target=_blank }

=== "External collaborators"

    If you do not have a UNM email address, ask your UNM project PI to request
    an account for you:

    [:material-account-arrow-right: Collaborator account request form](https://forms.office.com/r/FJ46nKbPvV){ .md-button target=_blank }

## 2. Join a CARC project

Resources (compute time, storage) are attached to projects, managed through
**ColdFront** at [coldfront.alliance.unm.edu](https://coldfront.alliance.unm.edu/){ target=_blank }.

Ask the project's Principal Investigator — typically your research advisor or
UNM collaborator — to add your account to their project. Once you are on a
project, you can use every resource allocated to it.

## For Principal Investigators

1. **Create a project** in [ColdFront](https://coldfront.alliance.unm.edu){ target=_blank }.
   PI eligibility follows the
   [UNM criteria for Principal Investigator status](https://osp.unm.edu/pi-resources/pi-eligibility.html){ target=_blank }.
2. **Request resources** (compute and storage allocations) for the project
   through ColdFront.
3. **Add accounts** — the PI is responsible for adding collaborators' and
   students' accounts to the project.

For walkthroughs of project management and the annual review process, see the
[video tutorials](../training/videos.md).

## Data access and stewardship

CARC follows the data stewardship policies defined by the University for data
ownership and succession for all data stored on CARC systems. Review the
restrictions on regulated data (HIPAA, PHI, PCI, FERPA, CUI) in the
[Good Neighbor Use Policy](good-neighbor-policy.md).

## Next steps

* [Log in to a cluster](logging-in.md) with SSH.
* [Transfer your data](transferring-data.md) onto CARC storage.
* [Run your first Slurm job](../running-jobs/slurm-intro.md).
* New to the command line? Start with the [introduction to Linux](linux-intro.md).

## Video walkthroughs

**Creating an Account** — from the [CARC video tutorials](../training/videos.md):

<iframe class="carc-video" src="https://www.youtube-nocookie.com/embed/ewyqLP-Eo_4" title="Creating an Account" loading="lazy" allow="accelerometer; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen></iframe>

**CARC Project Management with ColdFront** — for PIs and project managers:

<iframe class="carc-video" src="https://www.youtube-nocookie.com/embed/HJcjOSNWp1U" title="CARC Project Management with ColdFront" loading="lazy" allow="accelerometer; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen></iframe>

More episodes — including the [Annual Project Review](https://www.youtube.com/watch?v=MGxMfmShVWk){target=_blank}
walkthrough — are on the [video tutorials page](../training/videos.md).

## Questions?

Submit a help request to <help@carc.unm.edu>, open a ticket at
[support.alliance.unm.edu](https://support.alliance.unm.edu/){ target=_blank },
or visit [office and consultation hours](../support/help.md).

---8<--- https://unm-carc.github.io/docs/getting-started/password-reset/

---
title: "Password reset and one-time passwords"
description: "Reset your CARC password and manage one-time-password (OTP) settings."
type: Guide
tags:
  - Accounts
  - Security
generated:
  by: "claude/fable-5"
  at: "2026-08-29T00:00:00Z"
sources:
  - id: quickbytes
    resource: "https://github.com/UNM-CARC/QuickBytes/blob/master/password_reset.md"
    title: "UNM-CARC QuickBytes: password_reset.md"
    author: "team:unm-carc"
    last_modified: "2026-06-22T09:56:10-06:00"
---

# Password reset and one-time passwords

To reset your password, you can use the link [here](https://mokey.alliance.unm.edu/auth/login){target=_blank}

After entering your CARC username, you can follow the prompts and reset your password. 

You can also log in with the above link to find other information about your CARC account, see which groups you are a part of, activate two-factor authentication, and add SSH keys to your account.

*This quickbyte was validated on 6/22/2026*

<p class="carc-provenance" markdown>Migrated from [UNM-CARC QuickBytes](https://github.com/UNM-CARC/QuickBytes/blob/master/password_reset.md){target=_blank} (last source update 2026-06-22). Spotted a problem? [Open an issue or pull request](https://github.com/UNM-CARC/QuickBytes){target=_blank}.</p>

---8<--- https://unm-carc.github.io/docs/getting-started/ssh-keys/

---
title: "SSH keys and client configuration"
description: "Generate SSH key pairs and configure your SSH client for convenient, secure logins."
type: Guide
tags:
  - SSH
  - Security
generated:
  by: "claude/fable-5"
  at: "2026-08-29T00:00:00Z"
sources:
  - id: quickbytes
    resource: "https://github.com/UNM-CARC/QuickBytes/blob/master/ssh_keygen_config.md"
    title: "UNM-CARC QuickBytes: ssh_keygen_config.md"
    author: "team:unm-carc"
    last_modified: "2025-03-27T13:45:50-06:00"
---

# SSH keys and client configuration

Once you start computing you will be logging in to the CARC systems fairly often and having to type your username at the machine address will become tedious. In order to alleviate this tedium it is beneficial to generate ssh keys and a ssh config file. The ssh keys bypass the need to enter your password each time you log in, and the config file stores the addresses of all the machines you are logging in to. 

### SSH key generation
First, set up your ssh key. To do this type in the terminal prompt:


    ssh-keygen

You will then be asked which file you would like to save your new key under. Press enter here so it will be saved at the default location.

Next, it will ask you to enter a passphrase. We rcommend you add a passphrase. You'll enter your password here, and then confirm it by entering that same password a second time. 

After this point, your ssh key has been created. You should see the randomart image for your key:

    The key's randomart image is:
    +---[RSA 3072]----+
    |            E.o+=|
    |         o . = .*|
    |        . = oo ==|
    |         + oo @.=|
    |        S +  B.=o|
    |       o   o. .=.|
    |      + o + . o+o|
    |     o o o o .+ o|
    |   .o       .  o |
    +----[SHA256]-----+

This is a mechanism built into ssh keys which take each character from the key itself and converts it into a unique image based off of the key you just created. It is meant to be a quick and easy way to compare ssh keys to make sure nothing has changed. 

After this point, your ssh key will have been created in your `~/.ssh` directory. This directory should now have the following structure:

    $ ls ~/.ssh
    id_rsa  id_rsa.pub

Now that your ssh key is created on your local machine, we need to copy over the public key to the server you want to connect to. 

    ssh-copy-id <CARC-USERNAME>@hopper.alliance.unm.edu

After you've entered your password, your public ssh key will be copied into your home directory on hopper. After this point, whenever you ssh from this machine it will check this key against the private key that was just generated in your .ssh directory, and let you in without prompting you for a password.

Since your home directory is shared across all machines at CARC you only need to do this step once to enable ssh key access across all CARC machines. 

### SSH config file
To make logging in to CARC even easier we also recommend setting up a ssh config file which allows you to simply type `ssh machinename` instead of your username at the machine address. To set up this file simply copy the example below and save it to a text document in your `ssh` folder, which is found at `~/.ssh/`. Change the user to your CARC username and you are set to log in quickly and efficiently. You can add machines based on which ones you have access to. 


```
Host hopper
    hostname hopper.alliance.unm.edu
    user CHANGEME
    port 22

Host easley
    hostname easley.alliance.unm.edu
    user CHANGEME
    ForwardX11 yes
    port 22
```


# Troubleshooting & Git

Note that on the CARC clusters by default, your ssh configuration file will contain the following:

    # Added by Warewulf  xxxx-xx-xx
    Host *
    IdentityFile ~/.ssh/cluster
    StrictHostKeyChecking=no

This helps ensure you're able to connect freely across all of the CARC clusters, for example while logged in to hopper you can just type `ssh easley`.

The issue here will arise if you need to add a new ssh key for some reason, say, you need to add an ssh key so you're able to make edits to a git repository from the CARC clusters. If this is the case, you can start by creating a new ssh key as explained in the previous steps of this tutorial. 

You should then edit your `~/.ssh/config` file as mentioned above, and change the Host line under the warewulf defined section to the following:

    # Added by Warewulf  xxxx-xx-xx
    Host hopper*
    IdentityFile ~/.ssh/cluster
    StrictHostKeyChecking=no

This will ensure git will use the default key on the system when cloning with ssh (which will be the new one you just created), and will properly verify your credentials after adding the new public key to your github account. If you do not do this step, you will recevie a permision error when trying to clone or push to a git repo using ssh.

*This quickbyte was validated on 3/27/2025*

<p class="carc-provenance" markdown>Migrated from [UNM-CARC QuickBytes](https://github.com/UNM-CARC/QuickBytes/blob/master/ssh_keygen_config.md){target=_blank} (last source update 2025-03-27). Spotted a problem? [Open an issue or pull request](https://github.com/UNM-CARC/QuickBytes){target=_blank}.</p>

---8<--- https://unm-carc.github.io/docs/getting-started/transferring-data/

---
title: "Transferring data"
description: "Move data to and from CARC systems with scp, rsync, sftp, and Globus."
type: Guide
tags:
  - Data
  - Storage
generated:
  by: "claude/fable-5"
  at: "2026-08-29T00:00:00Z"
sources:
  - id: quickbytes
    resource: "https://github.com/UNM-CARC/QuickBytes/blob/master/transfer_data.md"
    title: "UNM-CARC QuickBytes: transfer_data.md"
    author: "team:unm-carc"
    last_modified: "2026-06-22T11:18:55-06:00"
---

# Transferring data

### Where is your data?

Your home directory, `/users/your-user-name/`, is shared across all CARC machines, meaning that once your data has been uploaded to your home directory, it is accessible regardless of which machine you are logged in to. Refer to our [CARC Systems documentation page](#) for details on CARC systems.

### Graphical User Interface (GUI) options

There are several options available for data transfer that employ a GUI for ease of use. Several options are listed below, linked to the homepage for each piece of software, with documentation on how to use it.

* [FileZilla](https://filezilla-project.org/){target=_blank}
* [WinSCP](https://winscp.net/eng/index.php){target=_blank}
* [Fetch](https://fetchsoftworks.com/){target=_blank}
* [CyberDuck](https://cyberduck.io/){target=_blank}

FileZilla is available for both Windows and Unix systems, whereas WinSCP is Windows-only and Fetch is macOS-only. GUI-based programs are very user-friendly and well-suited to those who are less comfortable with the Linux command-line interface. Unfortunately, the programs listed above, and other GUI-based programs, use File Transfer Protocol (FTP), which has a relatively low transfer speed and is best suited to smaller file sizes.

### Command-line interface (CLI) options

For larger files, it is recommended that you use one of several programs implemented as a command-line interface. These programs have several benefits over their GUI-based counterparts, including higher transfer speeds and the ability to resume a transfer if it is interrupted, without having to restart from the beginning. Below are two popular options with example commands and links for more advanced usage.

#### Secure Copy (SCP)

Transfer from local machine to CARC:
```bash
scp /your-file your-username@easley.alliance.unm.edu:target-directory/
```

Transfer from CARC to local machine:
```bash
scp your-username@easley.alliance.unm.edu:your-file /target-directory/
```

#### Remote Sync (RSYNC)

Transfer from local machine to CARC:
```bash
rsync -vhatP /your-file your-username@easley.alliance.unm.edu:target-directory
```

Transfer from CARC to local machine:
```bash
rsync -vhatP your-username@easley.alliance.unm.edu:your-file /target-directory/
```

The `-vhatP` flags instruct rsync to print the progress of the transfer verbosely and in a human-readable format.

As you can see, the syntax for these two programs is very similar; however, the options for advanced usage are unique to each one. The examples above cover only basic data transfers — refer to the links provided, or use `man programname` for the CLI options, to optimize each tool for maximum data transfer efficiency and speed.

*This quickbyte was validated on 6/22/2026*

## Video walkthrough

**Transferring data** — from the [CARC video tutorials](../training/videos.md):

<iframe class="carc-video" src="https://www.youtube-nocookie.com/embed/2UphEzHOHGM" title="Transferring data" loading="lazy" allow="accelerometer; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen></iframe>

<p class="carc-provenance" markdown>Migrated from [UNM-CARC QuickBytes](https://github.com/UNM-CARC/QuickBytes/blob/master/transfer_data.md){target=_blank} (last source update 2026-06-22). Spotted a problem? [Open an issue or pull request](https://github.com/UNM-CARC/QuickBytes){target=_blank}.</p>

---8<--- https://unm-carc.github.io/docs/getting-started/x11-forwarding/

---
title: "X11 forwarding"
description: "Display graphical applications from CARC machines on your local screen with X11 forwarding."
type: Guide
tags:
  - SSH
  - Visualization
generated:
  by: "claude/fable-5"
  at: "2026-08-29T00:00:00Z"
sources:
  - id: quickbytes
    resource: "https://github.com/UNM-CARC/QuickBytes/blob/master/X11_forwarding.md"
    title: "UNM-CARC QuickBytes: X11_forwarding.md"
    author: "team:unm-carc"
    last_modified: "2026-06-17T14:15:52-06:00"
---

# X11 forwarding

X11 Forwarding allows any graphical user interface to open on your local machine while the software itself is being run on a CARC cluster compute node. This QuickByte will show you how to set up and use X11 forwarding.

## On a Mac

macOS does not include an X11 server, so you will need to install [XQuartz](https://www.xquartz.org){target=_blank} before using X11 forwarding. You can install it in one of two ways:

**Option 1 — Download the installer** from [xquartz.org](https://www.xquartz.org){target=_blank} and run the `.dmg` package.

**Option 2 — Install via Homebrew** (if you have Homebrew installed):

```bash
brew install --cask xquartz
```

After installing, log out and back in before using X11 forwarding.

> **Note:** If you are on the latest version of macOS, check the [XQuartz releases page](https://github.com/XQuartz/XQuartz/releases){target=_blank} to confirm your macOS version is supported before installing.

## On a PC (Windows)

Download and install [MobaXterm](https://mobaxterm.mobatek.net){target=_blank}. It combines an SSH client, X server, and file transfer client into one application and handles X11 forwarding automatically — no additional configuration required. The free Home Edition is sufficient for CARC use.

## Step by Step Example of X11 Forwarding with MATLAB

### 1. Log in to Easley with X11 forwarding enabled

For Mac, launch XQuartz, then open a terminal from the 'applications' menu. Use the `-Y` flag when connecting via SSH to enable trusted X11 forwarding:

```bash
ssh -Y username@easley.alliance.unm.edu
```

> **Note:** `-Y` (trusted forwarding) is recommended over `-X` (untrusted forwarding) as it works more reliably across different system configurations, but if one doesn't work, try the other. Also note that X11 forwarding requires a native terminal with XQuartz (Mac) or MobaXterm (Windows) — it will not work from the OOD web terminal at ood.alliance.unm.edu.

If you get the error `"/usr/bin/xauth: file /users/user/.Xauthority does not exist"`, run:

```bash
touch ~/.Xauthority
```

Then log out and back in before continuing.

### 2. Start an interactive session with X11 forwarding

```bash
srun --x11 --pty bash
```

### 3. Load and launch MATLAB

Once you have been assigned a node, load the MATLAB module and start MATLAB. The GUI should automatically open on your local machine.

```bash
module load matlab
matlab
```

### 4. Use the MATLAB GUI

Use the MATLAB GUI to load your add-on or run your application as you normally would.

### 5. Exit when finished

When you are finished, close the GUI by clicking the X. Then, exit the interactive session to release the node for other users:

```bash
exit
```

## Video Tutorial

A CARC QuickByte video on X11 forwarding is available here:
https://www.youtube.com/watch?v=-5ic9JWHuqI&list=PLvr5gRBLi7VAzEB_t5aXOLHLfdIu2s1hZ&index=12

If you have any trouble at any point, please reach out to us at help@carc.unm.edu

*This quickbyte was validated on 6/17/2026*

## Video walkthrough

**X11 Forwarding** — from the [CARC video tutorials](../training/videos.md):

<iframe class="carc-video" src="https://www.youtube-nocookie.com/embed/-5ic9JWHuqI" title="X11 Forwarding" loading="lazy" allow="accelerometer; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen></iframe>

<p class="carc-provenance" markdown>Migrated from [UNM-CARC QuickBytes](https://github.com/UNM-CARC/QuickBytes/blob/master/X11_forwarding.md){target=_blank} (last source update 2026-06-17). Spotted a problem? [Open an issue or pull request](https://github.com/UNM-CARC/QuickBytes){target=_blank}.</p>

---8<--- https://unm-carc.github.io/docs/systems/cluster-specifications/

---
title: "Cluster specifications (legacy reference)"
description: "Historical hardware tables for CARC clusters, including retired systems such as Wheeler, Taos, Gibbs, and Xena."
type: Reference
tags:
  - Systems
  - Hardware
  - Legacy
status: deprecated
generated:
  by: "claude/fable-5"
  at: "2026-08-29T00:00:00Z"
sources:
  - id: webinfo
    resource: "https://github.com/UNM-CARC/webinfo/blob/master/systems_information.md"
    title: "UNM-CARC webinfo: systems_information.md"
    author: "team:unm-carc"
    last_modified: "2023-01-23T10:24:36-07:00"
---

# Cluster specifications (legacy reference)

!!! warning "Legacy content"
    This page is kept for history and links. Wheeler, Taos, Gibbs, and Xena have been retired — see the [Systems overview](overview.md) for current clusters.

## CARC Supercomputer and Cluster Resources

| **Machine Name** | **Wheeler** | **Taos** | **Gibbs** | **Xena** | **Hopper** |
| :---: | :---: | :---: | :---: | :---: | :---: |
| **Model/Type** | SGI AltixXE<br>Xeon X5550<br>2.67 GHz | Dell PowerEdge R630<br>Xeon E5-2698 V4<br>2.20 GHz | Dell PowerEdge R620<br>Intel Xeon E5-2670<br>2.6 GHz | Dell PowerEdge R730<br>Intel Xeon E5-2640<br>2.6 GHz<br>and<br>PowerEdge R930<br>Intel Xeon E7-4809<br>2.0 Ghz | Dell PowerEdge R640<br>Intel Xeon Gold 6226R<br>2.9 GHz<br>and<br>Dell PowerEdge R740<br>Intel Xeon Gold 6242<br>2.8 GHz |
| **Linux Operating<br>System** | CentOS 7 | CentOS 7 | Scientific Linux | CentOS 7 | Rocky Linux |
| **Interconnect** | Mellanox IS5600<br>InfiniScale IV<br>ConnectX-2 IB QDR<br>(MT26428) | Mellanox SX6000<br>ConnectX-3 IB FDR<br>(MT4099) | InfiniBand QDR | InfiniBand FDR | InfiniBand HDR |
| **Nodes** | 304 | 9 | 24 | 32 | 61 |
| **Cores/Node** | 8 | variable | 16 | 16, 32 | 32 |
| **Total Cores** | 2432 | 180 | 384 | 576 | 2176 |
| **RAM/Core** | 6GB | variable | 4 GB | 4 GB, 32 GB, 96 GB | variable |
| **Local disk/node** | Diskless | 1 TB | 1 TB | 1 TB | 448 GB |
| **Peak FLOPS<br>(theoretical),<br>in TFLOPS** | 25 | (TBD) | 3.996 | 18 | (TBD) |
| **Processor Architecture** | Intel Xeon<br>Nehalem EP | Intel Xeon<br>Broadwell | Intel SandyBridge | Intel Xeon E7-2640<br>Intel Xeon E7-4809<br>(Haswell) | Intel Xeon Gold 6226R<br>Intel Xeon Gold 6242<br>(Cascade Lake) |
| **Local Scratch Space<br>(TB)** | 40 | 27 | 6.3 | 73 | (TBD) |


### Xena Cluster Specs

| **Queue Name** | bigmem-1TB | bigmem-3TB | dualGPU | singleGPU |
| :---: | :---: | :---: | :---: | :---: |
| **Nodes** | 2 | 2 | 4 | 24 |
|**Cores/Node** | 32 | 32 | 16 | 16 |
|**Memory/Node** | 1 TB | 3 TB| 64 GB | 64 GB |
| **Total Cores** | 64 | 64 | 64 | 384 |
| **Processor<br>Architecture** | Intel Xeon<br>CPU E7-4809 | Intel Xeon<br>CPU E7-4809 | Intel Xeon<br>CPU E5-2640 | Intel Xeon<br>CPU E5-2640 |
| **CPU GHz** | 2.00 | 2.00 | 2.60 | 2.60 |
| **GPU** | N/A | N/A | 2 x Nvidia Tesla<br>K40M per node | 1 x Nvidia Tesla<br>K40M per node |

### Historical queue limits

Queue limits in force when these systems were retired:

#### Xena

| Queue                | GPU                             | Bigmem                                | Debug    |
|---:                  |:---:                            |:---:                                  |:---:     |
| Number of Processors | 192                             | 128                                   | 8        |
| Number of Nodes      | 12 (singleGPU) <br> 4 (dualGPU) | 1                                     | 2        |
| Processors per Node  | 16                              | 32                                    | 4        |
| Walltime(H:M:S)      | 48:00:00                        | 48:00:00                              | 04:00:00 |
| Memory Limit         | 60 Gb (singleGPU and dualGPU)   | 1 Tb (bigmem-1TB)<br>3 Tb (bigmem-3TB)| 60 Gb    |

#### Wheeler

|                Queue: |   Default  |    Debug   |
|----------------------:|:----------:|:----------:|
| Number of Processors  |     400    |     32     |
|      Number of Nodes  |     50     |      4     |
|   Processors per Node |      8     |      8     |
|       Walltime(H:M:S) |  48:00:00  |  04:00:00  |
|         Memory Limit  |    44 Gb   |    44 Gb   |

<p class="carc-provenance" markdown>Migrated from [UNM-CARC QuickBytes](https://github.com/UNM-CARC/webinfo/blob/master/systems_information.md){target=_blank} (last source update 2023-01-23). Spotted a problem? [Open an issue or pull request](https://github.com/UNM-CARC/QuickBytes){target=_blank}.</p>

---8<--- https://unm-carc.github.io/docs/systems/overview/

---
title: "Systems overview"
description: "Current CARC clusters (Easley and Hopper), storage tiers, and web portals such as JupyterHub, Open OnDemand, and XDMoD."
type: Reference
tags:
  - Systems
  - Hardware
stale_after: "2027-08-31T00:00:00Z"
generated:
  by: "claude/fable-5"
  at: "2026-08-29T00:00:00Z"
sources:
  - id: carc-facilities
    resource: "https://carc.unm.edu/about-carc/facilities-description.html"
    title: "CARC facilities description (carc.unm.edu)"
    author: "team:unm-carc"
---

# Systems overview

CARC operates several high-performance computing clusters serving disciplines
from traditional scientific computing to data analytics, artificial
intelligence, and machine learning.

## Compute clusters

| Cluster | Nodes | CPU cores | GPUs | Interconnect | Notes |
| ------- | :---: | :-------: | ---- | ------------ | ----- |
| **Easley** | 65 | 4,160 | 36× NVIDIA L40S (AI/ML) + 8× NVIDIA H100 (double precision) | NVIDIA NDR 800 Gbps InfiniBand | Newest cluster; 23.3 TB total RAM |
| **Hopper** | 61 | 2,176 | 37× NVIDIA A100 | NVIDIA HDR 400 Gbps InfiniBand | General and GPU-accelerated workloads |

For queue limits and walltimes, see [resource limits](resource-limits.md).
Historical specifications for retired systems (Wheeler, Taos, Gibbs, and
Xena) are kept in the [legacy cluster reference](cluster-specifications.md).

## Storage

CARC provides multiple tiers of high-performance storage:

* **720 TB** of all-flash IBM Storage Scale (GPFS) scratch
* **2 PB** of BeeGFS working scratch
* **2.4 PB** of NetApp enterprise storage, with automated hourly, daily,
  weekly, and monthly snapshots retained up to four months for user-directed
  recovery

See [storage and backups](storage.md) for how home, project, and scratch
spaces work, and [resource limits](resource-limits.md) for quotas.

In partnership with UNM Libraries, CARC also supports virtual machine
infrastructure for custom research applications, secure data hosting, and
flexible computing environments.

## Web portals

| Portal | URL | Purpose |
| ------ | --- | ------- |
| JupyterHub (Hopper) | [hopper.alliance.unm.edu](https://hopper.alliance.unm.edu){ target=_blank } | Interactive notebooks on Hopper |
| JupyterHub (Easley) | [easley.alliance.unm.edu/jupyter](https://easley.alliance.unm.edu/jupyter){ target=_blank } | Interactive notebooks on Easley |
| Open OnDemand | [ood.alliance.unm.edu](https://ood.alliance.unm.edu){ target=_blank } | Browser-based files, shells, and interactive apps |
| ColdFront | [coldfront.alliance.unm.edu](https://coldfront.alliance.unm.edu){ target=_blank } | Project, allocation, and publication management |
| XDMoD | [xdmod.alliance.unm.edu](https://xdmod.alliance.unm.edu){ target=_blank } | System usage metrics by PI |
| Help desk | [support.alliance.unm.edu](https://support.alliance.unm.edu){ target=_blank } | Create or manage help tickets |

## Status and downtime

Check the live monitors directly:

* [Cluster login & website status](https://stats.uptimerobot.com/kqt0LYLwFd){ target=_blank } — per-system up/down and response times
* [UNM IT alerts](https://italerts.unm.edu/){ target=_blank } — campus-wide IT outage notices
* [Network performance](http://perfsonar.alliance.unm.edu){ target=_blank } — perfSONAR measurements for the CARC network
* [Easley external DNS check](https://dnschecker.org/#A/easley.alliance.unm.edu){ target=_blank } — worldwide resolution of easley.alliance.unm.edu
* [System usage (XDMoD)](https://xdmod.alliance.unm.edu/){ target=_blank } — utilization metrics by system and principal investigator

## Networking

CARC systems connect to campus through multiple 10 Gbps links, including a
dedicated 10 Gbps connection to UNM's Science DMZ research network. External
connectivity includes 100 Gbps connections to ESnet and the Western Regional
Network through the Albuquerque Gigapop.

## Export control

The U.S. Government controls the export of sensitive equipment, software, and
technology. Installation of export-controlled software on CARC systems
requires prior written approval from
[UNM Export Control](https://carc.unm.edu/systems/export-control.html){ target=_blank } —
see the [Good Neighbor Use Policy](../getting-started/good-neighbor-policy.md).

---8<--- https://unm-carc.github.io/docs/systems/resource-limits/

---
title: "Storage and compute usage policies"
description: "Storage quotas, Slurm fairshare policy, and per-cluster queue limits."
type: Policy
tags:
  - Policy
  - Storage
  - Slurm
stale_after: "2027-08-31T00:00:00Z"
generated:
  by: "claude/fable-5"
  at: "2026-08-29T00:00:00Z"
sources:
  - id: webinfo
    resource: "https://github.com/UNM-CARC/webinfo/blob/master/resource_limits.md"
    title: "UNM-CARC webinfo: resource_limits.md"
    author: "team:unm-carc"
    last_modified: "2023-04-21T16:04:29-06:00"
---

# Storage and compute usage policies

Home directories have a soft limit of 100GB and a hard limit of 200GB. Once you exceed the soft limit we will ask you to reduce your usage. You will not be able to write data beyond the hard limit. Project space is limited to 250 GB. Scratch storage is limited to 1 TB. Center-wide project scratch space is limited to 1 TB and user scratch is limited to 100G (`/carc/scratch`). To purchase additional storage please see our [pricing spreadsheet](https://carc.unm.edu/research/premium-research-computing-services.html){target=_blank}.

The `quotas` command shows your quota usage.

## Compute usage policy

To ensure that all research and class projects get their fair share of the clusters and to prevent any one group from using a disproportionate amount of resources, we utilize Slurm’s built-in job accounting and fairshare system.
The cluster is a limited resource and Fairshare allows us to ensure everyone gets a fair opportunity to use it regardless of how big or small the group is.
Your compute resource allocation is shared among everyone in the slurm account you select. 

For more on Slurm accounts see [Slurm accounting](../running-jobs/slurm-accounting.md).
Note that your slurm account is not the same as your CARC login account.

To see the predicted start time of your job based on your fairshare score, use the following command:
`squeue --start --job <job_id>`

## Hopper Configuration


|                Queue: |   General  |    Debug   | Condo    | Private |
|----------------------:|:----------:|:----------:| :---:    | :---:   |
| Number of Processors  |     64     |     8      | 192      | Determined by <br> queue owner |
|      Number of Nodes  |     2      |      2     | 6        |  |
|   Processors per Node |    32      |      8     | 32       |  |
|       Walltime(H:M:S) |  48:00:00  |  04:00:00  | 48:00:00 |  |
|        Memory Limits  |    90 Gb   |    90 Gb   |          |  |

<p class="carc-provenance" markdown>Migrated from [UNM-CARC QuickBytes](https://github.com/UNM-CARC/webinfo/blob/master/resource_limits.md){target=_blank} (last source update 2023-04-21). Spotted a problem? [Open an issue or pull request](https://github.com/UNM-CARC/QuickBytes){target=_blank}.</p>

---8<--- https://unm-carc.github.io/docs/systems/storage-permissions/

---
title: "Storage permissions on BeeGFS"
description: "Manage file and directory permissions, including ACLs, on CARC BeeGFS scratch storage."
type: Guide
tags:
  - Storage
  - Security
generated:
  by: "claude/fable-5"
  at: "2026-08-29T00:00:00Z"
sources:
  - id: quickbytes
    resource: "https://github.com/UNM-CARC/QuickBytes/blob/master/storage_permissions_BeeGFS.md"
    title: "UNM-CARC QuickBytes: storage_permissions_BeeGFS.md"
    author: "team:unm-carc"
    last_modified: "2026-06-17T13:48:26-06:00"
---

# Storage permissions on BeeGFS

CARC Scratch project directories are located under paths like:

```text
/carc/scratch/projects/<pi_username>/<pi_username><project_id>
```

These directories are on BeeGFS, where quotas are enforced by **group ownership**.

This can be confusing because a file can be physically located inside a project directory but still be charged against a user's personal quota if the file is owned by the wrong group.

The key point is:

!!! warning "Key point"

    **Location does not determine quota usage. Group ownership does.**

---

## The common problem

A user copies files into a project directory and assumes the files now count against the project quota.

For example:

```text
/carc/scratch/projects/smith/smith12345/data/
```

But if the files are owned by the user's personal group rather than the project group, BeeGFS may charge that storage to the user's personal group quota instead.

That can cause confusing quota errors:

!!! danger "Why am I out of personal quota?"

    *"I copied the data to the project directory, so why am I out of
    personal quota?"*

    The answer is usually that the files are in the right place, but they
    are owned by the wrong group.

---

## Users, groups, and file ownership

Every Linux file has:

1. A user owner
2. A group owner
3. Permissions

You can see these with:

```bash
ls -l
```

Example:

```text
-rw-r----- 1 jsmith smith12345 1048576 Jun 10 14:22 data.csv
```

This means:

```text
-rw-r-----   permissions
jsmith       user owner
smith12345   group owner
```

For BeeGFS project quota accounting, the important field is usually the **group owner**.

In this example, the file belongs to the `smith12345` group, so it should count against the `smith12345` project quota.

---

## Project group versus personal group

Suppose your project directory is:

```text
/carc/scratch/projects/smith/smith12345
```

and the project group is:

```text
smith12345
```

A file like this is usually charged to the project quota:

```text
-rw-r----- 1 jsmith smith12345 1048576 Jun 10 14:22 data.csv
```

A file like this may be charged to the user's personal group quota:

```text
-rw-r----- 1 jsmith jsmith 1048576 Jun 10 14:22 data.csv
```

Both files can live in the same project directory. The difference is the group owner.

---

## Checking ownership

To check the ownership of files:

```bash
ls -l
```

To check the ownership of the current directory:

```bash
ls -ld .
```

To check the ownership of a project directory:

```bash
ls -ld /carc/scratch/projects/<pi_username>/<pi_username><project_id>
```

Example:

```text
drwxrws--- 25 root smith12345 4096 Jun 10 14:00 /carc/scratch/projects/smith/smith12345
```

The group owner here is:

```text
smith12345
```

That is the project group users should normally expect files to use inside this project directory.

---

## Checking your groups

To see which groups your account belongs to:

```bash
id
```

or:

```bash
groups
```

Example:

```text
uid=12345(jsmith) gid=12345(jsmith) groups=12345(jsmith),67890(smith12345)
```

This user belongs to both:

```text
jsmith
smith12345
```

If you are not a member of the project group, you may not be able to create files with that group ownership.

---

## Permissions are not ownership

Linux permissions and Linux ownership are related, but they are not the same thing.

Permissions control who can read, write, or execute a file.

Ownership controls which user and group own the file.

Two files can have identical permissions but different group owners:

```text
-rw-r----- 1 jsmith jsmith     1048576 Jun 10 14:22 personal_file.dat
-rw-r----- 1 jsmith smith12345 1048576 Jun 10 14:22 project_file.dat
```

For BeeGFS quota purposes, the important difference is the group owner:

```text
jsmith
smith12345
```

Changing file permissions does not necessarily change quota accounting. If the group owner is wrong, the quota charge may be wrong.

---

## The setgid bit on project directories

Many shared project directories are configured with the setgid bit.

You can see this in the directory permissions:

```text
drwxrws---
```

The `s` in the group permission field means the directory has the setgid bit set.

This usually tells Linux:

!!! note "What setgid means"

    New files and directories created here should inherit the group
    ownership of the parent directory.

That helps project directories behave as shared spaces.

!!! warning "setgid is not foolproof"

    Some copy tools, synchronization programs, editors, and applications
    may:

    - preserve the source group
    - explicitly set their own group
    - create temporary files elsewhere and then move them into place
    - use transfer behavior that bypasses the expected destination ownership

    Because of this, always verify group ownership after large transfers.

---

## Fixing group ownership

If files are in the project directory but owned by the wrong group, the group can often be corrected with `chgrp`.

Example:

```bash
chgrp -R smith12345 mydata
```

This recursively changes the group owner of `mydata` to `smith12345`.

To fix files in the current directory:

```bash
chgrp -R smith12345 .
```

Then verify:

```bash
ls -l
```

Alternatively, you can change both the user and group owner at once with `chown`:

```bash
chown -R jsmith:smith12345 mydata
```

For most users, `chgrp` is usually the safer command to document because it changes only the group owner.

---

## Recommended transfer method: rsync

For Linux-to-Linux transfers, `rsync` is usually preferred.

Basic example:

```bash
rsync -av source/ /carc/scratch/projects/smith/smith12345/data/
```

If the user has permission to set the group ownership, `rsync` can explicitly assign ownership during transfer:

```bash
rsync -av --chown=$USER:smith12345 source/ /carc/scratch/projects/smith/smith12345/data/
```

Replace `smith12345` with the actual project group.

After transfer, verify:

```bash
ls -l /carc/scratch/projects/smith/smith12345/data/
```

For large transfers, this is often safer than copying files and fixing ownership afterward.

---

## Warning about scp

`scp` is convenient, but it is not ideal for this quota problem.

In particular:

```text
scp does not have an --chown option
```

Files copied with `scp` may not end up with the expected project group ownership.

If you use `scp`, check the files afterward:

```bash
ls -l
```

If the group owner is wrong, fix it:

```bash
chgrp -R smith12345 .
```

For large project transfers, prefer `rsync` when possible.

---

## Copying files from Windows

Windows file systems do not use Linux user and group ownership in the same way.

When files are uploaded from Windows using tools such as:

- WinSCP
- FileZilla
- MobaXterm
- VS Code Remote SSH
- Globus, depending on endpoint configuration
- WSL with `rsync`

the resulting ownership is assigned on the Linux side.

That means Windows users should always verify ownership after upload:

```bash
ls -l
```

If files landed with the wrong group:

```bash
chgrp -R smith12345 mydata
```

For large or repeated transfers from Windows, the most reliable options are usually:

1. Use WSL and `rsync`
2. Use a transfer tool that allows post-transfer commands
3. Upload the files, then run `chgrp -R <project_group>` on the cluster

Example using WSL with `rsync`:

```bash
rsync -av --chown=$USER:smith12345 /mnt/c/Users/jsmith/data/ username@cluster:/carc/scratch/projects/smith/smith12345/data/
```

If `--chown` is not permitted, transfer the files and then run `chgrp` on the cluster.

---

## Recommended user workflow

After copying data into project storage:

1. Go to the project directory.

```bash
cd /carc/scratch/projects/<pi_username>/<pi_username><project_id>
```

2. Check ownership.

```bash
ls -l
```

3. Confirm that the group owner is the project group.

Good:

```text
-rw-r----- 1 jsmith smith12345 1048576 Jun 10 14:22 data.csv
```

Possibly wrong:

```text
-rw-r----- 1 jsmith jsmith 1048576 Jun 10 14:22 data.csv
```

4. Fix group ownership if needed.

```bash
chgrp -R smith12345 .
```

5. Check again.

```bash
ls -l
```

---

## Summary

Remember:

- BeeGFS project quotas are based on group ownership.
- A file can be inside a project directory but still count against the wrong quota.
- Use `ls -l` to check file ownership.
- Use `ls -ld` to check directory ownership.
- The important field is the group owner.
- Permissions and ownership are not the same thing.
- The setgid bit helps new files inherit the project group, but some tools can still produce unexpected ownership.
- `rsync --chown=$USER:<project_group>` is often the safest transfer method.
- `scp` does not provide a reliable way to set destination group ownership.
- Windows transfer tools may require a follow-up `chgrp`.
- When in doubt, check with `ls -l` before starting a large job.
- When using `chgrp`, make sure to copy your data in chunks where each chunk is smaller than your remaining personal scratch quota.
- You can always check your quotas with the `quotas` command.

*This quickbyte was validated on 6/17/2026*

<p class="carc-provenance" markdown>Migrated from [UNM-CARC QuickBytes](https://github.com/UNM-CARC/QuickBytes/blob/master/storage_permissions_BeeGFS.md){target=_blank} (last source update 2026-06-17). Spotted a problem? [Open an issue or pull request](https://github.com/UNM-CARC/QuickBytes){target=_blank}.</p>

---8<--- https://unm-carc.github.io/docs/systems/storage/

---
title: "Storage and backups"
description: "CARC storage spaces (home, project, scratch), where to compute from, and what is backed up."
type: Guide
tags:
  - Storage
  - Data
generated:
  by: "claude/fable-5"
  at: "2026-08-29T00:00:00Z"
sources:
  - id: quickbytes
    resource: "https://github.com/UNM-CARC/QuickBytes/blob/master/storage_and_backup.md"
    title: "UNM-CARC QuickBytes: storage_and_backup.md"
    author: "team:unm-carc"
    last_modified: "2021-03-18T12:38:03-06:00"
---

# Storage and backups

CARC provides home, project, and scratch directories on all compute systems.

User home directories are subdirectories of `/users`. User project directories are subdirectories of `/project`. Scratch directories are subdirectories of `/scratch`.

## Storage at CARC

CARC supports a number of different storage devices for users to read, write, and store their data and results on.  The different devices vary in size, I/O speed (Figure 1), distance to the reading/writing process using them, whether they have quotas to limit their use, and whether they are backed up or not. Certain storage devices, like the home directories and machine-scratch’s, are shared resources, and serve multiple users simultaneously. This means that problems caused by poorly chosen storage locations could interrupt the work of other users as well as your own. Don’t hesitate to open a help ticket if you have any questions about, for example, where to store a large dataset, how to read it quickly, how to write a Slurm batch script that first moves data onto the compute node in order to take advantage of the fast I/O, or any other issue related to storage at CARC.

For storage limits see the [resource limits page](resource-limits.md).

## Types of Storage

At CARC, there are four different types of storage:

* Home directory - `/users/username` - Upon logging into any CARC machine, you will find yourself in your home directory, in `/users/username`, replacing “username” with your actual username (note, the home directory also goes by `~` and `$HOME`). You may notice that regardless of which machine you login to, the contents of your home directory are identical. This is because they are not part of any specific machine, but rather they are stored on separate computer entirely, and are then mounted by every head and compute node at CARC.

* Machine-wide scratch disk - `~/machine-scratch` → `/machine/scratch/username` - The most common place where data is stored after it is generated by running calculations, and before it is further analyses and then either downloaded, deleted, or moved when a long term data storage or archival plan is implemented. These are not backed up and CARC reserves the right to delete this data without advanced warning.

* Hard drive (only on the compute nodes) - `/tmp` - On the machines that support these, compute nodes have their own hard drives installed and they can be accessed simply by creating a directory in `/tmp` and then placing data there (see the sample script below). Since the hard drive is dedicated to that compute node, this is one of the fastest places for I/O. Files stores here will be cleared at the end of the job.

* Shared memory (only on the compute nodes) - `/dev/shm` - This is actually direct access to the machine's memory for use in storage. The directory at `/dev/shm` appears to the user as a normal read/write accessible directory just like `/tmp`, but files written or read from any directory within are simply being stored into memory as if they were on a disk. This provides extremely fast I/O speeds, and is very useful if small temporary files are written and read often. Be careful with the amount of data you write here, however, because this competes directly with all processes using the compute nodes RAM, including yours. Like `/tmp`, `/dev/shm` is also cleared at the end of a job, so you must move any data you want to keep off the compute node before the end of the calculation or walltime.

![storage_img](../assets/images/quickbytes/storage_diagram.jpeg)

Figure 1. The various storage locations plotted by their relative size and I/O speed.

## Choosing a storage type

In order to determine which storage type to use, it may be helpful to consider which of the following broad categories your data falls into:

1. Data that is hard to produce and long-lived - Eg. source code, scripts, documents, results. This data is either used to produce or is the product of other calculations and work

2. Results and temporary data - This is data that is produced by your calculations such as simulation logs and are typically further analyzed or have some data of interest extracted and summarized, perhaps even reported in a publication. This data can be regenerated relatively easily, by rerunning the calculation that produced it, and is often deleted once further processed or at most, at the end of a project.

We recommend that the first type of data be stored in your home directory, which is available from everywhere inside the CARC network, and is backed up each week. The second kind of data is best produced/stored on the machine-scratch drives. In cases of very high I/O, that data can temporarily be moved to the compute node and onto either the hard drive (if the machine in use has an internal hard drive) or onto shared memory (if the files are very small). An obvious benifit to using devices that are very close to the CPU and are typically only used by a single user at a time, is the high read and write speed availible (Figure 1). The cost of this speed is the fact that the data must be moved to the compute node, read in by the calculation, and anything that is intended to be saved must be moved back off the compute node before the end of walltime or before the job ends, when the compute node is returned to the general resource pool.

## Example use of a node's hard drive

This Slurm script first copies an input file (large_input_data.dat) to the compute node, runs the calculation ("run_my_program"), and then copies all results back into the submission directory:

```bash
#!/bin/bash
#SBATCH --nodes=1
#SBATCH --ntasks=8
#SBATCH --time=1:00:00
#SBATCH --job-name=local_storage

# Define a directory on the node-local disk, create it once the job
# has started, move data there, then cd to it and run
TEMP_DIR=/tmp/$USER/$SLURM_JOB_ID
mkdir -p "$TEMP_DIR"
cp -r $SLURM_SUBMIT_DIR/large_input_data.dat "$TEMP_DIR"
cd $TEMP_DIR

# Now run my program
run_my_program

# The job has finished, so move data back to where it came from
cp -r $TEMP_DIR/* $SLURM_SUBMIT_DIR

# Finally clean up the temporary directory
rm -r $TEMP_DIR
```

## Video walkthrough

**Storage Systems** — from the [CARC video tutorials](../training/videos.md):

<iframe class="carc-video" src="https://www.youtube-nocookie.com/embed/WwsbLyl7d1A" title="Storage Systems" loading="lazy" allow="accelerometer; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen></iframe>

<p class="carc-provenance" markdown>Migrated from [UNM-CARC QuickBytes](https://github.com/UNM-CARC/QuickBytes/blob/master/storage_and_backup.md){target=_blank} (last source update 2021-03-18). Spotted a problem? [Open an issue or pull request](https://github.com/UNM-CARC/QuickBytes){target=_blank}.</p>

---8<--- https://unm-carc.github.io/docs/running-jobs/example-slurm-scripts/

---
title: "Example Slurm scripts"
description: "Ready-to-adapt sbatch scripts for serial, parallel, and GPU jobs."
type: Reference
tags:
  - Slurm
  - Jobs
  - Examples
generated:
  by: "claude/fable-5"
  at: "2026-08-29T00:00:00Z"
sources:
  - id: quickbytes
    resource: "https://github.com/UNM-CARC/QuickBytes/blob/master/submitting_sbatch_jobs.md"
    title: "UNM-CARC QuickBytes: submitting_sbatch_jobs.md"
    author: "team:unm-carc"
    last_modified: "2026-06-17T14:22:00-06:00"
---

# Example Slurm scripts

To submit a job script to SLURM, use the `sbatch` command followed by the name of your script:

```bash
sbatch my_script.sh
```

This page walks through several example scripts of increasing complexity, then covers how to submit and run them.

---

## Creating a Script File

Before you can submit a job, you need to create a script file. You can use any text editor available on Easley. The three most common are:

**nano** — simplest, recommended for beginners:
```bash
nano my_script.sh
```
When finished editing, save with `Ctrl+O` then `Enter`, and exit with `Ctrl+X`.

**vi/vim** — available on every Unix system:
```bash
vi my_script.sh
```
Press `i` to enter insert mode and start typing. When finished, press `Esc`, then type `:wq` and hit `Enter` to save and quit. Type `:q!` to quit without saving.

**emacs** — feature-rich editor:
```bash
emacs my_script.sh
```
When finished editing, save with `Ctrl+X Ctrl+S` and exit with `Ctrl+X Ctrl+C`.

---

## Hello World

This example uses the Bash shell to print a simple "Hello World" message. The shell is specified by the shebang line at the top of the script (`#!/bin/bash`). If you do not specify a shell, your default shell will be used. Since this script uses only built-in Bash commands, no software modules are loaded — module usage is introduced in the next example.

Create a file called `hello.sh` and add the following:

```bash
#!/bin/bash
## Introduction for writing a Slurm script
## Requesting 1 node, 8 processors per node, and 2 hours of walltime.

#SBATCH --job-name=my_job
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=8
#SBATCH --time=02:00:00
#SBATCH --partition=general
#SBATCH --mail-user=myemailaddress@unm.edu
#SBATCH --mail-type=BEGIN,END,FAIL

## Change to the directory the Slurm script was submitted from
cd $SLURM_SUBMIT_DIR

## Print a hello message indicating the host this is running on
export THIS_HOST=$(hostname)
echo "Hello World from host $THIS_HOST"
```

The number of tasks per node (`--ntasks-per-node`) must always be less than or equal to the number of physical CPU cores available on the node. On Easley, `--ntasks-per-node` should be <=64. We recommend always requesting the maximum number of processors per node to avoid multiple jobs on one node having to share memory. For more information see CARC systems information.

---

## Multi-Processor Example

This example runs a command across multiple CPUs on a single node using MPI.

Create a file called `multiprocessor.sh` and add the following:

```bash
#!/bin/bash
## Multi-processor example
## The Center for Advanced Research Computing
## at The University of New Mexico

#SBATCH --job-name=my_job
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=8
#SBATCH --time=02:00:00
#SBATCH --partition=general
#SBATCH --mail-user=myemailaddress@unm.edu
#SBATCH --mail-type=BEGIN,END,FAIL

# Load the OpenMPI module
module load openmpi

# Change to the directory where the Slurm script was submitted from
cd $SLURM_SUBMIT_DIR

# Run "hostname" on every CPU
# $SLURM_NTASKS is the total number of CPUs requested: 1 node x 8 CPUs = 8
mpirun -np $SLURM_NTASKS hostname
```

---

## Multi-Node Example

This example spreads work across multiple nodes using MPI. When running across multiple nodes, use `srun` instead of `mpirun` — SLURM passes node information to `srun` automatically.

Create a file called `multinode.sh` and add the following:

```bash
#!/bin/bash
## Multi-node example
## The Center for Advanced Research Computing
## at The University of New Mexico

#SBATCH --job-name=my_job
#SBATCH --nodes=4
#SBATCH --ntasks-per-node=8
#SBATCH --time=02:00:00
#SBATCH --partition=general
#SBATCH --mail-user=myemailaddress@unm.edu
#SBATCH --mail-type=BEGIN,END,FAIL

# Change to the directory the Slurm script was submitted from
cd $SLURM_SUBMIT_DIR

# Load the OpenMPI module
module load openmpi

# Run "hostname" on every CPU across all nodes
# $SLURM_NTASKS is the total number of CPUs requested: 4 nodes x 8 CPUs = 32
srun hostname
```

---

## Submitting Jobs

### Batch Submission

To submit a script to the batch scheduler, use `sbatch` followed by the script name. SLURM returns a job ID when the job is successfully submitted, which you can use to check the status of your job.

```bash
sbatch hello.sh
```

```
Submitted batch job 156452
```

Check the status of your job with:

```bash
squeue --me
```

Once the job completes, the output will be written to a file named `slurm-<jobid>.out` in the directory you submitted from. View it with:

```bash
cat slurm-156452.out
```

```
Hello World from host easley004
```

For more information on available options type `man sbatch`.

### Interactive Jobs

At times — such as when debugging — it can be useful to run a job interactively. Use `salloc` followed by your resource request, and SLURM will allocate a node and log you into it directly.

```bash
salloc --nodes=1 --ntasks=8 --time=00:05:00
```

```
salloc: Granted job allocation 156469
salloc: Nodes easley004 are ready for job
```

Once on the node, load your modules and run your script normally:

```bash
module load openmpi
bash helloworld_parallel.sh
```

```
Hello World from host easley004
Hello World from host easley004
Hello World from host easley004
Hello World from host easley004
Hello World from host easley004
Hello World from host easley004
Hello World from host easley004
Hello World from host easley004
```

When you are finished, type `exit` to release the node back to the pool.

> **Note:** CARC recommends submitting jobs via `sbatch` wherever possible, as job submission will catch errors in resource requests before the job runs. Reserve interactive sessions for debugging and development.

*This quickbyte was validated on 6/17/2026*

<p class="carc-provenance" markdown>Migrated from [UNM-CARC QuickBytes](https://github.com/UNM-CARC/QuickBytes/blob/master/submitting_sbatch_jobs.md){target=_blank} (last source update 2026-06-17). Spotted a problem? [Open an issue or pull request](https://github.com/UNM-CARC/QuickBytes){target=_blank}.</p>

---8<--- https://unm-carc.github.io/docs/running-jobs/gnu-parallel/

---
title: "GNU Parallel"
description: "Run many small tasks efficiently inside a single Slurm job with GNU Parallel."
type: Guide
tags:
  - Slurm
  - Parallel
generated:
  by: "claude/fable-5"
  at: "2026-08-29T00:00:00Z"
sources:
  - id: quickbytes
    resource: "https://github.com/UNM-CARC/QuickBytes/blob/master/GNU%20Parallel.md"
    title: "UNM-CARC QuickBytes: GNU Parallel.md"
    author: "team:unm-carc"
    last_modified: "2026-07-07T11:08:42-06:00"
---

# GNU Parallel

## Overview

GNU Parallel allows you to run multiple independent jobs simultaneously instead of sequentially, reducing total execution time.

Instead of creating a queue of processes that execute one after another, GNU Parallel distributes work across available CPU resources. This approach is especially effective for embarrassingly parallel workloads, where tasks do not depend on one another.

Typical use cases include:

* Batch file conversion
* Compression and decompression
* Parameter sweeps
* High-throughput MATLAB workflows
* Parallel Python execution

---

## Basic GNU Parallel Examples

### Converting Files in Parallel

> **Known issue:** the `imagemagick` module is currently unavailable on Easley. A fix is expected within a day — until then, skip this example.

The following example converts all `.csv` files into `.txt` files.

```bash
module load parallel
module load imagemagick

find . -name "*.csv" | parallel convert {} {.}.txt
```

This command:

1. Finds all files ending in `.csv`
2. Passes each file to GNU Parallel
3. Executes conversions concurrently

`{}` represents the full filename and `{.}` removes the extension.

---

### Compressing and Decompressing Files

`parallel` isn't on your `PATH` by default — load it once per session:

```bash
module load parallel
```

Compress files:

```bash
parallel gzip ::: *.txt
```

Decompress files:

```bash
parallel gunzip ::: *.gz
```

This pattern also works for collections of images or other independent files.

---

## MATLAB with GNU Parallel

GNU Parallel supports running many MATLAB processes simultaneously.

Two commonly used features are:

* `--arg-file` — read inputs from a file
* `{}` — substitute each input line into the command

Create an input file named `msizes`:

```bash
cat > msizes <<'EOF'
1
2
3
4
EOF
```

And the MATLAB program it drives, `program.m`:

```bash
cat > program.m <<'EOF'
% Generate a random matrix
rmatrix = rand(msize);

% Create output filename
fname = sprintf('%d.csv', msize);

% Save results
writematrix(rmatrix, fname);

exit
EOF
```

Running large numbers of MATLAB jobs directly on the login node is not allowed — request an interactive allocation first:

```bash
srun --nodes=1 --ntasks-per-node=4 --pty bash
```

Then, on the allocated compute node, load both modules and launch MATLAB once per line in `msizes`:

```bash
module load matlab
module load parallel

parallel --arg-file msizes \
    'matlab -batch "msize={}; program"'
```

This launches four MATLAB jobs simultaneously.

`matlab -batch` is preferred over older combinations such as `-nojvm -nodisplay -r`.

Example output:

```text
1.csv
2.csv
3.csv
4.csv
```

---

## Running Across Multiple Nodes with `--sshloginfile`

Slurm allocates the nodes for your job, but GNU Parallel has no automatic awareness of any node besides the one your script is actually running on. Without telling it otherwise, `parallel -j N` runs all `N` jobs on that single node — oversubscribing it — rather than spreading them across every node you requested with `--nodes`.

To actually use every allocated node, pass `--sshloginfile "$CARC_NODEFILE"`. CARC's Slurm setup exposes the list of nodes in your current allocation through the `$CARC_NODEFILE` environment variable (the Slurm-era equivalent of the old PBS `$PBS_NODEFILE`).

You can confirm this yourself on an interactive multi-node allocation:

```bash
srun --nodes 2 --pty bash
```

```bash
echo $CARC_NODEFILE
cat $CARC_NODEFILE
```

When you add `--sshloginfile`, set `-j` to the tasks **per node** (`$SLURM_NTASKS_PER_NODE`), not the job's total task count (`$SLURM_NTASKS`) — `--sshloginfile` already handles spreading work across nodes, so using the total would oversubscribe each individual node instead.

### One-Time Setup: Passwordless SSH Between Compute Nodes

`--sshloginfile` works by having GNU Parallel `ssh` from the node running your script to every other node in your allocation. Since your home directory is shared (NFS) across all compute nodes, you only need to do this once, ever — not per job.

**1. Generate a keypair and trust it for yourself** — this checks first and does nothing if you already have one:

```bash
if [ ! -f ~/.ssh/id_ed25519 ]; then
    ssh-keygen -t ed25519 -f ~/.ssh/id_ed25519 -N ""
fi
grep -qxF "$(cat ~/.ssh/id_ed25519.pub)" ~/.ssh/authorized_keys 2>/dev/null \
    || cat ~/.ssh/id_ed25519.pub >> ~/.ssh/authorized_keys
chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys ~/.ssh/id_ed25519
```

If you already use `~/.ssh/id_ed25519` for something else (e.g. GitHub), use a different filename (e.g. `~/.ssh/id_ed25519_carc`) throughout this section instead — just make sure the same key ends up in your own `authorized_keys`.

**2. Pre-trust every node's host key**, so a job landing on a node you've never connected to before doesn't hang waiting on a host-key prompt that a batch script has no terminal to answer:

```bash
for i in $(seq -w 1 63); do echo easley0$i; done | ssh-keyscan -f - >> ~/.ssh/known_hosts 2>/dev/null
```

Without both of these steps, a `--sshloginfile` job will silently hang until it hits its time limit — there's no error message, since the script is stuck waiting on a password/host-key prompt that never arrives.

---

## Running MATLAB at Scale with Slurm

Running large numbers of MATLAB jobs directly on login nodes is not recommended, so please use Slurm to allocate compute resources.

Create the submission script (uses the same `msizes` and `program.m` from the single-node example above — recreate them here if you're starting fresh):

```bash
cat > matlab_parallel.slurm <<'EOF'
#!/bin/bash

#SBATCH --job-name=matlab_parallel
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=4
#SBATCH --time=00:10:00

module load matlab
module load parallel

cd "$SLURM_SUBMIT_DIR"

echo "Starting MATLAB jobs at $(date)"

parallel \
    -j "$SLURM_NTASKS_PER_NODE" \
    --sshloginfile "$CARC_NODEFILE" \
    --workdir "$SLURM_SUBMIT_DIR" \
    --env PATH \
    --arg-file msizes \
    'matlab -batch "msize={}; program"'

echo "Finished at $(date)"
EOF
```

`--workdir "$SLURM_SUBMIT_DIR"` is required because SSH sessions default to your home directory, not wherever this script's own `cd` took you — without it, MATLAB can't find `program.m`. `--env PATH` forwards the module-loaded PATH (with `matlab` on it) to the remote SSH sessions, which otherwise start with a bare, non-login environment that doesn't have `module`-loaded paths at all.

Submit the job:

```bash
sbatch matlab_parallel.slurm
```

---

## Matching Jobs to Resources

When using GNU Parallel, the number of jobs should approximately match the number of allocated tasks.

General guideline:

```text
nodes × tasks-per-node ≈ number of parallel jobs
```

Examples:

| Nodes | Tasks per Node | Parallel Jobs |
| ----- | -------------- | ------------- |
| 1     | 8              | 8             |
| 2     | 8              | 16            |
| 4     | 16             | 64            |

Running substantially more jobs than available CPU resources may increase runtime due to oversubscription.

---

## Running Embarrassingly Parallel Python Jobs

GNU Parallel can also distribute Python workloads.

### One-Time Setup: Conda Environment

Create the environment this example uses (only needs to be done once):

```bash
module load miniconda3
conda create -n numpy_py3 python=3.11 numpy -y
```

### Example Python Program

Create `matrix_inv.py`:

```bash
cat > matrix_inv.py <<'EOF'
import argparse
import numpy as np
from numpy.random import rand
from numpy.linalg import inv


parser = argparse.ArgumentParser()
parser.add_argument("matrix", type=int)

args = parser.parse_args()


def matinv(size):
    return inv(rand(size, size))


result = matinv(args.matrix)

np.savetxt(
    f"{args.matrix}.csv",
    result,
    delimiter=","
)
EOF
```

---

### Example Input File

Create `mat_in` — each line becomes one parallel task:

```bash
cat > mat_in <<'EOF'
1000
2000
3000
4000
5000
6000
7000
8000
EOF
```

---

### Example Slurm Script

```bash
cat > python_parallel.slurm <<'EOF'
#!/bin/bash

#SBATCH --job-name=gnu_parallel_python
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=4
#SBATCH --time=01:00:00

module load parallel
module load miniconda3

source activate numpy_py3

cd "$SLURM_SUBMIT_DIR"

parallel \
    -j "$SLURM_NTASKS_PER_NODE" \
    --sshloginfile "$CARC_NODEFILE" \
    --workdir "$SLURM_SUBMIT_DIR" \
    --env PATH \
    --arg-file mat_in \
    python matrix_inv.py
EOF
```

Note: this uses plain `parallel` with `--env PATH`, not `env_parallel`. `env_parallel` forwards your *entire* shell environment to each remote session, and conda's activation hooks add enough bloat to that environment to blow past the shell's argument-length limit (`Command line too long`). `--env PATH` forwards just the one variable that's actually needed to find `python` in the activated environment.

Submit with:

```bash
sbatch python_parallel.slurm
```

---

## Notes

* Use Slurm allocations for production workloads.
* GNU Parallel works best for independent tasks with minimal communication.
* Match the number of jobs to allocated resources whenever possible.

---

*This QuickByte was updated and validated on June 23, 2026.*

## Video walkthrough

**Parallelization 3: GNU Parallel** — from the [CARC video tutorials](../training/videos.md):

<iframe class="carc-video" src="https://www.youtube-nocookie.com/embed/Rl06WD60afA" title="Parallelization 3: GNU Parallel" loading="lazy" allow="accelerometer; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen></iframe>

<p class="carc-provenance" markdown>Migrated from [UNM-CARC QuickBytes](https://github.com/UNM-CARC/QuickBytes/blob/master/GNU%20Parallel.md){target=_blank} (last source update 2026-07-07). Spotted a problem? [Open an issue or pull request](https://github.com/UNM-CARC/QuickBytes){target=_blank}.</p>

---8<--- https://unm-carc.github.io/docs/running-jobs/modules/

---
title: "Environment modules"
description: "Find, load, and manage software with environment modules on CARC clusters."
type: Guide
tags:
  - Modules
  - Software
generated:
  by: "claude/fable-5"
  at: "2026-08-29T00:00:00Z"
sources:
  - id: quickbytes
    resource: "https://github.com/UNM-CARC/QuickBytes/blob/master/module_management.md"
    title: "UNM-CARC QuickBytes: module_management.md"
    author: "team:unm-carc"
    last_modified: "2026-07-07T11:23:33-06:00"
---

# Environment modules

### Modules

There are many software packages installed on CARC systems, as well as standard built-in functions native to Unix. To manage these additional software packages, CARC systems use modules. These modules set the appropriate environment variables and dependencies for software optimization and to avoid conflicts with other software.

For more information, visit [this page](https://lmod.readthedocs.io/en/latest/010_user.html){target=_blank}, or use the command `module help`.

### Using modules to set application environments

Modules are used to set environment variables and dependencies for the purpose of managing access to applications and libraries on CARC systems. The command `module avail` lists all the modules available on the system you are logged into. Note that this list can be extremely long — if you'd like to stop it from printing, use Ctrl+C (this works the same way on Mac, Windows, and Linux terminals, since you're connected to a remote Linux system either way).

To load a module, use the `module load` command. For example, to load the module for the Intel compilers, use the command:

```bash
module load intel-oneapi-compilers
```

Note that there's no plain `intel` module on Easley — `module load intel` fails with "The following module(s) are unknown." Easley's Intel software all lives under the `intel-oneapi-*` family instead (`intel-oneapi-compilers`, `intel-oneapi-mkl`, `intel-oneapi-mpi`, etc.).

Another useful command related to module management is `module spider`. For example, if you issue the command:

```bash
module spider intel
```

you will see output similar to:

```
------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  intel-oneapi-advisor:
------------------------------------------------------------------------------------------------------------------------------------------------------------------------
     Versions:
        intel-oneapi-advisor/2025.2.0-asm7
        intel-oneapi-advisor/2025.2.0-omxu
------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  For detailed information about a specific "intel-oneapi-advisor" package (including how to load the modules) use the module's full name.
  ...

------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  intel-oneapi-compilers: intel-oneapi-compilers/2025.1.1-b3qi
------------------------------------------------------------------------------------------------------------------------------------------------------------------------
    This module can be loaded directly: module load intel-oneapi-compilers/2025.1.1-b3qi
...
```

`spider` matches on partial names, so searching `intel` surfaces the entire `intel-oneapi-*` family at once — you can see there are multiple related Intel packages available, each with its own version(s), as is the case for most software installed on CARC systems.

To see all currently loaded modules, use the command `module list`. As an example, let's load the software modules for OpenMPI and GCC, then use `module list`:

```bash
module load openmpi gcc
```

```bash
module list
```

```
Currently Loaded Modules:
  1) miniconda3/latest             3) gcc/14.2.0-j33x             5) openmpi/4.1.7-762w
  2) binutils/2.43.1-ifi2qjn (H)   4) openssh/9.9p1-d4o73h6 (H)
  Where:
   H:  Hidden Module
```

Modules are usually loaded as part of a Slurm script, and that environment doesn't persist beyond the job, so `module avail` and `module load` are the main commands you'll be using day to day. However, if you're working on a node interactively, you may need to unload modules manually. The command `module unload modulename` unloads modules one at a time — for example, after loading the modules above:

```bash
module unload openssh
```

```
Lmod Warning: 
--------------------------------------------------------------------------------------------------------
The following dependent module(s) are not currently loaded: openssh/9.9p1-d4o73h6 (required by:
openmpi/4.1.7-762w)
--------------------------------------------------------------------------------------------------------
```

This warning is expected and can be safely ignored — Lmod is just noting that OpenMPI normally depends on OpenSSH, but it doesn't stop the module from being unloaded. Running `module list` again confirms OpenSSH is gone while the rest remain loaded:

```bash
module list
```

```
Currently Loaded Modules:
  1) binutils/2.43.1-ifi2qjn (H)   2) gcc/14.2.0-j33x   3) openmpi/4.1.7-762w
  Where:
   H:  Hidden Module
```

To unload all modules at once, use the command:

```bash
module purge
```

*This quickbyte was validated on 6/22/2026*

## Video walkthrough

**Environment Modules** — from the [CARC video tutorials](../training/videos.md):

<iframe class="carc-video" src="https://www.youtube-nocookie.com/embed/tz-w3vc7cGQ" title="Environment Modules" loading="lazy" allow="accelerometer; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen></iframe>

<p class="carc-provenance" markdown>Migrated from [UNM-CARC QuickBytes](https://github.com/UNM-CARC/QuickBytes/blob/master/module_management.md){target=_blank} (last source update 2026-07-07). Spotted a problem? [Open an issue or pull request](https://github.com/UNM-CARC/QuickBytes){target=_blank}.</p>

---8<--- https://unm-carc.github.io/docs/running-jobs/pbs-to-slurm/

---
title: "PBS to Slurm migration"
description: "Translate PBS/Torque commands and scripts to their Slurm equivalents."
type: Reference
tags:
  - Slurm
  - PBS
  - Legacy
generated:
  by: "claude/fable-5"
  at: "2026-08-29T00:00:00Z"
sources:
  - id: quickbytes
    resource: "https://github.com/UNM-CARC/QuickBytes/blob/master/pbs2slurm.md"
    title: "UNM-CARC QuickBytes: pbs2slurm.md"
    author: "team:unm-carc"
    last_modified: "2026-06-25T10:51:21-06:00"
---

# PBS to Slurm migration

Most CARC systems historically supported PBS/TORQUE for scheduling jobs in HPC environments. However, current CARC systems primarily use Slurm (Simple Linux Utility for Resource Management) for job scheduling.

> **Note:** PBS is **not supported on Easley**. Although PBS may still be available on Hopper, we recommend using **Slurm** for all new jobs and workflows to ensure compatibility across CARC systems and to align with current support and documentation.

Slurm differs from PBS in its syntax, commands for resource allocation, job submission and monitoring, and environment variables.

Detailed Slurm documentation is available here:
https://slurm.schedmd.com/documentation.html

To submit jobs on Slurm-based systems, you must submit a Slurm job script. If you already have a PBS script, converting it to Slurm is usually straightforward.

Additional references:

* PBS job submission: http://carc.unm.edu/user-support-2/using-carc-systems1/running-jobs/submitting-jobs.html
* Slurm QuickBytes: https://github.com/UNM-CARC/QuickBytes/blob/master/Intro_to_slurm.md

---

## Converting PBS Commands to Slurm Commands

The table below lists commonly used PBS commands and their Slurm equivalents.

| PBS Command             | Slurm Command                | Description                                     |
| ----------------------- | ----------------------------- | ----------------------------------------------- |
| `qsub <job_script.pbs>` | `sbatch <job_script.slurm>`  | Submit a batch job                              |
| `qsub -I <options>`     | `salloc <options>`           | Request an interactive job                      |
| `qstat -u <user>`       | `squeue -u <user>`           | Display jobs submitted by a user                |
| `qstat -f <job-id>`     | `scontrol show job <job-id>` | Show detailed information for a job             |
| `qdel <job-id>`         | `scancel <job-id>`           | Cancel a job                                    |
| `pbsnodes <options>`    | `sinfo`                      | Display available nodes and cluster information |

---

## Resource Allocation Directives

Both PBS and Slurm scripts begin with a shell interpreter declaration.

Use:

```bash
#!/bin/bash
```

Resource directives are prefixed with:

* `#PBS` for PBS
* `#SBATCH` for Slurm

Common resource allocation options are shown below.

| PBS Directive            | Slurm Directive                               | Description                               |
| ------------------------ | --------------------------------------------- | ----------------------------------------- |
| `-N <name>`              | `--job-name=<name>`                           | Job name                                  |
| `-l procs=<N>`           | `--ntasks=<N>`                                | Number of tasks/processes                 |
| `-l nodes=a:ppn=b`       | `--nodes=a` + `--ntasks-per-node=b`           | Request `a` nodes with `b` tasks per node |
| `-l walltime=<HH:MM:SS>` | `--time=<HH:MM:SS>`                           | Maximum wall-clock runtime                |
| `-l mem=<memory>`        | `--mem=<memory>`                              | Memory requested per node                 |
| `-M <email>`             | `--mail-user=<email>`                         | Email address for notifications           |
| `-m <a,b,e>`             | `--mail-type=BEGIN,END,FAIL,REQUEUE,ALL`      | Email notification conditions             |
| `-o <out_file>`          | `--output=<out_file>`                         | Standard output file                      |
| `-e <error_file>`        | `--error=<error_file>`                        | Standard error file                       |
| `-j oe`                  | Default behavior in many Slurm configurations | Combine stdout and stderr                 |

> **Recommendation:** Prefer `--nodes` and `--ntasks-per-node` instead of collapsing everything into `--ntasks`, since this maps more directly to how resources are allocated in Slurm.

---

## Running Commands with `srun`

In Slurm, program execution lines within a batch script should generally be prefixed with `srun`. This ensures the command is properly launched on the allocated compute resources (rather than just on the node that happens to execute the script), and lets Slurm track and account for the resources that command actually uses. For single-task jobs the difference may not be obvious, but for multi-task or multi-node jobs, omitting `srun` can cause your program to run incorrectly or only on a single task/node instead of being distributed as requested.

```bash
srun python test.py
```

---

## Environment Variables

PBS and Slurm expose similar environment variables during job execution.

| PBS Variable        | Slurm Variable        | Description                                |
| ------------------- | ---------------------- | ------------------------------------------ |
| `$PBS_O_HOST`       | `$SLURM_SUBMIT_HOST`  | Host where the job was submitted           |
| `$PBS_JOBID`        | `$SLURM_JOB_ID`       | Job ID                                     |
| `$PBS_O_WORKDIR`    | `$SLURM_SUBMIT_DIR`   | Directory from which the job was submitted |
| `cat $PBS_NODEFILE` | `$SLURM_JOB_NODELIST` | Allocated nodes                            |

If you need individual node names in Slurm:

```bash
scontrol show hostnames $SLURM_JOB_NODELIST
```

---

## Example: PBS Script

Below is a sample PBS script that runs `test.py`.

```bash
#!/bin/bash

#PBS -l nodes=1:ppn=1
#PBS -l walltime=01:00:00
#PBS -N test
#PBS -o test.out
#PBS -e test.err
#PBS -m bae
#PBS -M user@unm.edu

cd "$PBS_O_WORKDIR"

python test.py
```

---

## Equivalent Slurm Script

The equivalent Slurm script is:

```bash
#!/bin/bash

#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
#SBATCH --time=00:05:00
#SBATCH --job-name=test
#SBATCH --output=test.out
#SBATCH --error=test.err

cd "$SLURM_SUBMIT_DIR"

module load miniconda3

srun python test.py
```

Submit the job with:

```bash
sbatch job_script.slurm
```

*This quickbyte was validated on 6/25/2026*

<p class="carc-provenance" markdown>Migrated from [UNM-CARC QuickBytes](https://github.com/UNM-CARC/QuickBytes/blob/master/pbs2slurm.md){target=_blank} (last source update 2026-06-25). Spotted a problem? [Open an issue or pull request](https://github.com/UNM-CARC/QuickBytes){target=_blank}.</p>

---8<--- https://unm-carc.github.io/docs/running-jobs/slurm-accounting/

---
title: "Slurm accounting and fairshare"
description: "How Slurm accounts, job accounting, and the fairshare system work at CARC."
type: Guide
tags:
  - Slurm
  - Allocations
generated:
  by: "claude/fable-5"
  at: "2026-08-29T00:00:00Z"
sources:
  - id: quickbytes
    resource: "https://github.com/UNM-CARC/QuickBytes/blob/master/slurm_accounting.md"
    title: "UNM-CARC QuickBytes: slurm_accounting.md"
    author: "team:unm-carc"
    last_modified: "2026-06-24T15:57:23-06:00"
---

# Slurm accounting and fairshare

## What Is Slurm Accounting?

Slurm accounting allows CARC to track resource usage and use that information to make scheduling decisions.

When a job runs, the resources consumed by that job (such as CPUs and memory) are tracked and associated with a Slurm account. At CARC, accounts correspond to CARC projects.

When a cluster is under heavy utilization, projects with lower historical resource usage may receive higher scheduling priority than projects with higher historical resource usage.

---

## Accounting Commands

You can view the accounts you belong to using the `myaccounts` command.

Example:

```bash
[yourUsername@easley ~]$ myaccounts
```
```bash
User            Account
yourUsername    systems
yourUsername    yourIdNumber
```

The account name corresponds to the project ID.

To view additional information about an account, use:

```bash
sacctmgr show account <account_name>
```

Example:

```bash
[yourUsername@easley ~]$ sacctmgr show account yourIdNumber
```
```bash
Account          Descr                  Org
yourIdNumber     hpc@unm sys admin      download
```

The **Org** column displays the username of the PI associated with the project.

---

## Choosing an Account for a Job

There are three ways a job can be associated with an account. Slurm checks them in the following order:

1. `--account=<account_name>` specified during job submission
2. `~/.default_slurm_account`
3. Your default account (typically the most recently assigned project)

Once a valid account is found, Slurm stops checking subsequent options.

---

### 1. `--account`

The `--account` option can be specified either:

* Directly in an `srun` command, or
* In an `sbatch` submission script using the `#SBATCH` directive

Examples:

```bash
srun --account=yourIdNumber my_program
```

or

```bash
#SBATCH --account=yourIdNumber
```

This option takes precedence over all other methods of account selection.

---

### 2. `~/.default_slurm_account`

If a file named `.default_slurm_account` exists in your home directory and contains a valid account name, Slurm will use that account whenever `--account` is not specified.

Example:

```bash
echo "yourIdNumber" > ~/.default_slurm_account
```

You can view the current default account with:

```bash
cat ~/.default_slurm_account
```

---

### 3. Default Account

If you do not specify `--account` and there is no valid `~/.default_slurm_account`, Slurm will charge usage to your configured default account.

You can view your current default account with:

```bash
sacctmgr show user <username>
```

Example:

```bash
[yourUsername@easley ~]$ sacctmgr show user yourUsername
```
```bash
User            Def Acct       Admin
yourUsername    yourIdNumber   None
```

You can then inspect that account with:

```bash
sacctmgr show account yourIdNumber
```

---

*This QuickByte was validated on June 22, 2026.*

<p class="carc-provenance" markdown>Migrated from [UNM-CARC QuickBytes](https://github.com/UNM-CARC/QuickBytes/blob/master/slurm_accounting.md){target=_blank} (last source update 2026-06-24). Spotted a problem? [Open an issue or pull request](https://github.com/UNM-CARC/QuickBytes){target=_blank}.</p>

---8<--- https://unm-carc.github.io/docs/running-jobs/slurm-intro/

---
title: "Introduction to Slurm"
description: "Slurm basics on CARC clusters: partitions, interactive jobs, and your first batch script."
type: Guide
tags:
  - Slurm
  - Jobs
  - New users
generated:
  by: "claude/fable-5"
  at: "2026-08-29T00:00:00Z"
sources:
  - id: quickbytes
    resource: "https://github.com/UNM-CARC/QuickBytes/blob/master/Intro_to_slurm.md"
    title: "UNM-CARC QuickBytes: Intro_to_slurm.md"
    author: "team:unm-carc"
    last_modified: "2026-07-06T13:37:22-06:00"
---

# Introduction to Slurm

## Getting Started

For a general introduction to Slurm — including an overview of its architecture, key commands (`sinfo`, `squeue`, `srun`, `sbatch`, `scancel`, `scontrol`), and basic job submission examples — refer to the **[official Slurm Quick Start User Guide](https://slurm.schedmd.com/quickstart.html){target=_blank}**. The guide is well-maintained and covers the core concepts you need to get up and running.

The sections below supplement that guide with information specific to CARC Easley at the University of New Mexico.

---

## CARC Easley Partitions

When you run `sinfo` on Easley, you will see output similar to the following:

```bash
[username@easley ~]$ sinfo
```

```
PARTITION   AVAIL  TIMELIMIT  NODES  STATE NODELIST
general*       up 2-00:00:00      1   comp easley002
general*       up 2-00:00:00      4   mix- easley[008,015,018,044]
general*       up 2-00:00:00      8    mix easley[003-004,012-014,016,020,023]
general*       up 2-00:00:00     34  alloc easley[001,005-007,009-011,017,019,021-022,024-043,046-048]
general*       up 2-00:00:00      1   down easley045
bigmem         up 2-00:00:00      1    mix easley050
bigmem         up 2-00:00:00      1  alloc easley049
h100           up 2-00:00:00      1   mix- easley051
h100           up 2-00:00:00      1   plnd easley054
h100           up 2-00:00:00      2  alloc easley[052-053]
l40s           up 2-00:00:00      1    mix easley055
l40s           up 2-00:00:00      2  alloc easley[056-057]
interactive    up    4:00:00      1   comp easley002
interactive    up    4:00:00      5   mix- easley[008,015,018,044,051]
interactive    up    4:00:00      1   plnd easley054
interactive    up    4:00:00     10    mix easley[003-004,012-014,016,020,023,050,055]
interactive    up    4:00:00     39  alloc easley[001,005-007,009-011,017,019,021-022,024-043,046-049,052-053,056-057]
interactive    up    4:00:00      1   down easley045
debug          up    1:00:00      1   comp easley002
debug          up    1:00:00      5   mix- easley[008,015,018,044,051]
debug          up    1:00:00      1   plnd easley054
debug          up    1:00:00     10    mix easley[003-004,012-014,016,020,023,050,055]
debug          up    1:00:00     39  alloc easley[001,005-007,009-011,017,019,021-022,024-043,046-049,052-053,056-057]
debug          up    1:00:00      1   down easley045
scavenger      up 2-00:00:00      1   comp easley002
scavenger      up 2-00:00:00      9   mix- easley[008,015,018,044,051,060-063]
scavenger      up 2-00:00:00      1   plnd easley054
scavenger      up 2-00:00:00     10    mix easley[003-004,012-014,016,020,023,050,055]
scavenger      up 2-00:00:00     41  alloc easley[001,005-007,009-011,017,019,021-022,024-043,046-049,052-053,056-059]
scavenger      up 2-00:00:00      1   down easley045
```

Key partitions you may have access to:

- **general** — The default community partition. Maximum wall time of 2 days. Use this if you are not a member of a specific condo group.
- **debug** — Short jobs only (4-hour limit). Useful for testing scripts before submitting long runs.
- **condo** — Purchased nodes available to specific research groups. If you are a member of a condo group, you likely already know your partition name. Check with your PI if you are unsure.
- **scavenger** - Whenever a purchased/reserved node is not in use, this partition grabs them and allows them to be used by the public, but be warned you will be kicked off if the owner begins a job on it.

To see detailed node information including CPU count, memory, and disk:

```bash
sinfo -N -l
```

If you omit `--partition` (or `-p`) from your job submission, your job will be submitted to the `general` partition by default.

---

## Useful `squeue` Flags

The official guide covers `squeue` basics. A few flags that are especially handy on a shared cluster:

```bash
squeue --me              # Show only your jobs
squeue -p general        # Show only jobs in the general partition
squeue -u <username>     # Show jobs for a specific user
```

---

## Canceling Jobs

```bash
scancel <JOBID>          # Cancel a specific job
scancel --me             # Cancel all of your jobs
```

---

## Notes on Resource Requests

- The more constraints you add to a job (e.g., requiring all tasks on the same node with `--ntasks-per-node`), the longer your queue time may be. Requesting resources spread across nodes often results in faster scheduling.
- Memory is specified per CPU with `--mem-per-cpu` (in MB) or for the whole job with `--mem`.
- Time limits use the format `D-HH:MM:SS` (e.g., `1-12:00:00` for 1 day and 12 hours) or `MM:SS` / `HH:MM:SS` for shorter jobs.

---

## Additional Resources

- [Official Slurm Quick Start Guide](https://slurm.schedmd.com/quickstart.html){target=_blank}
- [CARC at UNM Documentation](https://carc.unm.edu){target=_blank}
- For help, contact the CARC support team or visit the CARC user portal.


*This quickbyte was validated on 6/22/2026.*

## Video walkthrough

**Slurm Job Scheduler** — from the [CARC video tutorials](../training/videos.md):

<iframe class="carc-video" src="https://www.youtube-nocookie.com/embed/cIRyG8C3CVk" title="Slurm Job Scheduler" loading="lazy" allow="accelerometer; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen></iframe>

<p class="carc-provenance" markdown>Migrated from [UNM-CARC QuickBytes](https://github.com/UNM-CARC/QuickBytes/blob/master/Intro_to_slurm.md){target=_blank} (last source update 2026-07-06). Spotted a problem? [Open an issue or pull request](https://github.com/UNM-CARC/QuickBytes){target=_blank}.</p>

---8<--- https://unm-carc.github.io/docs/running-jobs/slurm-reference/

---
title: "Slurm command reference"
description: "Common Slurm commands and sbatch directives with examples."
type: Reference
tags:
  - Slurm
  - Jobs
generated:
  by: "claude/fable-5"
  at: "2026-08-29T00:00:00Z"
sources:
  - id: quickbytes
    resource: "https://github.com/UNM-CARC/QuickBytes/blob/master/slurm-sbatch.md"
    title: "UNM-CARC QuickBytes: slurm-sbatch.md"
    author: "team:unm-carc"
    last_modified: "2026-06-22T10:00:39-06:00"
---

# Slurm command reference

__Simple Linux Utility for Resource Management (Slurm)__ is a computer software which performs job (a unit of work or unit of execution) computational resource allocation in an HPC center. Slurm is widely used in HPC centers and clusters across the world, including at CARC, where it's the job scheduler and resource manager we use.

Jobs can be run either interactively or as a submitted Slurm batch script that is run non-interactively and subsequently controlled through Slurm. In both cases resources are requested and jobs submitted through Slurm, which then places your request into a queue.

At CARC, all batch jobs are submitted through the machine's head node via the Slurm resource manager and scheduler.

### Slurm Batch Scripts
To submit jobs at CARC you submit a Slurm batch script to the Slurm resource manager. This Slurm script starts by telling Slurm what kind of resources you are requesting for your job. These lines in your script start with `#SBATCH` followed by flags that specify things like wall time, nodes, and processors requested. To get a complete list of options available type `man sbatch` from the command prompt when logged in to a CARC machine.

After your Slurm instructions, you then load your software modules (refer to the help page 'Managing software modules' for more information) followed by software specific instructions. All Slurm batch scripts take this same basic structure for job submission. For some example scripts refer to the help page 'Example Slurm Scripts' to help you get started with computing at CARC.

*This QuickByte was validated on 6/22/2026*

<p class="carc-provenance" markdown>Migrated from [UNM-CARC QuickBytes](https://github.com/UNM-CARC/QuickBytes/blob/master/slurm-sbatch.md){target=_blank} (last source update 2026-06-22). Spotted a problem? [Open an issue or pull request](https://github.com/UNM-CARC/QuickBytes){target=_blank}.</p>

---8<--- https://unm-carc.github.io/docs/running-jobs/submitting-jobs/

---
title: "Submitting jobs"
description: "Submit, monitor, and cancel batch and interactive jobs with Slurm."
type: Guide
tags:
  - Slurm
  - Jobs
generated:
  by: "claude/fable-5"
  at: "2026-08-29T00:00:00Z"
sources:
  - id: quickbytes
    resource: "https://github.com/UNM-CARC/QuickBytes/blob/master/submitting_jobs.md"
    title: "UNM-CARC QuickBytes: submitting_jobs.md"
    author: "team:unm-carc"
    last_modified: "2025-12-03T11:52:10-07:00"
---

# Submitting jobs

There are two ways you can run your jobs, namely submitting a Slurm script and running a job interactively. Either way, jobs are submitted to CARC by the command `sbatch`. For more information on available options type `man sbatch`
### Submitting the Slurm Script to the Batch Scheduler
In order to run our simple Slurm script, we will need to submit it to the batch scheduler using the command `sbatch` followed by the name of the script we would like to run. For more information please see our page on writing a [Slurm batch script](https://github.com/UNM-CARC/QuickBytes/blob/master/pbs_scripts2.md){target=_blank}.
In the following example, we submit our simple `hello.sbatch` script to the batch scheduler using `sbatch`. Note that it returns the job identifier when the job is successfully submitted. You can use this job identifier to query the status of your job from your shell.  
For example:

```bash
sbatch hello.sbatch
Submitted batch job 156452
```
### Interactive Slurm Jobs
Normally a job is submitted for execution on a cluster or supercomputer using the command `sbatch script.sbatch`. CARC recommends that all jobs are submitted this way as job submission fails if there are errors in resources requested. However, at times, such as when debugging, it can be useful to run a job interactively. To run a job in this way type `salloc` followed by resources requested, and the batch manager will log you into a node where you can directly run your code.  
For example, here is the output from an interactive session running our simple `helloworld_paralell.sbatch` script:

```bash
salloc --nodes=1 --ntasks=8 --time=00:05:00

salloc: Granted job allocation 156469
salloc: Nodes easley004 are ready for job

module load openmpi

bash  helloworld_parallel.sbatch
Job 156469 running on easley004
Hello World from host easley004
Hello World from host easley004
Hello World from host easley004
Hello World from host easley004
Hello World from host easley004
Hello World from host easley004
Hello World from host easley004
Hello World from host easley004
```
Three commands were executed here. The first,

```bash
salloc --nodes=1 --ntasks=8 --time=00:05:00
```
asked the batch manager to provide one node of easley with all 8 of that node’s cores for use. It is good practice to request all available processors on a node to avoid multiple users being assigned to the same node. The walltime was specified as 5 minutes, since this was a simple code that would execute quickly. The second command, 

```bash
module load openmpi
```
loaded the openMPI software module to parallelize our script across all 8 processors; this ensures that the necessary MPI libraries would be available during execution. The third command, 

```bash
bash  helloworld_parallel.sbatch
```
ran the commands found within our `helloworld_parallel.sbatch` script.

<p class="carc-provenance" markdown>Migrated from [UNM-CARC QuickBytes](https://github.com/UNM-CARC/QuickBytes/blob/master/submitting_jobs.md){target=_blank} (last source update 2025-12-03). Spotted a problem? [Open an issue or pull request](https://github.com/UNM-CARC/QuickBytes){target=_blank}.</p>

---8<--- https://unm-carc.github.io/docs/interactive/jupyterhub/

---
title: "JupyterHub"
description: "Run Jupyter notebooks on Hopper and Easley compute nodes through CARC JupyterHub."
type: Guide
tags:
  - Interactive
  - Jupyter
  - New users
generated:
  by: "claude/fable-5"
  at: "2026-08-29T00:00:00Z"
sources:
  - id: carc-web
    resource: "https://carc.unm.edu/index.html"
    title: "CARC website (carc.unm.edu)"
    author: "team:unm-carc"
---

# JupyterHub

CARC runs JupyterHub portals that launch Jupyter notebook servers **on
cluster compute nodes**, so your notebooks have direct access to CARC
storage, modules, and hardware.

| Cluster | JupyterHub URL |
| ------- | -------------- |
| Hopper | [hopper.alliance.unm.edu](https://hopper.alliance.unm.edu){ target=_blank } |
| Easley | [easley.alliance.unm.edu/jupyter](https://easley.alliance.unm.edu/jupyter){ target=_blank } |

Log in with your CARC username and password. New here? Start with
[Getting started at CARC](../getting-started/overview.md).

## Starting a server

After login, JupyterHub asks for your session options (Slurm account,
resources, and duration — the exact form depends on the cluster). Your
notebook server is a [Slurm job](../running-jobs/slurm-intro.md) under the
hood, so:

* it may queue briefly until resources are available;
* it stops when its time limit is reached — save your notebooks;
* the resources you hold count against your project's
  [fairshare](../running-jobs/slurm-accounting.md), so stop your server
  (*File → Hub Control Panel → Stop My Server*) when you are done.

## Using your own environments as kernels

The default kernels cover common cases, but most research needs its own
packages. Create a [conda environment](../software/conda-environments.md) and
register it as a Jupyter kernel — the walkthrough is in
[Conda environments in JupyterHub](../software/conda-jupyterhub.md).

## Scaling beyond one node

Notebooks don't have to stay single-threaded:

* [Parallel Python with Dask and scikit-learn](../software/dask-scikit-learn.md) -
  scale scikit-learn across cluster workers from a notebook.
* [MPI parallelization from JupyterHub](../software/jupyterhub-mpi.md) - drive
  mpi4py/ipyparallel from a notebook session.
* For long or heavy computations, move to a
  [batch job](../running-jobs/example-slurm-scripts.md) — notebooks are for
  exploration, batch is for production runs.

## Related pages

* [Open OnDemand](open-ondemand.md) - browser access to files, shells, and other interactive apps.
* [Installing deep learning packages](../software/deep-learning-packages.md) - GPU-ready PyTorch/TensorFlow environments.
* [Getting help](../support/help.md) - stuck server or missing kernel? Open a ticket.

---8<--- https://unm-carc.github.io/docs/interactive/open-ondemand/

---
title: "Open OnDemand"
description: "Use CARC clusters from your browser: files, shells, job management, and interactive apps."
type: Guide
tags:
  - Interactive
  - Open OnDemand
  - New users
generated:
  by: "claude/fable-5"
  at: "2026-08-29T00:00:00Z"
sources:
  - id: carc-web
    resource: "https://carc.unm.edu/index.html"
    title: "CARC website (carc.unm.edu)"
    author: "team:unm-carc"
  - id: ood-docs
    resource: "https://osc.github.io/ood-documentation/latest/"
    title: "Open OnDemand documentation (OSC)"
    author: "team:osc"
---

# Open OnDemand

[Open OnDemand](https://openondemand.org){ target=_blank } (OOD) gives you a
complete web interface to CARC clusters — no SSH client, no X11 setup, just a
browser. It is the easiest way to work on CARC systems if you are new to HPC
or away from your usual machine.

[:material-monitor-dashboard: Open the CARC OnDemand portal](https://ood.alliance.unm.edu){ .md-button .md-button--primary target=_blank }

Log in with your CARC username and password. If you don't have an account yet,
start with [Getting started at CARC](../getting-started/overview.md).

## What you can do in OnDemand

<div class="grid cards" markdown>

-   :material-folder-open:{ .lg .middle } __Files__

    ---

    Browse your home, project, and scratch spaces; upload and download files;
    edit text files in the browser. For large transfers, prefer the
    [command-line tools](../getting-started/transferring-data.md).

-   :material-console:{ .lg .middle } __Cluster shell__

    ---

    Open a login-node terminal in a browser tab — everything you can do over
    SSH, without an SSH client.

-   :material-tray-full:{ .lg .middle } __Jobs__

    ---

    Compose, submit, and monitor [Slurm jobs](../running-jobs/slurm-intro.md)
    from a form-based interface, and inspect their output files when they
    finish.

-   :material-application-brackets:{ .lg .middle } __Interactive apps__

    ---

    Launch graphical and notebook sessions that run on compute nodes. The
    available apps depend on the cluster — check the *Interactive Apps* menu
    in the portal for the current list.

</div>

## How interactive apps work

When you launch an interactive app, OnDemand submits a Slurm job on your
behalf. That means:

* You choose the resources (cores, memory, GPUs, walltime) in the launch
  form — the same considerations as any [batch job](../running-jobs/submitting-jobs.md)
  apply, and the [Good Neighbor Use Policy](../getting-started/good-neighbor-policy.md)
  asks you not to leave hardware idle.
* Your session may wait in the queue until resources are free, just like any
  other job. Small, short requests start faster.
* The session ends when its walltime expires — save your work.
* Usage counts against your project's
  [fairshare](../running-jobs/slurm-accounting.md) like any other job.

!!! tip "When to use OnDemand vs. SSH"

    OnDemand is ideal for file management, quick edits, monitoring jobs, and
    interactive sessions. For long-running automated workflows, scripted
    submission, and bulk data movement, the [command line](../getting-started/logging-in.md)
    remains the more powerful tool.

## Related pages

* [JupyterHub](jupyterhub.md) - a dedicated notebook portal on Hopper and Easley.
* [Introduction to Slurm](../running-jobs/slurm-intro.md) - understand what OnDemand submits for you.
* [Getting help](../support/help.md) - if the portal misbehaves, open a ticket.

---8<--- https://unm-carc.github.io/docs/software/alphafold/

---
title: "AlphaFold"
description: "Run AlphaFold protein structure prediction on CARC systems."
type: Guide
tags:
  - Bioinformatics
  - GPU
  - Machine learning
generated:
  by: "claude/fable-5"
  at: "2026-08-29T00:00:00Z"
sources:
  - id: quickbytes
    resource: "https://github.com/UNM-CARC/QuickBytes/blob/master/alphafold.md"
    title: "UNM-CARC QuickBytes: alphafold.md"
    author: "team:unm-carc"
    last_modified: "2024-07-23T07:24:04-06:00"
---

# AlphaFold

Alphafold predicts the 3D structure of proteins from their amino acid sequence. A deep learning system that uses a combination of sequence alignment, evolutionary information, and physical principles to generate its predictions.
Primarily written in python, the first version of alphafold was released in 2016, and has been updated as recently as 2022. There are two ways to run alphafold here at CARC. Option 1 is to use [localcolabfold.](https://github.com/YoshitakaMo/localcolabfold){target=_blank} You may find that Localcolabfold is easier to get running, however it will come with tradeoffs in certain areas, For Example, localcolabfold uses the pdb70 database, and there is not a great way to choose a different database to use. If you are unsure which version is best for you, we recommend you review the readme & issues for localcolabfold to determine if there are any features you may need. 

# Alphafold w/ LocalColabFold:

Localcolabfold is set up as part of our module system at CARC. As with anything as part of the module system, you can use `module spider localcolabfold` to get more information about the module. This will also list other dependencies, if any, such as a different compiler. 

localColabFold runs with the command `colabfold_batch`. The only other thing you will need to provide is the input fasta file. Below is an example of a slurm script to run using localcolabfold on Hopper. 


     #!/bin/bash
     #SBATCH --job-name alphafold
     #SBATCH --mail-user=<your_email>
     #SBATCH --partition general
     #SBATCH -N 1
     #SBATCH --ntasks 1
     #SBATCH --cpus-per-task 64
     #SBATCH --time 1:00:00
     
     module load localcolabfold
     
     INPUT_FILE_PATH=/path/to/input.fasta
     now=$(date +"%m_%d_%H_%M_%S")
     OUTPUTDIR=$SLURM_SUBMIT_DIR/$SLURM_JOB_NAME-$now
     
     mkdir -p $OUTPUTDIR
     
     cd $SLURM_SUBMIT_DIR
     
     srun colabfold_batch $INPUT_FILE_PATH $OUTPUTDIR

Reference the [localcolabfold documentation](https://github.com/YoshitakaMo/localcolabfold?tab=readme-ov-file#flags){target=_blank} for flags you may find useful.


# Alphafold w/ Singularity Image:

## Choose your alphafold version ## 
There are multiple versions of alphafold installed using singularity images. You can view each of the versions installed with the command:

     ls /projects/shared/singularity/alphafold*

 and hit tab. You'll see below we currently have version 2.0 and 2.3.1 installed. For this tutorial we will be using version 2.0. 
For this tutorial we will be using version 2.0. 


Now we can create a new directory with

    mkdir alphafold
    
and move into that directory with

    cd alphafold

## Running Alphafold ##
Inside the alphafold directory, you will be able to run the program using the slurm script, this script will differ based on the machine you are using. GPU nodes are available on the current CARC clusters — see the [systems overview](../systems/overview.md). 

We will use the Hopper script below. Create a new file using your favorite editor. For example, 

    vim alphafold.sh
    
then hit `i` to go into insert mode, and past the contents from the below script into this file. You can then add your email to get alerts about the run. When you are finished editing this file, type `ESC` to exit insert mode, followed by `:wq` to write & quite the file, this will save your changes. 

### Hopper Script ###

    #!/bin/bash
    #SBATCH --job-name alphafold
    #SBATCH --time=08:00:00
    #SBATCH --ntasks=1
    #SBATCH --cpus-per-task=32
    #SBATCH --mem=0G
    #SBATCH --partition=general
    #SBATCH --output alphafold.out
    #SBATCH --error alphafold.err

    #SBATCH --mail-user < your email > 
    #SBATCH --mail-type all
    
    module load singularity
    
    # Specify input/output paths
    SINGULARITY_IMAGE_PATH=/projects/shared/singularity/
    ALPHAFOLD_DATA_PATH=/carc/scratch/shared/alphafold/data/70
    ALPHAFOLD_MODELS=$ALPHAFOLD_DATA_PATH/params
    ALPHAFOLD_INPUT_FASTA=$SLURM_SUBMIT_DIR/input_test.fasta
    NOW=$(date +"%Y_%m_%d_%H_%M_%S")
    ALPHAFOLD_OUTPUT_DIR=$SLURM_SUBMIT_DIR/alphafold_output-$NOW
    
    mkdir -p $ALPHAFOLD_OUTPUT_DIR
    
    #Run the command
    singularity run --nv \
     --bind $ALPHAFOLD_DATA_PATH:/data \
     --bind $ALPHAFOLD_MODELS \
     --bind $ALPHAFOLD_OUTPUT_DIR:/alphafold_output \
     --bind $ALPHAFOLD_INPUT_FASTA:/input.fasta \
     --bind .:/etc \
     --pwd  /app/alphafold $SINGULARITY_IMAGE_PATH/alphafold-2.0.sif \
     --fasta_paths=/input.fasta  \
     --uniref90_database_path=/data/uniref90/uniref90.fasta  \
     --data_dir=/data \
     --mgnify_database_path=/data/mgnify/mgy_clusters.fa   \
     --bfd_database_path=/data/bfd/bfd_metaclust_clu_complete_id30_c90_final_seq.sorted_opt \
     --uniclust30_database_path=/data/uniclust30/uniclust30_2018_08/uniclust30_2018_08 \
     --pdb70_database_path=/data/pdb70/pdb70  \
     --template_mmcif_dir=/data/pdb_mmcif/mmcif_files  \
     --obsolete_pdbs_path=/data/pdb_mmcif/obsolete.dat \
     --max_template_date=2020-05-14   \
     --output_dir=/alphafold_output  \
     --model_names='model_1' \
     --preset=casp14

#### Input File ####
These scripts expect you to have a file named `input_test.fasta` where you will give your input sequence. This should be in the format:

(alphafold/input_test.fasta)

    > 350 residue example sequence   
    MTANHLESPNCDWKNNRMAIVHMVNVTPLRMMEEPRAAVEAAFEGIMEPAVVGDMVEYWNKMISTCCNYYQMGSSRSHLEEKAQMVDRFWFCPCIYYASGKWRNMFLNILHVWGHHHYPRNDLKPCSYLSCKLPDLRIFFNHMQTCCHFVTLLFLTEWPTYMIYNSVDLCPMTIPRRNTCRTMTEVSSWCEPAIPEWWQATVKGGWMSTHTKFCWYPVLDPHHEYAESKMDTYGQCKKGGMVRCYKHKQQVWGNNHNESKAPCDDQPTYLCPPGEVYKGDHISKREAENMTNAWLGEDTHNFMEIMHCTAKMASTHFGSTTIYWAWGGHVRPAATWRVYPMIQEGSHCQC

#### localtime ####
Your job will fail within the first few moments if the input file is not formatted properly. It will also require you to have a `localtime` file in the directory in which you are running, which you can create with 

    touch localtime
    
This file does not directly impact the simulation in any way, but is used to track the time to make your tests reproducible. If this file is empty, it will default to using the current time in UTC, but you could also place the correct time in:

(alphafold/localtime)

    2022-10-28T16:15:29
    
## Run ##
You can now run your alphafold sequence with the command 

    sbatch alphafold.sh 

This will hand your script you made above to the slurm scheduler. If you added your email to the script, you will receive an email that it has been added to the queue, once it starts, and once it ends. It will also email if it fails before a successful completion. If your run fails, more information can be found in the `alphafold.out` & `alphafold.err` files which will be generated as each run begins.

You can check if your run is still running with 

    squeue --me 

This will list all jobs you currently have both queued & running. On the general partitions your time will be [limited to between 4 and 48 hours](../systems/resource-limits.md) of runtime.


## Output ##
After a successful job, you will notice multiple output files. They will be placed in the `./alphafold_output-<Timestamp>/input/*` Where you will have the resulting .pdb, .pkl, and .json files. 
    
Now that it is on your local computer, you can now view this file on your computer if you have the proper software to view a pdb file.

<p class="carc-provenance" markdown>Migrated from [UNM-CARC QuickBytes](https://github.com/UNM-CARC/QuickBytes/blob/master/alphafold.md){target=_blank} (last source update 2024-07-23). Spotted a problem? [Open an issue or pull request](https://github.com/UNM-CARC/QuickBytes){target=_blank}.</p>

---8<--- https://unm-carc.github.io/docs/software/conda-channels-pip/

---
title: "Conda channels and pip"
description: "Use conda channels (conda-forge, bioconda) and mix pip installs safely inside environments."
type: Guide
tags:
  - Python
  - Conda
generated:
  by: "claude/fable-5"
  at: "2026-08-29T00:00:00Z"
sources:
  - id: quickbytes
    resource: "https://github.com/UNM-CARC/QuickBytes/blob/master/anaconda_pip_channels.md"
    title: "UNM-CARC QuickBytes: anaconda_pip_channels.md"
    author: "team:unm-carc"
    last_modified: "2020-01-28T10:52:58-07:00"
---

# Conda channels and pip

#### Installing packages with pip

Not all versions of all software have Conda packages available however, especially for some python libraries. Pip, the python package manager, is automatically installed by default in all environments created by Conda, and can install packages alongside those installed by Conda without conflict.  
For example, say you need the library psutil, but you specifically need version 5.3.0. When you search for psutil using `conda` you get the following:

```bash
$ conda search psutil=5.3
Loading channels: done
# Name                  Version           Build  Channel
psutil                    5.3.1          py27_0  conda-forge
psutil                    5.3.1  py27h4c169b4_0  pkgs/main
psutil                    5.3.1          py35_0  conda-forge
psutil                    5.3.1  py35h6e9e629_0  pkgs/main
psutil                    5.3.1          py36_0  conda-forge
psutil                    5.3.1  py36h0e357b8_0  pkgs/main
```

Unfortunately there are no packages built for psutil version 5.3.0. We can use pip to install the version we want however.

```bash
$ source activate py-2.7

(py-2.7)$ pip install psutil==5.3.0
Collecting psutil==5.3.0
  Downloading https://files.pythonhosted.org/packages/1c/da/555e3ad3cad30f30bcf0d539cdeae5c8e7ef9e2a6078af645c70aa81e418/psutil-5.3.0.tar.gz (397kB)
    100% |████████████████████████████████| 399kB 1.3MB/s
Building wheels for collected packages: psutil
  Running setup.py bdist_wheel for psutil ... done
  Stored in directory: /users/yourusername/.cache/pip/wheels/ff/c5/4f/1ee2208203f1cfeda16e91fccd8bfce5f4840b683671729d57
Successfully built psutil

(py-2.7)$ conda list

# packages in environment at /users/yourusername/.Conda/envs/py-2.7:
#
# Name                    Version                   Build
ca-certificates           2018.03.07                    0
certifi                   2018.8.24                py27_1
libedit                   3.1.20170329         h6b74fdf_2
libffi                    3.2.1                hd88cf55_4
libgcc-ng                 8.2.0                hdf63c60_1
libstdcxx-ng              8.2.0                hdf63c60_1
ncurses                   6.1                  hf484d3e_0
openssl                   1.0.2p               h14c3975_0
pip                       10.0.1                   py27_0
**psutil                  5.3.0                     <pip>
python                    2.7.15               h1571d57_0
readline                  7.0                  h7b6447c_5
setuptools                40.2.0                   py27_0
sqlite                    3.24.0               h84994c4_0
tk                        8.6.8                hbc83047_0
wheel                     0.31.1                   py27_0
zlib                      1.2.11               ha838bed_2
```
When installing packages using `pip` it is important to first activate the Conda environment that you want to install the package in since pip is strictly a package manager and cannot modify Conda environments from outside that environment. You can see that our psutil package, marked with a double asterisk, is version 5.3.0, just like we wanted. Under the 'build' column however you will see that `conda` is not sure which build it is since it was installed with `pip`, as indicated by the `<pip>` designator.
#### Performance with Conda versus Pip
One thing to note when installing packages is that it is always preferable to first install necessary packages with `conda` first, only then use `pip` to install only those packages that were not available through Anaconda repositories. 
It is usually best practice to install needed packages and dependencies with `conda` and use `pip` to install any remaining packages that were not available instead of vice versa.

#### Adding package repositories (channels)

Sometimes the default repositories, or channels for Conda, do not have the package you are looking for, but that does not mean that it is necessarily unavailable entirely. Say you are working with some Illumina sequence data and need the Burrows-Wheeler Aligner (bwa) in your pipeline, so you activate your bioinformatics environment and type `conda install bwa` which prints the following:

```bash
Solving environment: failed

PackagesNotFoundError: The following packages are not available from current channels:

  - bwa

Current channels:

  - https://repo.anaconda.com/pkgs/main/linux-64
  - https://repo.anaconda.com/pkgs/main/noarch
  - https://repo.anaconda.com/pkgs/free/linux-64
  - https://repo.anaconda.com/pkgs/free/noarch
  - https://repo.anaconda.com/pkgs/r/linux-64
  - https://repo.anaconda.com/pkgs/r/noarch
  - https://repo.anaconda.com/pkgs/pro/linux-64
  - https://repo.anaconda.com/pkgs/pro/noarch

To search for alternate channels that may provide the conda package you're
looking for, navigate to

    https://anaconda.org

and use the search bar at the top of the page.
```
We can search other channels that may have the package we are interested in with the `-c` flag. For example, BioConda is a large repository that hosts several thousand bioinformatics packages. We can search for our bwa package by specifying that channel.

```bash
$ conda search -c bioConda bwa
```

Which yields better results:

```bash
Loading channels: done
# Name                  Version           Build  Channel
bwa                       0.5.9               0  bioConda
bwa                       0.5.9               1  bioConda
bwa                       0.6.2               0  bioConda
bwa                       0.6.2               1  bioConda
bwa                      0.7.3a               0  bioConda
bwa                      0.7.3a               1  bioConda
bwa                      0.7.3a      ha92aebf_2  bioConda
bwa                       0.7.4      ha92aebf_0  bioConda
bwa                       0.7.8               0  bioConda
bwa                       0.7.8               1  bioConda
bwa                       0.7.8      ha92aebf_2  bioConda
bwa                      0.7.12               0  bioConda
bwa                      0.7.12               1  bioConda
bwa                      0.7.13               0  bioConda
bwa                      0.7.13               1  bioConda
bwa                      0.7.15               0  bioConda
bwa                      0.7.15               1  bioConda
bwa                      0.7.16      pl5.22.0_0  bioConda
bwa                      0.7.17      ha92aebf_3  bioConda
bwa                      0.7.17      pl5.22.0_0  bioConda
bwa                      0.7.17      pl5.22.0_1  bioConda
bwa                      0.7.17      pl5.22.0_2  bioConda
```

We can then install our bwa package using `conda install -c bioConda bwa` and continue with our analyses. You can permanently add channels by appending your `.condarc` file either directly in a text editor, or with `conda` by using the `config` command:

```bash
$ conda config --append channels bioconda
```
This will permanently add the BioConda channel to your configuration file meaning Conda will automatically search BioConda as well as the default channels when looking for packages.

For more information on managing channels and installing with pip please refer to the Conda support documentation at this [link](https://conda.io/docs/user-guide/tasks/manage-channels.html){target=_blank}.

<p class="carc-provenance" markdown>Migrated from [UNM-CARC QuickBytes](https://github.com/UNM-CARC/QuickBytes/blob/master/anaconda_pip_channels.md){target=_blank} (last source update 2020-01-28). Spotted a problem? [Open an issue or pull request](https://github.com/UNM-CARC/QuickBytes){target=_blank}.</p>

---8<--- https://unm-carc.github.io/docs/software/conda-environments/

---
title: "Managing conda environments"
description: "Create, activate, export, and remove conda environments on CARC clusters."
type: Guide
tags:
  - Python
  - Conda
generated:
  by: "claude/fable-5"
  at: "2026-08-29T00:00:00Z"
sources:
  - id: quickbytes
    resource: "https://github.com/UNM-CARC/QuickBytes/blob/master/anaconda_intro.md"
    title: "UNM-CARC QuickBytes: anaconda_intro.md"
    author: "team:unm-carc"
    last_modified: "2020-01-28T11:30:20-07:00"
---

# Managing conda environments

### What is Anaconda?

Fundamentally, Anaconda is a distribution of Python and R with a collection of associated packages optimized for data science. The installation and management of these packages is handled with the Anaconda package manager Conda. Conda is more than just a package manager however, it also creates and manages the environments that packages are installed in to. The usage of environments means you can have multiple versions of certain software installed in different environments and avoid conflicts or incompatibilities between software or dependencies. This is accomplished by installing packages into a separate directory which is then appended to your `PATH` when that environment is activated.

### Creating a new conda environment

Let's create an environment on Hopper to run a python machine learning script that uses the TensorFlow library, python version 3.5, and the pandas library. Once you log in to Hopper using `ssh` load the anaconda software module with the command:

`module load anaconda3`

We use `conda` to create new environments and install/upgrade packages within environments. To create our machine learning environment we type:

`conda create --name TensorFlow python=3.5 pandas tensorflow`

The command you are calling here is `conda` and you are telling it you want to `create` a new environment named TensorFlow with the packages python version 3.5 specifically, pandas, and tensorflow. When you enter this command `conda` prints out the plan for this environment to `stdout`:

```bash
Solving environment: done

## Package Plan ##

  environment location: /users/yourusername/.conda/envs/TensorFlow

  added / updated specs:
    - pandas
    - python=3.5
    - tensorflow


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    certifi-2018.8.24          |           py35_1         139 KB
    termcolor-1.1.0            |           py35_1           7 KB
    pip-10.0.1                 |           py35_0         1.8 MB
    pytz-2018.5                |           py35_0         231 KB
    protobuf-3.6.0             |   py35hf484d3e_0         615 KB
    werkzeug-0.14.1            |           py35_0         426 KB
    astor-0.7.1                |           py35_0          43 KB
    libprotobuf-3.6.0          |       hdbcaa40_0         4.1 MB
    markdown-2.6.11            |           py35_0         104 KB
    mkl_fft-1.0.4              |   py35h4414c95_1         148 KB
    mkl_random-1.0.1           |   py35h629b387_0         364 KB
    tensorboard-1.10.0         |   py35hf484d3e_0         3.3 MB
    tensorflow-base-1.10.0     |mkl_py35h3c3e929_0        82.1 MB
    python-dateutil-2.7.3      |           py35_0         261 KB
    numpy-base-1.15.1          |   py35h81de0dd_0         4.2 MB
    wheel-0.31.1               |           py35_0          63 KB
    _tflow_1100_select-0.0.3   |              mkl           2 KB
    python-3.5.5               |       hc3d631a_4        28.3 MB
    setuptools-40.2.0          |           py35_0         571 KB
    grpcio-1.12.1              |   py35hdbcaa40_0         1.7 MB
    gast-0.2.0                 |           py35_0          15 KB
    absl-py-0.4.0              |   py35h28b3542_0         144 KB
    six-1.11.0                 |   py35h423b573_1          21 KB
    tensorflow-1.10.0          |mkl_py35heddcb22_0           4 KB
    pandas-0.23.4              |   py35h04863e7_0        10.0 MB
    numpy-1.15.1               |   py35h3b04361_0          37 KB
    ------------------------------------------------------------
                                           Total:       138.6 MB

The following NEW packages will be INSTALLED:

    _tflow_1100_select: 0.0.3-mkl
    absl-py:            0.4.0-py35h28b3542_0
    astor:              0.7.1-py35_0
    blas:               1.0-mkl
    ca-certificates:    2018.03.07-0
    certifi:            2018.8.24-py35_1
    gast:               0.2.0-py35_0
    grpcio:             1.12.1-py35hdbcaa40_0
    intel-openmp:       2018.0.3-0
    libedit:            3.1.20170329-h6b74fdf_2
    libffi:             3.2.1-hd88cf55_4
    libgcc-ng:          8.2.0-hdf63c60_1
    libgfortran-ng:     7.3.0-hdf63c60_0
    libprotobuf:        3.6.0-hdbcaa40_0
    libstdcxx-ng:       8.2.0-hdf63c60_1
    markdown:           2.6.11-py35_0
    mkl:                2018.0.3-1
    mkl_fft:            1.0.4-py35h4414c95_1
    mkl_random:         1.0.1-py35h629b387_0
    ncurses:            6.1-hf484d3e_0
    numpy:              1.15.1-py35h3b04361_0
    numpy-base:         1.15.1-py35h81de0dd_0
    openssl:            1.0.2p-h14c3975_0
    pandas:             0.23.4-py35h04863e7_0
    pip:                10.0.1-py35_0
    protobuf:           3.6.0-py35hf484d3e_0
    python:             3.5.5-hc3d631a_4
    python-dateutil:    2.7.3-py35_0
    pytz:               2018.5-py35_0
    readline:           7.0-ha6073c6_4
    setuptools:         40.2.0-py35_0
    six:                1.11.0-py35h423b573_1
    sqlite:             3.24.0-h84994c4_0
    tensorboard:        1.10.0-py35hf484d3e_0
    tensorflow:         1.10.0-mkl_py35heddcb22_0
    tensorflow-base:    1.10.0-mkl_py35h3c3e929_0
    termcolor:          1.1.0-py35_1
    tk:                 8.6.7-hc745277_3
    werkzeug:           0.14.1-py35_0
    wheel:              0.31.1-py35_0
    xz:                 5.2.4-h14c3975_4
    zlib:               1.2.11-ha838bed_2

Proceed ([y]/n)?
```

This gives you the list of all packages you requested to be installed and their dependencies, as well as the package version and build. Of note is the environment location pathway at the top of the package plan, you will notice that `conda` by default installs into your local directory and does not need administrative access to install packages. This means that you can administer your own Anaconda environments at CARC. 

When you verify the package plan `conda` will proceed with downloading package binaries and installing them into the environment directory. You will see the progress of installation and a message with how to activate your environment once complete:

```bash
Downloading and Extracting Packages
certifi-2018.8.24    |  139 KB | ####################################### | 100%
python-3.6.6         | 15.4 MB | ####################################### | 100%
tensorflow-base-1.10 | 55.3 MB | ####################################### | 100%
setuptools-40.2.0    |  554 KB | ####################################### | 100%
libprotobuf-3.6.0    |  3.8 MB | ####################################### | 100%
sqlite-3.24.0        |  2.2 MB | ####################################### | 100%
mkl-2018.0.3         | 149.2 MB| ###################################### | 100%
mkl_random-1.0.1     |  349 KB | ####################################### | 100%
mkl_fft-1.0.4        |  137 KB | ####################################### | 100%
openssl-1.0.2p       |  3.4 MB | ####################################### | 100%
six-1.11.0           |   21 KB | ####################################### | 100%
tensorflow-1.10.0    |    4 KB | ####################################### | 100%
numpy-base-1.15.1    |  4.0 MB | ####################################### | 100%
protobuf-3.6.0       |  604 KB | ####################################### | 100%
numpy-1.15.1         |   37 KB | ####################################### | 100%
intel-openmp-2018.0. | 1004 KB | ####################################### | 100%
absl-py-0.4.0        |  143 KB | ####################################### | 100%
tensorboard-1.10.0   |  3.3 MB | ####################################### | 100%
_tflow_1100_select-0 |    3 KB | ####################################### | 100%
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use:
# > source activate TensorFlow
#
# To deactivate an active environment, use:
# > source deactivate
#
```

Now we have our machine learning environment created to run our machine learning python script. To activate the environment we just created you use the command `source activate my_environment_name`, which is `source activate TensorFlow` for this example. Remember to include the lines below in your PBS script when working with Anaconda environments:

```bash
# load anaconda software module
module load anaconda3

# activate your desired anaconda environment
source activate environment_name
```
For more information on managing environments visit the Conda documentation site at this [link](https://conda.io/docs/user-guide/index.html){target=_blank}, or by adding the flag `--help` to any `conda` command, for example, `conda create --help` will print a help page for creating environments.

## Video walkthrough

**Conda environments** — from the [CARC video tutorials](../training/videos.md):

<iframe class="carc-video" src="https://www.youtube-nocookie.com/embed/gMJzDSeGk50" title="Conda environments" loading="lazy" allow="accelerometer; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen></iframe>

<p class="carc-provenance" markdown>Migrated from [UNM-CARC QuickBytes](https://github.com/UNM-CARC/QuickBytes/blob/master/anaconda_intro.md){target=_blank} (last source update 2020-01-28). Spotted a problem? [Open an issue or pull request](https://github.com/UNM-CARC/QuickBytes){target=_blank}.</p>

---8<--- https://unm-carc.github.io/docs/software/conda-intro/

---
title: "Conda and Anaconda: introduction"
description: "What conda is, how environments work, and how to use Anaconda/Miniconda on CARC systems."
type: Guide
tags:
  - Python
  - Conda
generated:
  by: "claude/fable-5"
  at: "2026-08-29T00:00:00Z"
sources:
  - id: quickbytes
    resource: "https://github.com/UNM-CARC/QuickBytes/blob/master/anaconda_general_intro.md"
    title: "UNM-CARC QuickBytes: anaconda_general_intro.md"
    author: "team:unm-carc"
    last_modified: "2020-01-28T10:52:07-07:00"
---

# Conda and Anaconda: introduction

### What is Anaconda?

At a basic level Anaconda is a distribution of Python and R, although there is an emphasis on working with python, that provides access collections of associated packages optimized specifically for data science maintained in repositories. The installation and management of these packages is handled with the Anaconda package manager Conda. While initially focused mainly on python packages the repositories hosted by Anaconda and others now house a large collection of non-python packages.  

Conda is more than just a package manager however, it also creates and manages the environments that packages are installed in to. The use of environments to isolate software means you can have multiple versions of the same software installed in different environments and avoid conflicts or incompatibilities between software or dependencies. This is accomplished by installing packages into a separate directory which is then appended to your `PATH` when that environment is activated.

The next couple of pages will provide a brief introduction on how to use Conda to create and maintain locally administered environments on the CARC machines. 

For more information on the usage and various features of Conda, please visit their website at this [link](https://conda.io/docs/){target=_blank}.

<p class="carc-provenance" markdown>Migrated from [UNM-CARC QuickBytes](https://github.com/UNM-CARC/QuickBytes/blob/master/anaconda_general_intro.md){target=_blank} (last source update 2020-01-28). Spotted a problem? [Open an issue or pull request](https://github.com/UNM-CARC/QuickBytes){target=_blank}.</p>

---8<--- https://unm-carc.github.io/docs/software/conda-jupyterhub/

---
title: "Conda environments in JupyterHub"
description: "Make your conda environments available as kernels in CARC JupyterHub."
type: Guide
tags:
  - Python
  - Conda
  - Jupyter
generated:
  by: "claude/fable-5"
  at: "2026-08-29T00:00:00Z"
sources:
  - id: quickbytes
    resource: "https://github.com/UNM-CARC/QuickBytes/blob/master/Conda_JupyterHub.md"
    title: "UNM-CARC QuickBytes: Conda_JupyterHub.md"
    author: "team:unm-carc"
    last_modified: "2023-06-27T14:50:38-06:00"
---

# Conda environments in JupyterHub

Custom environments created by users can also be used on JupyterHub. This QuickByte shows you how to make python environments  accessible on JupyterHub. 

## In Terminal 

If you are creating a conda environment from scrach that you know you will want to use on JupyterHub, as you are creating the environment add the ipykernel to the packages you want included.

For example, if you were making a natural language processing libraries environment, you could create an environment like this:

``` 
module load miniconda3
conda create -n nltk nltk ipykernel
```

Alternatively, if you already have an environment created and would like it to be available on JupyterHub, then add the ipykernal. 

``` 
source activate nltk
conda install ipykernel
```

Remember that you can also access the terminal though JupyterHub. To do this click Terminal under the New dropdown menu. 

![term_Jup](../assets/images/quickbytes/JuphuB_terminal.png)

## On JupyterHub

After your environments have been modified to include the ipykernal, you can open notebooks on JupyterHub by opening a New Notebook under File and selecting the environment. 

![term_Jup](../assets/images/quickbytes/JupHub_envi.png)

<p class="carc-provenance" markdown>Migrated from [UNM-CARC QuickBytes](https://github.com/UNM-CARC/QuickBytes/blob/master/Conda_JupyterHub.md){target=_blank} (last source update 2023-06-27). Spotted a problem? [Open an issue or pull request](https://github.com/UNM-CARC/QuickBytes){target=_blank}.</p>

---8<--- https://unm-carc.github.io/docs/software/dask-scikit-learn/

---
title: "Parallel Python with Dask and scikit-learn"
description: "Scale scikit-learn workloads across cluster nodes from JupyterHub using Dask."
type: Tutorial
tags:
  - Python
  - Jupyter
  - Parallel
  - Dask
generated:
  by: "claude/fable-5"
  at: "2026-08-29T00:00:00Z"
sources:
  - id: quickbytes
    resource: "https://github.com/UNM-CARC/QuickBytes/blob/master/parallel_jupyterhub_with_dask_and_scikit-learn.md"
    title: "UNM-CARC QuickBytes: parallel_jupyterhub_with_dask_and_scikit-learn.md"
    author: "team:unm-carc"
    last_modified: "2019-10-23T10:22:17-06:00"
---

# Parallel Python with Dask and scikit-learn

[Dask](https://dask.org/){target=_blank} uses existing Python APIs and data structures to make it easy to switch between Numpy, Pandas, Scikit-learn to their Dask-powered equivalents. [SciKit-learn](https://scikit-learn.org/stable/){target=_blank} is a machine learning tool for Python.

### Log in to JupyterHub

On a computer connected to ethernet on main campus, open an internet browser go to https://hopper.alliance.unm.edu where you will be asked to log in. Use your carc username and password. This logs you into a compute node where your programs in Jupyter notebook will be running. Because it is beginning an interactive job it may not be instant depending on resources available at the time. Once logged in, you can see all the files in your home directory. 

To be kind to other users when you are finished with JupyterHub for the day, please be sure to go to "control panel" in the top righthand corner and click "stop my server". This will free up the node for other users. Otherwise, the default walltime is 12 hours. 


## Setup cluster resources with Dask-jobqueue


```python
from dask_jobqueue import PBSCluster
```


```python
from dask.distributed import Client, progress
```


```python
import time
```


```python
cluster = PBSCluster(memory="42GB",cores=8, resource_spec="nodes=1:ppn=8", queue="default", walltime='01:00:00')
```


```python
print(cluster.job_script())
```

    #!/usr/bin/env bash
    
    #PBS -N dask-worker
    #PBS -q default
    #PBS -l nodes=1:ppn=8
    #PBS -l walltime=01:00:00
    JOB_ID=${PBS_JOBID%%.*}
    
    /opt/local/anaconda3/envs/jupyterhub/bin/python -m distributed.cli.dask_worker tcp://172.16.2.42:46451 --nthreads 8 --memory-limit 42.00GB --name name --nanny --death-timeout 60
    



```python
cluster.scale(4)
```

# A loop to check when all the resources are ready


```python
for x in range(10):
    print(cluster)
    time.sleep(5)
```

    PBSCluster('tcp://172.16.2.42:46451', workers=0, threads=0, memory=0 B)
    PBSCluster('tcp://172.16.2.42:46451', workers=1, threads=8, memory=42.00 GB)
    PBSCluster('tcp://172.16.2.42:46451', workers=4, threads=32, memory=168.00 GB)
    PBSCluster('tcp://172.16.2.42:46451', workers=4, threads=32, memory=168.00 GB)
    PBSCluster('tcp://172.16.2.42:46451', workers=4, threads=32, memory=168.00 GB)
    PBSCluster('tcp://172.16.2.42:46451', workers=4, threads=32, memory=168.00 GB)
    PBSCluster('tcp://172.16.2.42:46451', workers=4, threads=32, memory=168.00 GB)
    PBSCluster('tcp://172.16.2.42:46451', workers=4, threads=32, memory=168.00 GB)
    PBSCluster('tcp://172.16.2.42:46451', workers=4, threads=32, memory=168.00 GB)
    PBSCluster('tcp://172.16.2.42:46451', workers=4, threads=32, memory=168.00 GB)



```python
client = Client(cluster)
```

# Run a simple parallel program to test functionality


```python
def slow_increment(x): 
    time.sleep(1)
    return x + 1 

```


```python
futures = client.map(slow_increment, range(5000))
```


```python
progress(futures)
```


    VBox()


## Demonstrate how dask integrates with Scikit-Learn


```python
# Scikit-learn bundles joblib, so you need to import from
# `sklearn.externals.joblib` instead of `joblib` directly
from sklearn.externals.joblib import parallel_backend
from sklearn.datasets import load_digits
from sklearn.model_selection import RandomizedSearchCV
from sklearn.svm import SVC
import numpy as np

digits = load_digits()

param_space = {
    'C': np.logspace(-6, 6, 13),
    'gamma': np.logspace(-8, 8, 17),
    'tol': np.logspace(-4, -1, 4),
    'class_weight': [None, 'balanced'],
}

model = SVC(kernel='rbf')
search = RandomizedSearchCV(model, param_space, cv=3, n_iter=50, verbose=10)

# Serialize the training data only once to each worker
with parallel_backend('dask', scatter=[digits.data, digits.target]):
    search.fit(digits.data, digits.target)

```

    Fitting 3 folds for each of 50 candidates, totalling 150 fits


    [Parallel(n_jobs=-1)]: Using backend DaskDistributedBackend with 32 concurrent workers.
    [Parallel(n_jobs=-1)]: Done   8 tasks      | elapsed:    2.2s
    [Parallel(n_jobs=-1)]: Done  21 tasks      | elapsed:    2.9s
    [Parallel(n_jobs=-1)]: Done  34 tasks      | elapsed:    3.5s
    [Parallel(n_jobs=-1)]: Done  49 tasks      | elapsed:    3.8s
    [Parallel(n_jobs=-1)]: Done  64 tasks      | elapsed:    4.1s
    [Parallel(n_jobs=-1)]: Done  81 tasks      | elapsed:    4.6s
    [Parallel(n_jobs=-1)]: Done 103 out of 150 | elapsed:    5.0s remaining:    2.3s
    [Parallel(n_jobs=-1)]: Done 119 out of 150 | elapsed:    5.3s remaining:    1.4s
    [Parallel(n_jobs=-1)]: Done 135 out of 150 | elapsed:    5.7s remaining:    0.6s
    [Parallel(n_jobs=-1)]: Done 150 out of 150 | elapsed:    6.1s finished
    /opt/local/anaconda3/envs/jupyterhub/lib/python3.6/site-packages/sklearn/model_selection/_search.py:842: DeprecationWarning: The default of the `iid` parameter will change from True to False in version 0.22 and will be removed in 0.24. This will change numeric results when test-set sizes are unequal.
      DeprecationWarning)



```python
print(search)
```

    RandomizedSearchCV(cv=3, error_score='raise-deprecating',
```bash
      estimator=SVC(C=1.0, cache_size=200, class_weight=None, coef0=0.0,
```
      decision_function_shape='ovr', degree=3, gamma='auto_deprecated',
      kernel='rbf', max_iter=-1, probability=False, random_state=None,
      shrinking=True, tol=0.001, verbose=False),
```bash
      fit_params=None, iid='warn', n_iter=50, n_jobs=None,
      param_distributions={'C': array([1.e-06, 1.e-05, 1.e-04, 1.e-03, 1.e-02, 1.e-01, 1.e+00, 1.e+01,
   1.e+02, 1.e+03, 1.e+04, 1.e+05, 1.e+06]), 'gamma': array([1.e-08, 1.e-07, 1.e-06, 1.e-05, 1.e-04, 1.e-03, 1.e-02, 1.e-01,
   1.e+00, 1.e+01, 1.e+02, 1.e+03, 1.e+04, 1.e+05, 1.e+06, 1.e+07,
   1.e+08]), 'tol': array([0.0001, 0.001 , 0.01  , 0.1   ]), 'class_weight': [None, 'balanced']},
      pre_dispatch='2*n_jobs', random_state=None, refit=True,
      return_train_score='warn', scoring=None, verbose=10)
```



```python

```

<p class="carc-provenance" markdown>Migrated from [UNM-CARC QuickBytes](https://github.com/UNM-CARC/QuickBytes/blob/master/parallel_jupyterhub_with_dask_and_scikit-learn.md){target=_blank} (last source update 2019-10-23). Spotted a problem? [Open an issue or pull request](https://github.com/UNM-CARC/QuickBytes){target=_blank}.</p>

---8<--- https://unm-carc.github.io/docs/software/deep-learning-packages/

---
title: "Installing deep learning packages"
description: "Install GPU-enabled deep learning frameworks (PyTorch, TensorFlow) into conda environments."
type: Guide
tags:
  - Python
  - GPU
  - Machine learning
status: draft
generated:
  by: "claude/fable-5"
  at: "2026-08-29T00:00:00Z"
sources:
  - id: quickbytes
    resource: "https://github.com/UNM-CARC/QuickBytes/blob/master/Install%20deep%20learning%20packages.md"
    title: "UNM-CARC QuickBytes: Install deep learning packages.md"
    author: "team:unm-carc"
    last_modified: "2021-10-15T00:00:00Z"
---

# Installing deep learning packages

!!! note "Match builds to current GPUs"
    Choose framework builds that match the GPUs on the
    [current clusters](../systems/overview.md) (A100 on Hopper; L40S and
    H100 on Easley), and check exact package versions before installing.

This step-by-step guide walks through installing deep learning and machine
learning tools in a [conda environment](conda-intro.md) on CARC systems.

## Set up the conda environment

1. Load the Anaconda module to get the `conda` command:

    ```bash
    module load anaconda3
    ```

2. Create a conda environment with a name:

    ```bash
    conda create --name <env_name> python==3.6
    ```

3. Verify the environment was created:

    ```bash
    conda info --envs
    ```

4. Activate the environment:

    ```bash
    source activate <env_name>
    ```

## Install deep learning packages

Install one or more of the following, as your work requires.

=== "TensorFlow (GPU)"

    ```bash
    conda install -c anaconda tensorflow-gpu
    ```

=== "Keras (GPU)"

    ```bash
    conda install -c anaconda keras-gpu
    ```

=== "PyTorch (CPU)"

    ```bash
    conda install pytorch torchvision -c pytorch
    ```

=== "PyTorch (GPU, K40 legacy)"

    First make sure Python 3.7 is installed in your current environment:

    ```bash
    conda create -n <env_name> python==3.7
    source activate <env_name>
    ```

    Then install the K40-compatible build that CARC staged in shared storage,
    plus the matching CUDA toolkit:

    ```bash
    conda install /projects/shared/pytorch/PyTorch1.5-K40-Compatible/pytorch-1.5.0-py3.7_cuda10.1.243_cudnn7.6.3_0.tar.bz2
    conda install cudatoolkit=10.1.243
    ```

## Verify GPU access from PyTorch

Run the following Python code on a GPU node (request one first — see
[example Slurm scripts](../running-jobs/example-slurm-scripts.md)):

```python
import torch
from torch import nn, tensor
from torch.cuda import device_count

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
x = torch.rand(5, 3)
print(x)
print("Is GPU Available?", torch.cuda.is_available(),
      " CUDA device count:", torch.cuda.device_count(),
      "current_device:", torch.cuda.current_device())
x = torch.tensor([1, 2, 3], device=device)
y = torch.tensor([1, 4, 9]).to(device)
print(x, y)
print(x + y)
```

Expected output:

```text
tensor([[0.3220, 0.2174, 0.1226],
        [0.7249, 0.8111, 0.8414],
        [0.5974, 0.5169, 0.5242],
        [0.1436, 0.5150, 0.5688],
        [0.3298, 0.1289, 0.5349]])
Is GPU Available? True  CUDA device count: 1  current_device: 0
tensor([1, 2, 3], device='cuda:0') tensor([1, 4, 9], device='cuda:0')
tensor([ 2,  6, 12], device='cuda:0')
```

## Additional machine learning packages

```bash
# OpenCV
conda install -c conda-forge opencv

# numpy, pandas, matplotlib, scikit-learn
conda install numpy pandas matplotlib scikit-learn
```

## Related pages

* [PyTorch on CARC GPUs](pytorch.md)
* [TensorFlow on CARC GPUs](tensorflow.md)
* [Conda channels and pip](conda-channels-pip.md)
* [Conda environments in JupyterHub](conda-jupyterhub.md)

<p class="carc-provenance" markdown>Migrated from [UNM-CARC QuickBytes](https://github.com/UNM-CARC/QuickBytes/blob/master/Install%20deep%20learning%20packages.md){target=_blank} (last source update 2021-10-15), then restructured with fenced code blocks and curated in this repository. Spotted a problem? [Open an issue or pull request](https://github.com/UNM-CARC/QuickBytes){target=_blank}.</p>

---8<--- https://unm-carc.github.io/docs/software/getting-r/

---
title: "Getting R software"
description: "Available R versions and how to load them with environment modules."
type: Guide
tags:
  - R
  - Modules
generated:
  by: "claude/fable-5"
  at: "2026-08-29T00:00:00Z"
sources:
  - id: quickbytes
    resource: "https://github.com/UNM-CARC/QuickBytes/blob/master/R_at_CARC/getting_R_software.md"
    title: "UNM-CARC QuickBytes: R_at_CARC/getting_R_software.md"
    author: "team:unm-carc"
    last_modified: "2019-12-11T12:02:59-07:00"
---

# Getting R software

## Getting R in the first place

There are three options for accessing R at Carc and I will run through both approaches since there are pros and cons to each.
### Option 1
The first option is to activate an installed R module. When logged in to a CARC system you can use the `module avail` command to see which R versions are available. If you have a CARC account open a terminal and log in to follow along. 

```
yourusername@hopper$ module avail r-
```

which will print out the following (although I have truncated the output):

```
----------------------- /opt/spack/share/spack/modules/linux-centos7-x86_64 ------------------------
   ...
   r-3.4.1-gcc-4.8.5-python2-gzeg24m
   r-3.4.1-gcc-4.8.5-python2-zpkgqap
   r-3.4.1-intel-17.0.4-mkl-python2-67zsm3b
   r-3.4.1-intel-17.0.4-mkl-python2-gygkoab
   r-3.4.2-intel-18.0.2-python2-xsxuxwx
   r-3.4.3-gcc-4.8.5-python2-gk66fni
   r-3.4.3-gcc-4.8.5-python2-qv6gwz6
   r-3.4.3-gcc-6.1.0-python2-lyqiytq
   r-3.4.3-gcc-7.3.0-python2-zhxbajj
   r-3.4.3-intel-18.0.1-python2-3l4dkgz
   r-3.4.3-intel-18.0.1-python2-lr24ix6
   r-3.4.3-intel-18.0.2-python2-q3covk7
   r-3.5.0-gcc-4.8.5-python2-khqxja7
   r-3.5.0-gcc-7.3.0-python2-rvq3qk5
   r-3.5.0-intel-18.0.2-python2-mkl-r6lx6yy
   r-3.5.3-gcc-7.3.0-python2-ziiolp5
   r-3.6.0-gcc-4.8.5-python2-i4uimtp
   r-3.6.0-gcc-7.3.0-python2-7akol5t
   ...
Use "module spider" to find all possible modules.
Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys".
```

These are the R modules available on the cluster. In order to activate a R software module you use the `module load` command. For example:

```
yourusername@hopper$ module load r-3.6.0-gcc-7.3.0-python2-7akol5t
yourusername@hopper$ R

R version 3.6.0 (2019-04-26) -- "Planting of a Tree"
Copyright (C) 2019 The R Foundation for Statistical Computing
Platform: x86_64-pc-linux-gnu (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

  Natural language support but running in an English locale

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

> 
```

Will load R-3.6.0 that has been compiled with GCC-7.3.0. Normally you will be running R jobs in batch mode as oppposed to interactively, which means you will have the `module load` command in your PBS script, but we will get to that later. 

### Option 2
The second option is to create a custom local Anaconda environment with the version of R that would like to run. In order to do this you need to first load an Anaconda software module and then create a new environment according to your specifications. For example, the following commands will create an Anaconda environment with R-3.4.3:

```
yourusername@hopper$ module load anaconda3
yourusername@hopper$ conda create --yes --name my_r_env r=3.4.3
Solving environment: done

## Package Plan ##

environment location: /users/yourusername/.conda/envs/my_r_env

added / updated specs: 
- r=3.4.3


The following packages will be downloaded:

package                    |            build
---------------------------|-----------------
gfortran_linux-64-7.3.0    |       h553295d_8         148 KB
r-cluster-2.0.6            |   r343h4829c52_0         519 KB
r-lattice-0.20_35          |   r343h086d26f_0         713 KB
r-base-3.4.3               |       h9bb98a2_5        38.3 MB
r-recommended-3.4.3        |           r343_0           3 KB
r-3.4.3                    |           r343_0           3 KB
r-mgcv-1.8_22              |   r343h086d26f_0         2.4 MB
r-matrix-1.2_12            |   r343h086d26f_0         2.5 MB
r-kernsmooth-2.23_15       |   r343h4829c52_4         101 KB
r-class-7.3_14             |   r343h086d26f_4          93 KB
r-nlme-3.1_131             |   r343h4829c52_0         2.2 MB
r-foreign-0.8_69           |   r343h086d26f_0         256 KB
r-survival-2.41_3          |   r343h086d26f_0         5.1 MB
gcc_linux-64-7.3.0         |       h553295d_8         149 KB
r-spatial-7.3_11           |   r343h086d26f_4         140 KB
gcc_impl_linux-64-7.3.0    |       habb00fd_1        73.2 MB
gxx_impl_linux-64-7.3.0    |       hdf63c60_1        18.7 MB
gxx_linux-64-7.3.0         |       h553295d_8         148 KB
r-boot-1.3_20              |   r343h889e2dd_0         625 KB
binutils_linux-64-2.31.1   |       h6176602_8         148 KB
r-mass-7.3_48              |   r343h086d26f_0         1.1 MB
bzip2-1.0.8                |       h7b6447c_0         105 KB
ca-certificates-2019.5.15  |                1         134 KB
r-codetools-0.2_15         |   r343h889e2dd_0          49 KB
r-rpart-4.1_11             |   r343h086d26f_0         899 KB
r-nnet-7.3_12              |   r343h086d26f_0         118 KB
libxml2-2.9.9              |       hea5a465_1         2.0 MB
------------------------------------------------------------
                                       Total:       149.8 MB

The following NEW packages will be INSTALLED:

_libgcc_mutex:          0.1-main               
_r-mutex:               1.0.0-anacondar_1      
binutils_impl_linux-64: 2.31.1-h6176602_1      
binutils_linux-64:      2.31.1-h6176602_8      
bwidget:                1.9.11-1               
bzip2:                  1.0.8-h7b6447c_0       
ca-certificates:        2019.5.15-1            
cairo:                  1.14.12-h8948797_3     
curl:                   7.65.2-hbc83047_0      
fontconfig:             2.13.0-h9420a91_0      
freetype:               2.9.1-h8a8886c_1       
fribidi:                1.0.5-h7b6447c_0       
gcc_impl_linux-64:      7.3.0-habb00fd_1       
gcc_linux-64:           7.3.0-h553295d_8       
gfortran_impl_linux-64: 7.3.0-hdf63c60_1       
gfortran_linux-64:      7.3.0-h553295d_8       
glib:                   2.56.2-hd408876_0      
graphite2:              1.3.13-h23475e2_0      
gxx_impl_linux-64:      7.3.0-hdf63c60_1       
gxx_linux-64:           7.3.0-h553295d_8       
harfbuzz:               1.8.8-hffaf4a1_0       
icu:                    58.2-h9c2bf20_1        
jpeg:                   9b-h024ee3a_2          
krb5:                   1.16.1-h173b8e3_7      
libcurl:                7.65.2-h20c2e04_0      
libedit:                3.1.20181209-hc058e9b_0
libffi:                 3.2.1-hd88cf55_4       
libgcc-ng:              9.1.0-hdf63c60_0       
libgfortran-ng:         7.3.0-hdf63c60_0       
libpng:                 1.6.37-hbc83047_0      
libssh2:                1.8.2-h1ba5d50_0       
libstdcxx-ng:           9.1.0-hdf63c60_0       
libtiff:                4.0.10-h2733197_2      
libuuid:                1.0.3-h1bed415_2       
libxcb:                 1.13-h1bed415_1        
libxml2:                2.9.9-hea5a465_1       
ncurses:                6.1-he6710b0_1         
openssl:                1.1.1c-h7b6447c_1      
pango:                  1.42.4-h049681c_0      
pcre:                   8.43-he6710b0_0        
pixman:                 0.38.0-h7b6447c_0      
r:                      3.4.3-r343_0           
r-base:                 3.4.3-h9bb98a2_5       
r-boot:                 1.3_20-r343h889e2dd_0  
r-class:                7.3_14-r343h086d26f_4  
r-cluster:              2.0.6-r343h4829c52_0   
r-codetools:            0.2_15-r343h889e2dd_0  
r-foreign:              0.8_69-r343h086d26f_0  
r-kernsmooth:           2.23_15-r343h4829c52_4 
r-lattice:              0.20_35-r343h086d26f_0 
r-mass:                 7.3_48-r343h086d26f_0  
r-matrix:               1.2_12-r343h086d26f_0  
r-mgcv:                 1.8_22-r343h086d26f_0  
r-nlme:                 3.1_131-r343h4829c52_0 
r-nnet:                 7.3_12-r343h086d26f_0  
r-recommended:          3.4.3-r343_0           
r-rpart:                4.1_11-r343h086d26f_0  
r-spatial:              7.3_11-r343h086d26f_4  
r-survival:             2.41_3-r343h086d26f_0  
readline:               7.0-h7b6447c_5         
tk:                     8.6.8-hbc83047_0       
tktable:                2.10-h14c3975_0        
xz:                     5.2.4-h14c3975_4       
zlib:                   1.2.11-h7b6447c_3      
zstd:                   1.3.7-h0b5b093_0       


Downloading and Extracting Packages
gfortran_linux-64-7. |  148 KB | ########################################################### | 100% 
r-cluster-2.0.6      |  519 KB | ########################################################### | 100% 
r-lattice-0.20_35    |  713 KB | ########################################################### | 100% 
r-base-3.4.3         | 38.3 MB | ########################################################### | 100% 
r-recommended-3.4.3  |    3 KB | ########################################################### | 100% 
r-3.4.3              |    3 KB | ########################################################### | 100% 
r-mgcv-1.8_22        |  2.4 MB | ########################################################### | 100% 
r-matrix-1.2_12      |  2.5 MB | ########################################################### | 100% 
r-kernsmooth-2.23_15 |  101 KB | ########################################################### | 100% 
r-class-7.3_14       |   93 KB | ########################################################### | 100% 
r-nlme-3.1_131       |  2.2 MB | ########################################################### | 100% 
r-foreign-0.8_69     |  256 KB | ########################################################### | 100% 
r-survival-2.41_3    |  5.1 MB | ########################################################### | 100% 
gcc_linux-64-7.3.0   |  149 KB | ########################################################### | 100% 
r-spatial-7.3_11     |  140 KB | ########################################################### | 100% 
gcc_impl_linux-64-7. | 73.2 MB | ########################################################### | 100% 
gxx_impl_linux-64-7. | 18.7 MB | ########################################################### | 100% 
gxx_linux-64-7.3.0   |  148 KB | ########################################################### | 100% 
r-boot-1.3_20        |  625 KB | ########################################################### | 100% 
binutils_linux-64-2. |  148 KB | ########################################################### | 100% 
r-mass-7.3_48        |  1.1 MB | ########################################################### | 100% 
bzip2-1.0.8          |  105 KB | ########################################################### | 100% 
ca-certificates-2019 |  134 KB | ########################################################### | 100% 
r-codetools-0.2_15   |   49 KB | ########################################################### | 100% 
r-rpart-4.1_11       |  899 KB | ########################################################### | 100% 
r-nnet-7.3_12        |  118 KB | ########################################################### | 100% 
libxml2-2.9.9        |  2.0 MB | ########################################################### | 100% 
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use:
# > source activate my_r_env
#
# To deactivate an active environment, use:
# > source deactivate
#
```
Then to use your newly created R environment you need to 1) make sure you have the Anaconda software module loaded, and 2), activate your conda envioronment.

```
yourusername@hopper$ module load anaconda3
yourusername@hopper$ source activate my_r_env
yourusername@hopper$ R

R version 3.4.3 (2017-11-30) -- "Kite-Eating Tree"
Copyright (C) 2017 The R Foundation for Statistical Computing
Platform: x86_64-conda_cos6-linux-gnu (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

  Natural language support but running in an English locale

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

> 
```
### Option 3

The third option is to user JupyterHub. To do so direct your browser to https://hopper.alliance.unm.edu and log in with your CARC credentials. Click on the "New" drop down menu and select "R". Now you have a R session running on the cluster through JupyterHub.

<p class="carc-provenance" markdown>Migrated from [UNM-CARC QuickBytes](https://github.com/UNM-CARC/QuickBytes/blob/master/R_at_CARC/getting_R_software.md){target=_blank} (last source update 2019-12-11). Spotted a problem? [Open an issue or pull request](https://github.com/UNM-CARC/QuickBytes){target=_blank}.</p>

---8<--- https://unm-carc.github.io/docs/software/gurobi-r/

---
title: "Gurobi optimizer with R"
description: "Use the Gurobi optimization solver from R on CARC clusters."
type: Guide
tags:
  - R
  - Optimization
generated:
  by: "claude/fable-5"
  at: "2026-08-29T00:00:00Z"
sources:
  - id: quickbytes
    resource: "https://github.com/UNM-CARC/QuickBytes/blob/master/Gurobi%20optimizer%20with%20R.md"
    title: "UNM-CARC QuickBytes: Gurobi optimizer with R.md"
    author: "team:unm-carc"
    last_modified: "2021-02-17T14:09:22-07:00"
---

# Gurobi optimizer with R

[Gurobi optimizer](https://www.gurobi.com/products/gurobi-optimizer/){target=_blank} is a problem solving software that can be used within R. It can solve integer, linear, and quadratic
programming optimizations. These techniques can help to find the answers to complex models. 


## Example of running Gurobi optimizer with R at CARC

There are modules for both Gurobi and R on the CARC clusters. All you need to do is load them, and then start an R
session. This command is for version 8.1.0, however there are other versions of gurobi available (enter `module avail gurobi` to see a full list).
```
username@hopper:~$ module load gurobi/8.1.0
username@hopper:~$ module load r-3.6.0-gcc-7.3.0-python2-7akol5t
username@hopper:~$ R
```
Once you have started an R session, you can install packages just as you would in R. If you ever run into issues loading 
packages in R at CARC, you can reach out for assistance by emailling help@carc.unm.edu. One piece of advice if you are using 
JupyterHub to run an R notebook at CARC is you may need to install packages from ther terminal window on JupyterHub because 
the notebook will not let you interactiively answer questions installs may need. 

Start by installing the gurobi package:
```
> install.packages('/opt/local/gurobi/8.1.0/linux64/R/gurobi_8.1-0_R_3.5.0.tar.gz')
Installing package into '/users/username/R/x86_64-pc-linux-gnu-library/3.6'
* installing *binary* package 'gurobi' ...
* DONE (gurobi)
```
You should now be able to load the gurobi library in an R session:
```
> library(gurobi)
Loading required package: slam
```
Note that if you get an error regarding slam, you can install it using the command:
```
install.packages("slam", repos = "https://cloud.r-project.org")
```
Now let's runs a quick model as an example of what Gurobi can do and to see if everything is working properly: 
```
> model <- list()
> model$A          <- matrix(c(1,2,3,1,1,0), nrow=2, ncol=3, byrow=T)
> model$obj        <- c(1,1,2)
> model$modelsense <- 'max'
> model$rhs        <- c(4,1)
> model$sense      <- c('<', '>')
> model$vtype      <- 'B'
> params <- list(OutputFlag=0)
> result <- gurobi(model, params)
> print('Solution:')
[1] "Solution:"
> print(result$objval)
[1] 3
> print(result$x)
[1] 1 0 1
```

<p class="carc-provenance" markdown>Migrated from [UNM-CARC QuickBytes](https://github.com/UNM-CARC/QuickBytes/blob/master/Gurobi%20optimizer%20with%20R.md){target=_blank} (last source update 2021-02-17). Spotted a problem? [Open an issue or pull request](https://github.com/UNM-CARC/QuickBytes){target=_blank}.</p>

---8<--- https://unm-carc.github.io/docs/software/jupyterhub-mpi/

---
title: "MPI parallelization from JupyterHub"
description: "Run MPI-parallel Python (mpi4py/ipyparallel) from CARC JupyterHub sessions."
type: Tutorial
tags:
  - Python
  - Jupyter
  - MPI
  - Parallel
generated:
  by: "claude/fable-5"
  at: "2026-08-29T00:00:00Z"
sources:
  - id: quickbytes
    resource: "https://github.com/UNM-CARC/QuickBytes/blob/master/parallelization_with%20Jupyterhub_using_mpi.md"
    title: "UNM-CARC QuickBytes: parallelization_with Jupyterhub_using_mpi.md"
    author: "team:unm-carc"
    last_modified: "2019-10-23T11:28:06-06:00"
---

# MPI parallelization from JupyterHub

The following steps will show you the steps to use MPI through ipython's ipyparallel interface. 

### Create a PBS profile on CARC 

Once you are logged in at carc run these steps:

```console
cd /projects/systems/shared/ipython_cluster_profiles
cp -r profile_pbs ~/.ipython/
```

Then check the files copied. 

```console
cd ~/.ipython/profile_pbs
```

Now on JupyterHub go to the IPython Clusters tab (refresh if already open) and you should see a pbs profile now available to you. Click the JupyterHub icon in the upper left of your screen if you can't see the clusters tab.

You can start a job by setting number of engine in the 'pbs' cluster profile and clicking start under actions. For this example we will request 8 ipython compute engines.

[Optional] Since ipython's ipyparallel system is requesting compute nodes through the torque PBS system you will have to wait until the nodes are running before you can run 
code on them. Check that the job is running in terminal with 

```console
watch qstat -tn -u <username>


You should see something like the following:

Every 2.0s: qstat -t -n -u $USER     Wed Oct 23 09:15:14 2019                                                                                      
hopper.alliance.unm.edu:
                                                                                  Req'd       Req'd	  Elap
Job ID                  Username    Queue    Jobname          SessID  NDS   TSK   Memory      Time    S   Time
----------------------- ----------- -------- ---------------- ------ ----- ------ --------- --------- - ---------
258370.hopper.alli  mfricke     default  jupyterhub        21730     1	1	--   08:00:00 R  00:06:45
   wheeler291/1
258371.hopper.alli  mfricke     default  ipython_controll  22553     1	1	--   01:00:00 R  00:06:11
   wheeler291/2
258372.hopper.alli  mfricke     default  ipython_engine	3213     2     16	--   01:00:00 R  00:06:11
   wheeler176/0-7+wheeler175/0-7
```

Notice the ipython engines are running with status 'R'. You can also check to see whether the compute engines are ready in your python notebook (see below).

To exit the watch command use control-C 

To change the walltime of your profile, in the ~/.ipython/profile_pbs directory edit the pbs.engine.template and the pbs.controller.template to fit the requirments for your job. By editing these files you can also change from the default to debug queue as you are testing your program. 

Now you can open a Jupyter notebook and follow the remainder of this tutorial.

## Creating an example function that uses MPI

Create a new file in your home directory and name it psum.py. Enter the following into psum.py and save the file.

```python
from mpi4py import MPI
import numpy as np

def psum(a):
    locsum = np.sum(a)
    rcvBuf = np.array(0.0,'d')
    MPI.COMM_WORLD.Allreduce([locsum, MPI.DOUBLE],
        [rcvBuf, MPI.DOUBLE],
        op=MPI.SUM)
    return rcvBuf
```

This function performs a distributed sum over all the nodes on the MPI communications group.

## Create a Jupyter Notebook to Call Our MPI Function

Create a new Python 3 notebook in Jupyterhub and name it mpi_test.ipynb. Enter the following into cells of your notebook. Many of the commands are run on the MPI cluster and so are asynchronous. To check whether an operation has completed we check the status with ".wait_interactive()". When the status reports "done" you can move on to the next step.
 
## Load required packages for ipyparallel and MPI


```python
import ipyparallel as ipp
from mpi4py import MPI
import numpy as np
```

## Create a cluster to use the CPUs allocated thrugh PBS


```python
cluster = ipp.Client(profile='pbs')
```

## Check if the cluster is ready. We are looking for 8 ids since we asked for 8 engines.

Engines in ipparallel parlence are the same as processes or workers in other parallel systems.


```python
cluster.ids
```
[0, 1, 2, 3, 4, 5, 6, 7]

```python
len(cluster[:])
```
8

## Assign the engines to a variable named "view" to allow us to interact with them

```python
view = cluster[:]
```

Enable ipython `magics´. These are ipython helper functions such as %


```python
view.activate()
```

## Check to see if the MPI communication world is of the expected size. It should be size 8 since we have 8 engines.

Note we are running the Get_size command on each engine to make sure they all see the same MPI comm world. %px simply executes the code following it on each compute engine in parallel.


```python
status_mpi_size=%px size = MPI.COMM_WORLD.Get_size()
```


```python
status_mpi_size.wait_interactive()
```
done


The output of viewing the size variable should be an array with the same number of entries as engines, and each entry should be the number of engines requested.


```python
view['size']
```
[8, 8, 8, 8, 8, 8, 8, 8]
    
## Run the external python code in ´psum.py´ on all the engines.
Recall that psum.py just loads the MPI libraries and defines the distributed sum function, psum. We are not actually calling the psum function yet.

```python
status_psum_run=view.run('psum.py')
```
```python
status_psum_run.wait_interactive()
```

done

## Send data to all nodes to by summed
The scatter command sends 32 values from 0 to 31 to the 8 compute engines. Each compute engine gets 32/8=4 values. This is the ipyparallel scatter command, not an MPI scatter command.
```python
status_scatter=view.scatter('a',np.arange(32,dtype='float'))
```
done

We can view the variable 'a' on all the compute engines. The value of 'a' for each compute engine is an element of the return array. In this case each value is itself an array.

```python
view['a']
```
[array([0., 1., 2., 3.]),
     array([4., 5., 6., 7.]),
     array([ 8.,  9., 10., 11.]),
     array([12., 13., 14., 15.]),
     array([16., 17., 18., 19.]),
     array([20., 21., 22., 23.]),
     array([24., 25., 26., 27.]),
     array([28., 29., 30., 31.])]

## Execute the psum function on all the compute engines and store the result in totalsum
MPI code has to be executed on each compute engine so they can each perform the MPI reduce. This is accomplished by running calling the psum function on all the compute engines simultaniosly. MPI will allow them to communicate with each other to calculate the sum. 
```python
status_psum_call=%px totalsum = psum(a)
```

```python
status_psum_call.wait_interactive()
```   
done

## Check the value of totalsum on each node
Total should be equal to 31(31+1)/2=496

```python
view['totalsum']
```

[array(496.),
     array(496.),
     array(496.),
     array(496.),
     array(496.),
     array(496.),
     array(496.),
     array(496.)]

Each compute engine calculated the sum of all the values. Since we ran this MPI function on all the compute engines they report the same value.

## Defining functions in the notebook
Rather than loading psum from file we can define it in the notebook using the ipython function decorator '@'. 

```python
@view.remote(block = True)
def inlinesum():
    from mpi4py import MPI
    import numpy as np
    locsum = np.sum(a)
    rcvBuf = np.array(0.0,'d')
    MPI.COMM_WORLD.Allreduce([locsum, MPI.DOUBLE],
        [rcvBuf, MPI.DOUBLE],
        op=MPI.SUM)    
    return rcvBuf
```

Now we can call inlinesum and it is automatically run on every compute engine. The call is through ipyparallels but the computation is still using MPI.

```python
inlinesum()
```
 [array(496.),
     array(496.),
     array(496.),
     array(496.),
     array(496.),
     array(496.),
     array(496.),
     array(496.),
     array(496.),
     array(496.),
     array(496.),
     array(496.),
     array(496.),
     array(496.),
     array(496.),
     array(496.)]

<p class="carc-provenance" markdown>Migrated from [UNM-CARC QuickBytes](https://github.com/UNM-CARC/QuickBytes/blob/master/parallelization_with%20Jupyterhub_using_mpi.md){target=_blank} (last source update 2019-10-23). Spotted a problem? [Open an issue or pull request](https://github.com/UNM-CARC/QuickBytes){target=_blank}.</p>

---8<--- https://unm-carc.github.io/docs/software/matlab-deep-learning/

---
title: "MATLAB deep learning"
description: "Train deep learning models in MATLAB using CARC GPU nodes."
type: Tutorial
tags:
  - MATLAB
  - GPU
  - Machine learning
status: draft
generated:
  by: "claude/fable-5"
  at: "2026-08-29T00:00:00Z"
sources:
  - id: quickbytes
    resource: "https://github.com/UNM-CARC/QuickBytes/blob/master/MATLAB%20Deep%20Learning%20on%20Xena.md"
    title: "UNM-CARC QuickBytes: MATLAB Deep Learning on Xena.md"
    author: "team:unm-carc"
    last_modified: "2022-05-25T16:04:24-06:00"
---

# MATLAB deep learning

MATLAB has great tools for deep learning and convolutional neural networks (CNNs).
These tools can make use of GPUs, which are available on the CARC clusters — see the [current systems](../systems/overview.md).
This Quickbytes tutorial will mimic the official Mathworks tutorial on using deep learning for JPEG Image Deblocking.
To see that tutorial, follow this [link.](https://www.mathworks.com/help/images/jpeg-image-deblocking-using-deep-learning.html#JPEGImageDeblockingUsingDeepLearningExample-2 "MathWorks Deep Learning Tutorial")

Before we begin, create a directory named 'deepLearningExample' from within your home directory.
```bash
hopper:~$ cd ~
hopper:~$ mkdir deepLearningExample
```

## Table of Contents

1. [Train CNN Interactively](#1)
    1. [Open MATLAB](#1.1)
    2. [Get Required Example Functions](#1.2)
        1. [Locate them interactively in MATLAB GUI](#1.2.1)
        2. [Locate them using the terminal](#1.2.2)
    3. [Create MATLAB script](#1.3)
        1. [Subsequent uses of this Script](#1.3.1)
    4. [Train Interactively](#1.4)
2. [Train CNN via Scheduled Job](#2)
    1. [Slurm Scripts](#2.1)
        1. [Single GPU](#2.1.1)
        2. [Dual GPU](#2.1.2)
    2. [Submit Script](#2.2)
    3. [View Results](#2.3)
3. [Test Model](#3)
    1. [MATLAB Script](#3.1)
    2. [Changes you can make to the test_model.m script](#3.2)
        1. [Different test image](#3.2.1)
        2. [Zoom in on specific ROI](#3.2.2)


## Train CNN Interactively <a name="1"></a>
It is possible to train the the example CNN in an interactive MATLAB session and track the progress.
This requires X11 fowarding.
For a full guide on how to do that, please view our [Quickbytes youtube tutorial.](https://www.youtube.com/watch?v=-5ic9JWHuqI&t=224s&ab_channel=UNMCARC "Quickbytes X11 Forwarding Tutorial").
Please note that you should not fully train a CNN interactively as the plotting requires large amounts of memory.
It is alright to use the interactive plotting to verify that a model is training.
In order to fully train a model, please schedule a job using a slurm script and disable the plotting.

### Open MATLAB <a name="1.1"></a>

It is highly recommended to request two GPUs when available.
The commands below show both single- and dual-GPU requests.

Once logged in with X11 forwarding, you can begin an interactive job on a compute node.
```bash
hopper:~$ srun --x11 --mem 0 --ntasks 1 --cpus-per-task 16 -G 1 --pty bash
```

This requests a single node and gpu with X11 forwarding.

To use a machine with two gpus, use this command instead:
```bash
hopper:~$ srun --x11 --mem 0 --ntasks 1 --cpus-per-task 16 -G 2 --pty bash
```

Once you are assigned a compute node, cd into our new directory, then start an interactive session of MATLAB:
```bash
node:~$ cd deepLearningExample/
node:~$ module load matlab
node:~$ matlab
```

This should bring up the MATLAB graphical user interface (GUI).

Use the file brower on the left side of the window to move into your 'deepLearningExample' directory within MATLAB.

### Get Required Example Functions <a name="1.2"></a>

Before continuing, you must move the given MathWorks MATLAB functions (.m files) into yourly created 'deepLearningExample` directory. 
There are two ways to locate these files.

1. Locate them interactively in MATLAB GUI.
2. Locate them using the terminal. 

#### Locate them interactively in MATLAB GUI <a name="1.2.1"></a>

Follow these steps to locate the files from with the MATLAB GUI opened in the above step.

1. Attempt to use an example function with blank arguments in the MATLAB Command Window.
```bash
>> downloadIAPRTC12Data('','')
```
2. That will cause an error and provide links to the examples MATLAB thinks you are using. Click on the `JPEG Image Deblocking Using Deep Learning` link.
3. This will move MATLAB's current wortking directory to that of the Example code.
4. In the file browser on the left side of the window, select all of the files in the current directory.
5. Right click the selected files and hit 'copy'.
6. Use the file brower on the left side to navigate back to your 'deepLearningExample' directory.
7. Right click in the file brower and select paste to place all the files in this directory.

#### Locate them using the terminal <a name="1.2.2"></a>

Follow these steps to copy the required example code into your new directory.

1. Ssh into the compute node assigned to you (make sure the MATLAB module is loaded).
```bash
hopper:~$ ssh $NODE   # the compute node assigned to you
```
2. Move into the MATLAB Examples directory.
```bash
node:~$ cd /tmp/Examples/R2021a/deeplearning_shared/JPEGImageDeblockingDeepLearningExample
```
3. Copy all the needed files.
```bash
node:~$ cp *.m ~/deepLearningExample
```
4. (Optional) Copy the pretrained example CNN.
```bash
node:~$ cp pretrianedJPEGDnCNN.mat ~/deepLearningExample
```


### Create MATLAB script <a name="1.3"></a>
Now, create the following MATLAB script with the name 'deep_learning_example.m' and ensure it lives within the directory created above ('~/deepLearningExample/').
```bash
dataDir = '~/deepLearningExample/data';

% Ensure the data is downloaded first
isDownloaded = false;
if ~isDownloaded
    downloadIAPRTC12Data('http://www-i6.informatik.rwth-aachen.de/imageclef/resources/iaprtc12.tgz',dataDir);
end

% Grab Subset of data for training
trainImagesDir = fullfile(dataDir, "iaprtc12","images","00");
exts = [".jpg",".bmp",".png"];
imdsPristine = imageDatastore(trainImagesDir,FileExtensions=exts);

% Prepare Training Data by compressing at various levels of quality

JPEGQuality = [5:5:40 50 60 70 80];

isCompressed = false;

if ~isCompressed
    [compressedDirName,residualDirName] = createJPEGDeblockingTrainingSet(imdsPristine,JPEGQuality);
else
    compressedDirName = fullfile(dataDir,"iaprtc12","images","00","compressedImages");
    residualDirName = fullfile(dataDir,"iaprtc12","images","00","residualImages");
end

% Create Random Patch Extraction Datastore for Training

imdsCompressed = imageDatastore(compressedDirName,FileExtensions=".mat",ReadFcn=@matRead);
imdsResidual = imageDatastore(residualDirName,FileExtensions=".mat",ReadFcn=@matRead);

augmenter = imageDataAugmenter(...
    RandRotation=@()randi([0,1],1)*90,...
    RandXReflection=true);


patchSize = 50;
patchesPerImage = 128;

% Prepare dataset and setup CNN options

dsTrain = randomPatchExtractionDatastore(imdsCompressed,imdsResidual,patchSize, ...
    PatchesPerImage=patchesPerImage, ...
    DataAugmentation=augmenter);
dsTrain.MiniBatchSize = patchesPerImage;

inputBatch = preview(dsTrain);
disp(inputBatch)

layers = dnCNNLayers

maxEpochs = 10;
initLearningRate = 0.1;
l2reg = 0.0001;
batchSize = 64;

options = trainingOptions("sgdm", ...
    Momentum=0.9, ...
    InitialLearnRate=initLearningRate, ...
    LearnRateSchedule="piecewise", ...
    GradientThresholdMethod="absolute-value", ...
    GradientThreshold=0.005, ...
    L2Regularization=l2reg, ...
    MiniBatchSize=batchSize, ...
    MaxEpochs=maxEpochs, ...
    % WARNING - turning on plots uses massive amounts of RAM. Do not run for more than 3 or 4 epochs.
    %Plots="training-progress", ...
    Plots="none", ...
    ExecutionEnvironment='multi-gpu',...
    Verbose=true);

% Train the network

doTraining = true;

if doTraining
    [net,info] = trainNetwork(dsTrain,layers,options);
    modelDateTime = string(datetime("now",Format="yyyy-MM-dd-HH-mm-ss"));
    save("trainedJPEGDnCNN-"+modelDateTime+".mat","net");
end

return
```
If you are using a single GPU machine, change this line:
```bash
ExecutionEnvironment='multi-gpu'
```
into this:
```bash
ExecutionEnvironment='gpu'
```


#### Subsequent uses of this Script <a name="1.3.1"></a>

The script uses the following booleans to allow you to skip various steps.

Once you have downloaded the data, change this line:
```bash
isDownloaded = false;
```
to this:
```bash
isDownloaded = true;
```

Once you have compressed the data, change this line:
```bash
isCompressed = false;
```
to this:
```bash
isCompressed = true;
```

### Train Interactively <a name="1.4"></a>

To train the network interactively, run the script from within the interactive MATLAB session's Command Window
```bash
>> deep_learning_example
```

In the MATLAB script, you can change the 'Plotting' option to view a plot of the training status in real time.
However, this uses massive amounts of RAM and should not be run for more than 3 or 4 epochs.
You can use the plotting as verification that a model is actually training, but you should not attempt to fully train a model with the plotting enabled.

Downloading and compressing the images can take quite a few minutes.
Training the model takes up to 9 hours to complete 10 epochs on the dual GPU machines.
Due to the way MATLAB trains a network with this depth (20 convolutional layers by default), these machines run out memory when training the network past 12 epochs with the rest of the settings left unchanged.
One way to reduce memory usage and training time is top use fewer compressed images in the training set.
Another option is to use a different layer setup with fewer hidden layers.

## Train CNN via Scheduled Job <a name="2"></a>

Since training this CNN can take many hours, it is a good idea to take advantage of the Slurm scheduler.
Interactive jobs can be stopped and interrupted due to internet connection issues.
In the options of the CNN itself we set `Verbose=true`, which will allow us to see the status of the model being trained in an ouput file that is updated in real time.


### Slurm Scripts <a name="2.1"></a>

It is reccomended that you use the dual GPU machines.
When using either of the scripts below, ensure the `ExecutionEnvironment` option in the above MATLAB script matches your choice of machine.

#### Single GPU <a name="2.1.1"></a>

To request a job with a single GPU, create the following script with the name 'dncnn_single_gpu.sh':

```bash
#!/bin/bash

#SBATCH --job-name DnCNN_singleGPU
#SBATCH --mail-user jmccullough12@unm.edu
#SBATCH --mail-type ALL
#SBATCH --output dncnn_single_gpu.out
#SBATCH --error dncnn_single_gpu.err
#SBATCH --time 48:00:00
#SBATCH --partition singleGPU
#SBATCH --ntasks 1
#SBATCH --mem 0
#SBATCH --cpus-per-task 16
#SBATCH -G 1

cd ~/deepLearningExample

module load matlab
matlab -nodisplay -r deep_learning_example > dncnn_single_training.out

```

#### Dual GPU <a name="2.1.2"></a>

To request a job with dual GPUs, create the following script with the name 'dncnn_dual_gpu.sh':

```bash
#!/bin/bash

#SBATCH --job-name DnCNN_DualGPU
#SBATCH --mail-user <YOUR EMAIL>
#SBATCH --mail-type ALL
#SBATCH --output dncnn_dual_gpu.out
#SBATCH --error dncnn_dual_gpu.err
#SBATCH --time 48:00:00
#SBATCH --partition dualGPU
#SBATCH --ntasks 1
#SBATCH --mem 0
#SBATCH --cpus-per-task 16
#SBATCH -G 2

cd ~/deepLearningExample

module load matlab
matlab -nodisplay -r deep_learning_example > dncnn_dual_training.out
```

### Submit Script <a name="2.2"></a>

To submit a job request using the single GPU script, use the following command:
```bash
hopper:~$ sbatch dncnn_single_gpu.sh
```

To submit a job request using the dual GPU script, use the following command:
```bash
hopper:~$ sbatch dncnn_dual_gpu.sh
```

### View Results <a name="2.3"></a>

While a network is being trained, you can see the results in real time with the `cat` command.

If you used the single GPU slurm script, use this command to view the output:
```bash
hopper:~$ cat ~/deepLearningExamples/dncnn_single_training.out
```

If you used the dual GPU slurm script, use this command to view the output:
```bash
hopper:~$ cat ~/deepLearningExamples/dncnn_dual_training.out
```

## Test the Model <a name="3"></a>

Once a model has finished training, it will be saved to a .mat file with a name like: `trainedJPEGDnCNNyyyy-MM-dd-HH-mm-ss.mat`
You can also use the pretrained example model (see the above instructions to get the example MATALB function files).

### MATLAB Script <a name="3.1"></a>

Use the following MATLAB Script to the results of a trianed model.
To specify which model to use, replace the `<MODEL FILE>` with the name of your model.

If the script is run interactively with X11 fowarding (see above instructions), an image will appear that shows the predictions made on a test image,.
The resulting comparisons are also saved to an imaged titled 'results.tif' which can be viewed with your preferred image viewer.

Create the following script with the name 'test_model.m' and ensure it lives in the 'deepLearningExample' directory.
```bash
% Open and test results of trained CNN model for JPEG Denoising

% Load the model from file

load("trainedJPEGDnCNN-2022-05-23-00-37-23.mat");

% Open test images

fileNames = ["sherlock.jpg","peacock.jpg","fabric.png","greens.jpg", ...
    "hands1.jpg","kobi.png","lighthouse.png","office_4.jpg", ...
    "onion.png","pears.png","yellowlily.jpg","indiancorn.jpg", ...
    "flamingos.jpg","sevilla.jpg","llama.jpg","parkavenue.jpg", ...
    "strawberries.jpg","trailer.jpg","wagon.jpg","football.jpg"];
filePath = fullfile(matlabroot,"toolbox","images","imdata")+filesep;
filePathNames = strcat(filePath,fileNames);
testImages = imageDatastore(filePathNames);


% Select an image to view - choose an image name from the above list.

testImage = "lighthouse.png";
Ireference = imread(testImage);

% Compress the test image in three different levels of quality

imwrite(Ireference,fullfile(tempdir,"testQuality10.jpg"),"Quality",10);
imwrite(Ireference,fullfile(tempdir,"testQuality20.jpg"),"Quality",20);
imwrite(Ireference,fullfile(tempdir,"testQuality50.jpg"),"Quality",50);

I10 = imread(fullfile(tempdir,"testQuality10.jpg"));
I20 = imread(fullfile(tempdir,"testQuality20.jpg"));
I50 = imread(fullfile(tempdir,"testQuality50.jpg"));

I10ycbcr = rgb2ycbcr(I10);
I20ycbcr = rgb2ycbcr(I20);
I50ycbcr = rgb2ycbcr(I50);

% Apply network to compressed test images

I10y_predicted = denoiseImage(I10ycbcr(:,:,1),net);
I20y_predicted = denoiseImage(I20ycbcr(:,:,1),net);
I50y_predicted = denoiseImage(I50ycbcr(:,:,1),net);

I10ycbcr_predicted = cat(3,I10y_predicted,I10ycbcr(:,:,2:3));
I20ycbcr_predicted = cat(3,I20y_predicted,I20ycbcr(:,:,2:3));
I50ycbcr_predicted = cat(3,I50y_predicted,I50ycbcr(:,:,2:3));

I10_predicted = ycbcr2rgb(I10ycbcr_predicted);
I20_predicted = ycbcr2rgb(I20ycbcr_predicted);
I50_predicted = ycbcr2rgb(I50ycbcr_predicted);

% View and save results

montage({I50,I20,I10,I50_predicted,I20_predicted,I10_predicted},Size=[2 3])
title("Compressed Images (above) Compared to Deblocked Images (below) with Quality Factor 50, 20 and 10 (Left to Right)")

imwrite(getframe(gca).cdata,'results.tif','tif');

% Change the following boolean to look closer at a specific region of
% interes in the test image

doROI = false;

if doROI
    roi = [30 440 100 80];
    i10 = imcrop(I10,roi);
    i20 = imcrop(I20,roi);
    i50 = imcrop(I50,roi);
    i10predicted = imcrop(I10_predicted,roi);
    i20predicted = imcrop(I20_predicted,roi);
    i50predicted = imcrop(I50_predicted,roi);
    montage({i50,i20,i10,i50predicted,i20predicted,i10predicted},Size=[2 3])
    title("Compressed Images ROI (above) Compared to Deblocked Images ROI (below) with Quality Factor 50, 20 and 10 (Left to Right)")
    imwrite(getframe(gca).cdata,'results_roi.tif','tif');
end


```
### Changes you can make to the test_model.m script <a name="3.2"></a>

The above script can be modified for two different kinds of functionality:

1. Use a different test image
2. Zoom in on a specific region of interest (ROI)

#### Different test image <a name="3.2.1"></a>
You can change the test image by changing the line: 
```bash
testImage = "lighthouse.png";
```
to any of the images in the list of filenames, directly above in the script.

#### Zoom in on specific ROI <a name="3.2.2"></a>
To zoom in on a specific region of interest, change the following line:
```bash
doROI = false;
```
to
```bash
doROI = true;
```

Then ROI can be changed by modifying this line:
```bash
roi = [30 440 100 80];
```

This region works well when using `lighthouse.png` as your test image.

Running the script with the boolean changed will display and save a image of the results zoomed in on the ROI.
If you are using X11 forwarding, the image should appear on your display.
The results are also saved to an image ('results_roi.tif') that can be viewed in your choice of image viewer.

<p class="carc-provenance" markdown>Migrated from [UNM-CARC QuickBytes](https://github.com/UNM-CARC/QuickBytes/blob/master/MATLAB%20Deep%20Learning%20on%20Xena.md){target=_blank} (last source update 2022-05-25). Spotted a problem? [Open an issue or pull request](https://github.com/UNM-CARC/QuickBytes){target=_blank}.</p>

---8<--- https://unm-carc.github.io/docs/software/matlab-gpu/

---
title: "MATLAB on GPUs"
description: "Accelerate MATLAB computations with GPUs on CARC clusters."
type: Guide
tags:
  - MATLAB
  - GPU
status: draft
generated:
  by: "claude/fable-5"
  at: "2026-08-29T00:00:00Z"
sources:
  - id: quickbytes
    resource: "https://github.com/UNM-CARC/QuickBytes/blob/master/Using%20GPUs%20on%20Xena%20with%20MATLAB.md"
    title: "UNM-CARC QuickBytes: Using GPUs on Xena with MATLAB.md"
    author: "team:unm-carc"
    last_modified: "2022-05-25T09:46:08-06:00"
---

# MATLAB on GPUs

1. [Using a single GPU](#1)
     1. [Use GPU in Interactive Session](#1.1)
          1. [Identify and Select GPU](#1.1.1)
          2. [Using Arrays on GPU](#1.1.2)
          3. [Initialize Array](#1.1.3)
          4. [Test if array is on GPU](#1.1.4)
          5. [Retrieve Array from GPU](#1.1.5)
          6. [Use functions on GPU Arrays](#1.1.6)
     2. [Schedule a job](#1.2)
          1. [MATLAB Script](#1.2.1)
          2. [PBS Script](#1.2.2)
          3. [Slurm Script](#1.2.3)
          4. [Submit Job to Queue](#1.2.4)
2. [Using multiple GPUs on a single node](#2)
    1. [MATLAB Script](#2.2)
    2. [Slurm Script](#2.2)
    3. [Submit Job to Queue](#2.3) 
3. [Using Multiple Nodes with their own GPUs](#3)


## Using a single GPU <a name="1"></a>

MATLAB allows the utilization of a single GPU that is part of a machine.
The following sections show how to access and utilize a GPU.

### Use GPU in Interactive Session <a name="1.1"></a>

First, we will open MATLAB in an interactive session on a GPU compute node.

#### Identify and Select GPU <a name="1.1.1"></a>

Start by requesting an interactive session:

```bash
hopper:~$ srun -G 1 --pty bash
```

Once you have a node allocated to you, load the MATLAB module and start a MATLAB session:

```bash
xena01:~$ module load matlab
xena01:~$ matlab

To get started, type doc.
For product information, visit www.mathworks.com.
>>
```

Now you can check to see the number of GPUs available:
```bash
>> gpuDeviceCount("available")
```
You should see the following:
```bash
ans = 
     1
```
This means that you have access to a single GPU.

To get information about the available gpus, use this function:
```bash
>> gpuDeviceTable
```
That will print something like this:
```bash
ans =

  1x5 table

    Index        Name        ComputeCapability    DeviceAvailable    DeviceSelected
    _____    ____________    _________________    _______________    ______________

      1      "Tesla K40m"          "3.5"               true              false
```

Next, you can tell MATLAB which GPU to use (pass in the desired index from the above table).
If you do not do this, MATLAB will automatically grab the lowest index GPU when you try to use one.
```bash
>> gpuDevice(1)
```

Running the `gpuDeviceTable` command again shows this change:
```bash
>> gpuDeviceTable

ans =

  1x5 table

    Index        Name        ComputeCapability    DeviceAvailable    DeviceSelected
    _____    ____________    _________________    _______________    ______________

      1      "Tesla K40m"          "3.5"               true              true

```

#### Using Arrays on GPU <a name="1.1.2"></a>

In order to utilize the GPU, data must be loaded into a `gpuArray` object.
For a full description of the `gpuArray` object, please visit the official MathWorks Documentation at [https://www.mathworks.com/help/parallel-computing/gpuarray.html](https://www.mathworks.com/help/parallel-computing/gpuarray.html){target=_blank}

##### Initialize Array <a name="1.1.3"></a>
First, create a normal array using any method you like.
In this example we will use the `magic(8)` function to create a magic square matrix that is 8x8.
```bash
>> A = magic(8)
```
Next, pass that into a `gpuArray` object.
This will copy the contents of a normal array into an array on the GPU.
```bash
>> B = gpuArray(A)
```

##### Test if array is on GPU <a name="1.1.4"></a>
The `isgpuarray` function tests if an array is on a GPU:
```bash
>> isgpuarray(A)

ans =

  logical

   0

>> isgpuarray(B)

ans =

  logical

   1
```
This confirms that array A is not on the GPU, but array B is.

##### Retrieve Array from GPU <a name="1.1.5"></a>
In order to retrieve an array from the GPU and put it back in the MATLAB workspace, use the `gather` function.
It will copy the contents of an array on the GPU into a normal array.
This is neccesary if you want to to perform non-GPU actions on your data after using the GPU.
```bash
>> C = gather(B)
```
Now, we can test to see if C is stored on the gpu:
```bash
>> isgpuarray(C)

ans =

  logical

   0
```

#### Use functions on GPU Arrays <a name="1.1.6"></a>
To perform functions on `gpuArray` objects, use the `arrayfun` function.
In this example, we will apply the MATLAB `sqrt` function to the array (B) that we created in the previous step:
```bash
result = arrayfun(@sqrt,B)
```
This will apply the `sqrt` function to every element in the GPU array.
`result` is also a GPU array:
```bash
>> isgpuarray(result)

ans =

  logical

   1

```

To see a list of MATLAB functions that are supported using gpus, visit [https://www.mathworks.com/help/parallel-computing/gpuarray.html](https://www.mathworks.com/help/parallel-computing/gpuarray.html){target=_blank}

You can also create your own functions to pass into `arrayfun`.

### Schedule a job <a name="1.2"></a>
It is good idea to do everything using a batch script and avoid the mistakes associated with interactive computing.
To get an idea of why performing functions on `gpuArray` objects is a good idea, let's create a simple MATLAB script that displays the amount of time it takes to perform the same computation on a cpu and on a gpu.
We will then create a PBS script that schedules a job with a GPU to run the MATLAB script for us.


#### MATLAB Script <a name="1.2.1"></a>
We will perform the `sqrt` function on a 5000x5000 array.
The use of `tic` and `toc` allow us to time the seperate applications of `sqrt`.

Create the following script with the name `gpu_matlab.m`:

```bash 
gpuDevice(1);

A = magic(5000);
disp("sqrt of 5000x5000 matrix on cpu:")
tic
B = arrayfun(@sqrt, A);
toc
disp("sqrt of 5000x5000 matrix on gpu:")
C = gpuArray(magic(5000));
tic
D = arrayfun(@sqrt,C);
toc
```
#### PBS Script <a name="1.2.2"></a>

Now, let's create a PBS script called `gpu_matlab.pbs`. 
Replace the `<DIR>` with the path to the directory containing the MATLAB script created above. 
This script will request the desired resrouces, load the MATLAB module, then run the script.
The output of the script will be sent to the file: `gpu_matlab.out`

```bash
#!/bin/bash

#PBS -N gpu_test
#PBS -l walltime=00:05:00
#PBS -l nodes=1:ppn=1:gpus=1
#PBS -j oe

cd <DIR>

module load matlab

matlab -nodisplay -r gpu_matlab > gpu_matlab.out

```

#### Slurm Script <a name="1.2.3"></a>

Now, let's create a Slurm script called `gpu_matlab.sh`. 
Replace the `<DIR>` with the path to the directory containing the MATLAB script created above. 
This script will request the desired resrouces, load the MATLAB module, then run the script.
The output of the script will be sent to the file: `gpu_matlab.out`

```bash
#!/bin/bash

#SBATCH --job-name gpu_matlab_job
#SBATCH --output gpu_matlab_job.out
#SBATCH --error gpu_matlab_job.err
#SBATCH --time 00:05:00
#SBATCH --ntasks 1
#SBATCH -G 1

cd <DIR>

module load matlab
matlab -nodisplay -r gpu_matlab > gpu_matlab.out
```


#### Submit Job to Queue <a name="1.2.4"></a>

Now we can submit the job to the scheduler from the head node:

```bash
hopper:~$ sbatch gpu_matlab.sh
```

View the results:
```bash
hopper:~$ cat gpu_matlab.out
```

## Using multiple GPUs on a single node <a name="2"></a>

Some CARC nodes contain multiple GPUs.
MATLAB allows for the utilization of multiple GPUs on a single node in the same way you use multiple CPUs.
To show how this works, below is an example MATLAB script that will create a logistic map using all available GPU's on the assigned node.

The `parpool` object is used to create workers to parallelize the execution.
Each worker will grab it's own GPU when performing actions with `gpuArray` objects.
For this to work properly, ensure that you have been allocated an equal number of CPUs as GPUs on the machine.
An example slurm script is included below to give an idea of how to ask for the proper resources to be allocated.

### MATLAB Script <a name="2.1"></a>

Create the following MATLAB script called `gpu_logistic_map.m`
This simple MATLAB script creates a Logistic Map by iterating the logistic equation on a set of random populations.
A worker is created for each available GPU.
They will then split up the work performed in the `parfor` loop.
The result is a logistic map figure saved as 'logistic_map.jpg'
It also contains calls to time the execution of the `parfor` loop.


```bash

N = 1000;
r = gpuArray.linspace(0,4,N);

numIterations = 1000;

numGPUs = gpuDeviceCount("available");
parpool(numGPUs);

numSimulations = 100;
X = zeros(numSimulations,N,'gpuArray');

disp("Timing execution of parfor loop:")

tic
parfor i=1:numSimulations
  X(i,:) = rand(1,N,'gpuArray')
  for n=1:numIterations
    X(i,:) = r.*X(i,:).*(1-X(i,:));
  end
end
toc

f = figure('visible','off');
plot(r,X,'.');
saveas(f,'logistic_map','jpg')

return
```

### Slurm Script <a name="2.2"></a>


When requesting two GPUs, you must also set `--cpus-per-task 2` and `-G 2` for MATLAB to correctly find and utilize the available GPUs.
These numbers should match, as MATLAB will use a CPU to access each GPU.
We ask for two CPUs and two GPUs.

Create the following slurm scrpt called `gpu_logistic_map.sh`.
Replace the `<DIR>` with the path to the directory containing the MATLAB script created above. 
This script will ask the scheduler for the proper resources.
Once the resrouces are allocated, the script will run the MATLAB script from the above step.
The MATLAB script will create a .jpg image once it has finished.
Any output of the MATLAB script is redirected to `gpu_logistic_map.out`.



```bash
#!/bin/bash

#SBATCH --job-name gpu_logistic_map_job
#SBATCH --output gpu_logistic_map_job.out
#SBATCH --error gpu_logistic_map_job.err
#SBATCH --time 00:10:00
#SBATCH --ntasks 1
#SBATCH --cpus-per-task 2
#SBATCH -G 2

cd <DIR>

module load matlab
matlab -nodisplay -r gpu_logistic_map > gpu_logistic_map.out

```

### Submit Job to Queue <a name="2.3"></a>

Now we can submit the job to the scheduler from the head node:
```bash
hopper:~$ sbatch gpu_logistic_map.sh
```
View the results:
```bash
hopper:~$ cat gpu_logistic_map.out
```

You can also view the `logistic_map.jpg` image using your preferred method.


## Using Multiple Nodes with their own GPUs <a name="3"></a>

Coming Soon! (Maybe)

<p class="carc-provenance" markdown>Migrated from [UNM-CARC QuickBytes](https://github.com/UNM-CARC/QuickBytes/blob/master/Using%20GPUs%20on%20Xena%20with%20MATLAB.md){target=_blank} (last source update 2022-05-25). Spotted a problem? [Open an issue or pull request](https://github.com/UNM-CARC/QuickBytes){target=_blank}.</p>

---8<--- https://unm-carc.github.io/docs/software/matlab-jobs/

---
title: "Running MATLAB jobs"
description: "Run MATLAB non-interactively in Slurm batch jobs on CARC clusters."
type: Guide
tags:
  - MATLAB
  - Jobs
generated:
  by: "claude/fable-5"
  at: "2026-08-29T00:00:00Z"
sources:
  - id: quickbytes
    resource: "https://github.com/UNM-CARC/QuickBytes/blob/master/running_matlab_jobs.md"
    title: "UNM-CARC QuickBytes: running_matlab_jobs.md"
    author: "team:unm-carc"
    last_modified: "2026-07-07T11:39:36-06:00"
---

# Running MATLAB jobs

## MATLAB

MATLAB is a software environment and programming language designed primarily for numerical computing and matrix operations—hence the name MATrix LABoratory.

If you are unfamiliar with MATLAB, visit the MathWorks website for additional documentation and tutorials:

https://www.mathworks.com/products/matlab.html

To view available MATLAB versions on a CARC system, run:

```bash id="m1q8aa"
module avail matlab
```

MATLAB is installed on CARC systems, but must be run on compute nodes rather than login (head) nodes.

To run MATLAB jobs at CARC, submit a Slurm job that launches MATLAB in batch mode.

---

## Batch Mode

Many users are familiar with running MATLAB through the graphical interface or interactive console.

At CARC, MATLAB programs must be run non-interactively in batch mode.

Batch mode allows MATLAB to execute a script and exit automatically without launching the graphical interface.

---

## Creating a Simple MATLAB Program

Suppose you have a MATLAB script named `my_program.m` that generates a 3×3 matrix of random numbers and writes the results to a CSV file.

Example:

```matlab id="k8x2cn"
% Generate a random 3x3 matrix
rmatrix = rand(3);

% Output filename
fname = 'randnums.csv';

% Write results
writematrix(rmatrix, fname);

exit
```

Save this file in your home directory.

---

## Running MATLAB on a Compute Node (Interactive)

If you want to test interactively, first request a compute node:

```bash id="q7v1sp"
srun --pty bash
```

Once on the compute node:

```bash id="p0d8xy"
module load matlab
matlab -batch "my_program"
```

This ensures MATLAB runs on a compute node, not the login node.

---

## Running MATLAB from the Command Line (Non-Interactive)

Rather than requesting an interactive shell first, you can launch MATLAB on a compute node directly from the login node with a single command. Load the module in your login shell first — `srun` propagates your current environment to the compute node, so the module needs to already be loaded before you call `srun`, not after:

```bash id="v3k8lp"
module load matlab
```

```bash id="v3k9lm"
srun --partition general matlab -batch "my_program"
```

Notes:

* The `.m` extension is omitted
* `-batch` runs the script and exits automatically
* This is the recommended method for quick one-off runs; for anything long enough to need `--time`, `--mem`, or email notifications, use the `sbatch` submission below instead

---

## Submitting a MATLAB Job with Slurm

Create a submission script named `my_matlab_job.sbatch`.

```bash id="x2n5ad"
#!/bin/bash

#SBATCH --job-name=my_matlab_job
#SBATCH --time=01:00:00
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --mail-type=END,FAIL
#SBATCH --mail-user=my_email@unm.edu
#SBATCH --output=slurm-%j.out

# Change to the directory that you submitted your Slurm script from.
# Without this, relative-path output files (like randnums.csv here)
# may not end up where you expect, or may not get written at all.
cd "$SLURM_SUBMIT_DIR"

module load matlab

matlab -batch "my_program"
```

Submit the job from your home directory:

```bash id="c4w8pq"
sbatch my_matlab_job.sbatch
```

---

## Important Notes

* MATLAB jobs must run on compute nodes, not login nodes.
* Do not run `.m` files directly with `srun`; they must be executed through MATLAB.
* Use `srun --pty bash` for interactive compute access when debugging.
* Easley uses Slurm. PBS may still exist on Hopper, but Slurm is recommended for all workflows.

---

## Viewing Output

All output is written to:

```text id="z1r8tt"
slurm-<jobid>.out
```

Monitor output live with:

```bash id="t6v9gh"
tail -f slurm-<jobid>.out
```

---

## Commands Used (for Debugging)

```bash id="a8m3kf"
module avail matlab
```

List available MATLAB versions.

```bash id="k2v9sd"
srun --pty bash
```

Request an interactive compute node session.

```bash id="u7p1wx"
module load matlab
```

Load MATLAB module.

```bash id="m9c2qa"
matlab -batch "my_program"
```

Run MATLAB script in batch mode.

```bash id="b5n7ld"
sbatch my_matlab_job.sbatch
```

Submit job to Slurm.

```bash id="r3q8yt"
tail -f slurm-<jobid>.out
```

Monitor job output.

*This QuickByte was validated on 6/23/2026*

<p class="carc-provenance" markdown>Migrated from [UNM-CARC QuickBytes](https://github.com/UNM-CARC/QuickBytes/blob/master/running_matlab_jobs.md){target=_blank} (last source update 2026-07-07). Spotted a problem? [Open an issue or pull request](https://github.com/UNM-CARC/QuickBytes){target=_blank}.</p>

---8<--- https://unm-carc.github.io/docs/software/matlab-parallel-server/

---
title: "MATLAB Parallel Server"
description: "Use MATLAB Parallel Server to scale parpool jobs across multiple nodes."
type: Guide
tags:
  - MATLAB
  - Parallel
generated:
  by: "claude/fable-5"
  at: "2026-08-29T00:00:00Z"
sources:
  - id: quickbytes
    resource: "https://github.com/UNM-CARC/QuickBytes/blob/master/ParallelMatlabServer.md"
    title: "UNM-CARC QuickBytes: ParallelMatlabServer.md"
    author: "team:unm-carc"
    last_modified: "2021-02-24T17:43:53-07:00"
---

# MATLAB Parallel Server

MATLAB supports parallelization on desktop computers which can be used to increase the speed of analysis drastically. MATLAB also provides the MATLAB Parallel Server (previously the MATLAB Distributed Computing Server) which allows you to write MATLAB code on your local desktop or laptop computer and perform the computation using the CARC high-performance clusters. This QuickByte leads you through the steps needed to set this up. If you run into problems please send an email to help@carc.unm.edu and we will be happy to help.

Please ensure you have the MATLAB Parallel Toolbox installed on your local comnputer.

## How MATLAB Parallel Works Behind the Scenes
MATLAB parallel allows the MATLAB session you interact with on your local computer, also known as the MATLAB client, with the PBS scheduler at CARC to create jobs that run on a core, also known as the MATLAB worker. The PBS (Portable BAtch system) scheduler allocates resources requested to users. One of the advantages of using MATLAB at CARC is the ability to scale up, or use many nodes for data intensive and/or computationally complex computations. To do this, you will request more workers from the PBS scheduler by following the tutorial bellow (Workers window). If you request multiple workers, it is important to keep in mind that one worker will be running the batch script you have sent from your MATLAB client. This worker is sending your scripts to the other workers that will perform your computations. You can think about this lead MATLAB worker as a mirror of your MATLAB client that communicates with the PBS scheduler and your scripts to accomplish your job. For more help on how to alter your scripts to take advantage of the scaleing up abilities at CARC please visit the Mathworks [tutorials](https://www.mathworks.com/help/parallel-computing/what-is-parallel-computing.html){target=_blank}.


### MATLAB Parallel Server Client Configuration

Once MATLAB is installed on your local machine (the MATLAB version on your local machine must match the version on the CARC cluster) click "add-ons" to open Add-on explorer. Search for PBS. 

Click on the link "Parallel Computing Toolbox plugin for MATLAB Parallel Server with PBS" (there is a plugin for slurm as well).

Click the "install" button and the plugin will install and a wizard is started.

![Install](../assets/images/quickbytes/ParallelMatlabInstall.png)

![Wizard1](../assets/images/quickbytes/ParallelMatlabWizard1.png)
Choose UNIX in the cluster type

![Wizard2](../assets/images/quickbytes/ParallelMatlabWizard2.png)

Select no for shared job location.

![Wizard3](../assets/images/quickbytes/ParallelMatlabWizard3.png)

Enter the address of the cluster you would like to use: for example, hopper.alliance.unm.edu.

Enter </users/> for the path to the PBS scripts (remote job storage location) that MATLAB will create on the cluster.

Select unique subfolders.
  
![Wizard4](../assets/images/quickbytes/ParallelMatlabWizard4.png)

Select the number of workers and number of threads per worker. This may depend on the program you are running but in general you should have one worker per core on the cluster. For set up and validation leave the number of workers at 1. Leave the threads per worker at 1 unless your software requires more threads.

Specify the path to the MATLAB installation on the compute nodes (shown by `module show matlab` on the cluster). It is important that you are running the same version of MATLAB as you are 
running on the cluster. 

![Wizard6](../assets/images/quickbytes/ParallelMatlabWizard6.png)

Choose flexnet for the license

![Wizard7](../assets/images/quickbytes/ParallelMatlabWizard7.png)

Name your profile. For example "R2019a_Wheeler_PBS"

![Wizard8](../assets/images/quickbytes/ParallelMatlabWizard8.png)

Review your profile settings and create the profile.

![Wizard10](../assets/images/quickbytes/ParallelMatlabWizard10.png)

You can create multiple profiles for different CARC clusters and numbers of workers.

```matlab
parallel.cluster.generic.runProfileWizard()
```

## Setting your IP Address

In the next steps MATLAB will need to know your local IP address to allow incoming/outgoing connections on your computer.

You will likely have to tell the system the IP address or hostname of your local machine. This is so the CARC cluster can communicate with your laptop or desktop. You will set the hostname with pctconfig (Parallel Config Toolbox). You can either type the hostname in directly or attempt to have MATLAB find it for you with the following commands:

OS X
```
[~,name]=system('ipconfig getifaddr en0');
pctconfig('hostname',name);
```

Linux
```
%%
[~,name]=system('hostname -i');
pctconfig('hostname',name);
```

Windows 10

Look up your computers IP address and enter:

```
%%
pctconfig('hostname',"<your IP address>");
```


### Validating the Configuration

Select "parallel" then create/manage clusters. 

Choose the profile you just created. In this example, the profile name is "R2019a_Wheeler_PBS"

![Validating1](../assets/images/quickbytes/ParallelMatlabValidate1.png)

Select the profile you just created and select the validation tab. Press the validate button.

![Validating2](../assets/images/quickbytes/ParallelMatlabValidate2.png)

It will ask for your CARC username, and here you can also select your ssh keyfile if you use one (cmd+shift+. to reveal hidden directories so you can see you ~/.ssh folder and select your private key), or just enter your password.

MATLAB will now validate your setup. If you run into trouble please contact CARC support at help@carc.unm.edu.

This completes the install and configuration.

We have found it best to restart MATLAB at this point, otherwise setting the hostname in the next step may not work.

### Writing Parallel Matlab Code


Once you have set the hostname you can run your parallel MATLAB code. Mathworks provides extensive documentation on using parallelism in MATLAB: [Mathworks Docs](https://www.mathworks.com/help/parallel-computing/getting-started-with-parallel-computing-toolbox.html){target=_blank}

The very simple program that follows demonstrates how to run parallel code using MATLAB. The code uses parfor, which can often be used to replace a for loop in serial MATLAB code, to distibute the work across 10 parallel workers.

```
n_workers = 10;                  % We will request 10 workers to run in parallel
p = parpool('R2019a_Wheeler_PBS', n_workers); % Create the pool of workers using the profile created earlier with 10 workers.
parfor i = 1:100                % Define a parallel loop that will be distributed accross the 10 workers.
i                               % Print the value of i for this iteration.
end                   
delete(p);                      % Clean up the worker pool
```

## Monitoring MATLAB Jobs
To check that your jobs are indeed running at CARC, you can log in (ssh) to the cluster you have submitted your job to and check your job status. The command bellow shows only your jobs. 

```
ssh username@hopper.alliance.unm.edu 

qstat -u <username>
```

If you are testing small scripts, they may run before you can type qstat. To watch your jobs, you can type watch before the qstat (or anyother) command and it will re-run the command every 2sec. This is a good way to watch progress.

<p class="carc-provenance" markdown>Migrated from [UNM-CARC QuickBytes](https://github.com/UNM-CARC/QuickBytes/blob/master/ParallelMatlabServer.md){target=_blank} (last source update 2021-02-24). Spotted a problem? [Open an issue or pull request](https://github.com/UNM-CARC/QuickBytes){target=_blank}.</p>

---8<--- https://unm-carc.github.io/docs/software/parallel-matlab/

---
title: "Parallel MATLAB: profile setup and batch submission"
description: "Configure a cluster profile and submit parallel MATLAB jobs."
type: Guide
tags:
  - MATLAB
  - Parallel
generated:
  by: "claude/fable-5"
  at: "2026-08-29T00:00:00Z"
sources:
  - id: quickbytes
    resource: "https://github.com/UNM-CARC/QuickBytes/blob/master/Parallel%20MATLAB%20profile%20setup%20and%20batch%20submission.md"
    title: "UNM-CARC QuickBytes: Parallel MATLAB profile setup and batch submission.md"
    author: "team:unm-carc"
    last_modified: "2022-05-11T10:43:18-06:00"
---

# Parallel MATLAB: profile setup and batch submission

### Setting up the cluster profile
In order to submit a batch script that takes advantage of MATLAB Parallel Server you first need to set up a cluster profile. Thankfully this has already been done and all you need to do as a user is import the profile and that only needs to be done once. If you would like to do this interactively you can start an interactive session with the following:

```bash
hopper:~$ srun --pty bash
```
Once you have a node allocated to you load the MATLAB module and start a MATLAB session:

```bash
wheeler001:~$ module load matlab
wheeler001:~$ matlab

To get started, type doc.
For product information, visit www.mathworks.com.
>>
```
Now simply import the cluster profile available in the root MATLAB folder:

```
>> profile = parallel.importProfile('/opt/local/MATLAB/<cluster>-normal.settings')
```
With the settings imported you can now launch parallel pools for computation using the imported cluster profile. The code below is an example to test parallel computing across two nodes while timing execution:

```
>> poolobj = parpool(profile, 16)
>> tic
>> n = 200
>> A = 500
>> a = zeros(1,n)
>> parfor i = 1:n
>> a(i) = max(abs(eig(rand(A))))
>> end % You may need to hit enter more than once to get the prompt back.
>> toc
>> delete(poolobj);
```
Even better is to do everything using a batch script and avoid the mistakes associated with interactive computing. Below is an example MATLAB script named `parallel_matlab.m` that will import our cluster profile and compare the time of computation for a sequential for loop and a parallel for loop with 16 cores ('workers' in MATLAB speak):

```
profile = parallel.importProfile('/opt/local/MATLAB/<cluster>-normal.settings')

poolobj = parpool(profile, 16);

tic
n = 200;
A = 500;
a = zeros(1,n);
for i=1:n;
    a(i) = max(abs(eig(rand(A))))
end
toc

tic
n = 200;
A = 500;
a = zeros(1,n);
parfor i=1:n;
    a(i) = max(abs(eig(rand(A))))
end
toc
delete(poolobj);
```
Now the PBS script we will call `parallel_matlab.pbs` to submit your sample MATLAB program:

```
#!/bin/bash

#PBS -N parallel_matlab
#PBS -l walltime=01:00:00
#PBS -l nodes=1:ppn=8
#PBS -j oe

cd #PBS_O_WORKDIR

module load matlab/R2019a

matlab -r -nodisplay parallel_matlab > parallel_matlab.out
```
Submit your PBS script with `qsub parallel_matlab.pbs` and hopefully all goes swimmingly. If you require assistance with MATLAB parallel computing please send an email to help@carc.unm.edu.

<p class="carc-provenance" markdown>Migrated from [UNM-CARC QuickBytes](https://github.com/UNM-CARC/QuickBytes/blob/master/Parallel%20MATLAB%20profile%20setup%20and%20batch%20submission.md){target=_blank} (last source update 2022-05-11). Spotted a problem? [Open an issue or pull request](https://github.com/UNM-CARC/QuickBytes){target=_blank}.</p>

---8<--- https://unm-carc.github.io/docs/software/parallel-r-future/

---
title: "Parallel R with the future package"
description: "Parallelize R code across cores and nodes using the future framework."
type: Tutorial
tags:
  - R
  - Parallel
generated:
  by: "claude/fable-5"
  at: "2026-08-29T00:00:00Z"
sources:
  - id: quickbytes
    resource: "https://github.com/UNM-CARC/QuickBytes/blob/master/Parallel_R_with_Future.ipynb"
    title: "UNM-CARC QuickBytes: Parallel_R_with_Future.ipynb"
    author: "team:unm-carc"
    last_modified: "2021-08-11T10:26:45-06:00"
---

# Parallel R with the future package

In R, numerous packages can be used to parallelize code (parallel, snow, foreach, etc.), but each of  these packages use unique syntaxes and none of them work for all cases of parallelization. Instead, the [future package](https://github.com/HenrikBengtsson/future){target=_blank} solves this problem! Briefly (see the [future package](https://github.com/HenrikBengtsson/future){target=_blank} for details), the future package works in three steps:

1. Choose how you want to parallelize your code
    - set the type of parallelization with the `plan()` object.  
2. Choose which part of the code you would like to run in parallel
    - place the part of the code that will be iterated within the `future({})` object. 
3. Evaluate the code. 
    - run the iterations using the `value()` object. 

The power of the future package is it separates the planning for the parallelization (steps 1 and 2) and then executes the code afterwards. This allows the user to control how and where to parallelize their code. Thus, the framework can be extended to any iterative process.


Table of Contents

   - Packages
   - For-loop Example and basic structure
   - Tidyverse example using the furrr package
   - Bayesian example using multiple nodes via the future.batchtools package
   - Appendix: Testing speed of Future



## Packages 

Before you run the code, I recommend installing R 4.0 and IRKernal via conda. Once you activate your conda environment, feel free to run the rest through jupyter: 
the CARC JupyterHub on [Hopper](https://hopper.alliance.unm.edu){target=_blank} or [Easley](https://easley.alliance.unm.edu/jupyter){target=_blank}.

Final note:
make sure you are on a node with 8 cores on an interactive node or logged into Jupyter before running the code. 



```R
# Load miniconda
module load miniconda3-4.7.12.1-gcc-4.8.5-lmtvtik

# create r_parallel which installs 4
conda create -n r_parallel_tutorial r=4 r-irkernel -c conda-forge

# enables you do use conda activate
eval "$(conda shell.bash hook)"

conda activate r_parallel_tutorial
```


```R
# double check you have multiple cores available

install.packages(
    c("future", "batchtools", "future.batchtools", "repurrrsive",
      "purrr","dplyr","furrr","tidyr", "snow", "ggplot2",
      "tibble", "labeling", "farver","backports"), 
    repos='http://cran.us.r-project.org', 
    Ncpus = 8 )
```

    also installing the dependencies ‘colorspace’, ‘hms’, ‘prettyunits’, ‘munsell’, ‘RColorBrewer’, ‘viridisLite’, ‘globals’, ‘listenv’, ‘parallelly’, ‘base64url’, ‘brew’, ‘checkmate’, ‘data.table’, ‘fs’, ‘progress’, ‘R6’, ‘rappdirs’, ‘stringi’, ‘withr’, ‘magrittr’, ‘generics’, ‘tidyselect’, ‘cpp11’, ‘gtable’, ‘isoband’, ‘scales’, ‘pkgconfig’
    
    
    Updating HTML index of packages in '.Library'
    
    Making 'packages.html' ...
     done
    



```R
library(future) # needed for all examples
library(purrr) # needed for tidyvese example
library(dplyr) # needed for tidyverse example
library(furrr) # needed for tidyverse example
library(repurrrsive) # loads in data for tidyverse example
library(future.batchtools) # needed for bayesian example
library(tidyr) # needed for tidyverse example
library(snow) # needed to load Hmsc for bayesian example
library(ggplot2) # needed to plot functions
library(labeling) # needed to plot functions
library(farver) # needed to plot functions



# double-check you have access to mulitple cores 
availableCores()

# check that you can run multicore (will fail for R-studio and windows!)
supportsMulticore()
```

    
    Attaching package: ‘dplyr’
    
    
    The following objects are masked from ‘package:stats’:
    
```bash
filter, lag
```
    
    
    The following objects are masked from ‘package:base’:
    
```bash
intersect, setdiff, setequal, union
```
    
    



<strong>system:</strong> 8



TRUE



```R
sessionInfo()
```


    R version 4.1.0 (2021-05-18)
    Platform: x86_64-conda-linux-gnu (64-bit)
    Running under: CentOS Linux 7 (Core)
    
    Matrix products: default
    BLAS/LAPACK: /users/mimann/.conda/envs/r_parallel_tutorial/lib/libopenblasp-r0.3.17.so
    
    locale:
     [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
     [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
     [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
     [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
     [9] LC_ADDRESS=C               LC_TELEPHONE=C            
    [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       
    
    attached base packages:
    [1] stats     graphics  grDevices utils     datasets  methods   base     
    
    other attached packages:
     [1] farver_2.1.0             labeling_0.4.2           ggplot2_3.3.5           
     [4] snow_0.4-3               tidyr_1.1.3              future.batchtools_0.10.0
     [7] repurrrsive_1.0.0        furrr_0.2.3              dplyr_1.0.7             
    [10] purrr_0.3.4              future_1.21.0           
    
    loaded via a namespace (and not attached):
     [1] pillar_1.6.2      compiler_4.1.0    prettyunits_1.1.1 progress_1.2.2   
     [5] base64enc_0.1-3   tools_4.1.0       digest_0.6.27     uuid_0.1-4       
     [9] gtable_0.3.0      jsonlite_1.7.2    evaluate_0.14     lifecycle_1.0.0  
    [13] tibble_3.1.3      checkmate_2.0.0   pkgconfig_2.0.3   rlang_0.4.11     
    [17] IRdisplay_1.0     IRkernel_1.2      parallel_4.1.0    withr_2.4.2      
    [21] repr_1.1.3        rappdirs_0.3.3    hms_1.1.0         generics_0.1.0   
    [25] vctrs_0.3.8       globals_0.14.0    grid_4.1.0        tidyselect_1.1.1 
    [29] data.table_1.14.0 glue_1.4.2        listenv_0.8.0     R6_2.5.0         
    [33] fansi_0.4.2       parallelly_1.27.0 base64url_1.4     pbdZMQ_0.3-5     
    [37] magrittr_2.0.1    scales_1.1.1      backports_1.2.1   codetools_0.2-18 
    [41] batchtools_0.9.15 ellipsis_0.3.2    htmltools_0.5.1.1 colorspace_2.0-2 
    [45] brew_1.0-6        utf8_1.2.2        stringi_1.7.3     munsell_0.5.0    
    [49] crayon_1.4.1     


## 1. For-loop example and basic structure

This example shows how the future package can be used for running a for-loop that iterates over a very slow function.

### Setting up and running loop in serial
First, we will create a function that pauses for 0.5 seconds. 
At the end, we will print the length of time it took to run. 


```R
# custom function that waits a half a second and then prints current step value
slow_function <- function(step){
  
  # wait half a second
  Sys.sleep(.5)
    paste0("Step ", step, " completed")

}


#### running loop in serial ####
# timestamp before loop 
t1 <- proc.time()
# pre-allocate output 
output <- rep(NA, 50)

# for-loop
for (i in 1:50){
  output[i] <- slow_function(i)

}

# print the output for first 6 steps
head(output)

# timestampe after loop 
t2 <- proc.time()


print("Elapsed time (seconds) for for-loop")
# length of time
time_Elapsed <- t2[[3]] - t1[[3]]
round(time_Elapsed, 2)
```


<style>
.list-inline {list-style: none; margin:0; padding: 0}
.list-inline>li {display: inline-block}
.list-inline>li:not(:last-child)::after {content: "\00b7"; padding: 0 .5ex}
</style>
<ol class=list-inline><li>'Step 1 completed'</li><li>'Step 2 completed'</li><li>'Step 3 completed'</li><li>'Step 4 completed'</li><li>'Step 5 completed'</li><li>'Step 6 completed'</li></ol>



    [1] "Elapsed time (seconds) for for-loop"



25.06


###  Rerunning for-loop in parallel
Now we will re-write the for-loop so it is compatible with future. We will choose to run it across every core in the node we are using. 

WARNING: if you plan to run this example locally on R-studio, you will need to change it to plan(multisession). 

#### 1. Choose how you want to parallelize your code
using the `plan()` object set to "multicore", we will establish we want to use all the cores available on the node. 

We will set the number of iterations to run to 50  and pre-allocate space for the future loop by creating the "command_set" object.  Since future puts every iteratation in a list, we will use a vector to pre-allocate empty lists for every iteration.


```R
#### rerunning loop in parallel using all available cores ####
# going to send iterations to each cores
# to run it only on specified number of cores, add the argument, workers. 
plan(multicore)

iterations <- 50
# rewrite loop so it works with future:
# future requires the iterative step to be saved as its own list, 
# thus we will create a list object with 50 slots for each step. 
command_set <- vector(mode = "list", length = iterations)


```

#### 2. Choose which part of the code you would like to run in parallel
Here, we are rewriting the for-loop to work with future.

We will place the section we want to iterate within the curly-brackets of the `future()` object. The output is written to a separate list within the y object (it creates a list of lists). When you run this code, it doesn't evaluate the loop but sets up the environments for each iteration to run in parallel. Thus, the time elapsed is all the time needed to plan the code. 

WARNING: Setting up the lists prior to evaluation can slow down the parallelization effort (see appendix).


```R

t1 <- proc.time()
for (i in 1:iterations){

  command_set[[i]] <- future(
    # code for each iteration within the curly brackets
    {slow_function(i)}
  ) 
}


t2 <- proc.time()


print("Elapsed time (seconds) for planning command set")
# length of time
time_Elapsed <- t2[[3]] - t1[[3]]
round(time_Elapsed, 2)

```

    [1] "Elapsed time (seconds) for planning command set"



11.27


Before Future evaluates the R code, it first writes the global environment to the "command_set" object. Each iteration is one list, thus extracting the first list will display what it will do for the first step in the loop. 


```R
print("What one iteration looks like in the command set")
# prints first iteration
command_set[[1]]
```

    [1] "What one iteration looks like in the command set"



    MulticoreFuture:
    Label: ‘<none>’
    Expression:
    {
```bash
slow_function(i)
```
    }
    Lazy evaluation: FALSE
    Asynchronous evaluation: TRUE
    Local evaluation: TRUE
    Environment: R_GlobalEnv
    Capture standard output: TRUE
    Capture condition classes: ‘condition’
    Globals: <none>
    Packages: <none>
    L'Ecuyer-CMRG RNG seed: <none> (seed = FALSE)
    Resolved: TRUE
    Value: 136 bytes of class ‘character’
    Early signaling: FALSE
    Owner process: 1d0b28ab-cd81-c92e-728e-34dd8b6a6a48
    Class: ‘MulticoreFuture’, ‘MultiprocessFuture’, ‘Future’, ‘environment’


####  3. Evaluate the code
The `value()` object will take the list and run the code in parallel. Since the output is a list of lists, we can collapse it into an array with the `unlist` object. 


```R
# evaluate, distribute each iteration here 
output <- value(command_set)

# ouput is here but separates lists. 
print("list output")
head(output) # using head to limit output


# we can combine them into a vector using unlist()
print("array output")
head(unlist(output)) # using head to limit output
```

    [1] "list output"



<ol>
```bash
<li>'Step 1 completed'</li>
<li>'Step 2 completed'</li>
<li>'Step 3 completed'</li>
<li>'Step 4 completed'</li>
<li>'Step 5 completed'</li>
<li>'Step 6 completed'</li>
```
</ol>



    [1] "array output"



<style>
.list-inline {list-style: none; margin:0; padding: 0}
.list-inline>li {display: inline-block}
.list-inline>li:not(:last-child)::after {content: "\00b7"; padding: 0 .5ex}
</style>
<ol class=list-inline><li>'Step 1 completed'</li><li>'Step 2 completed'</li><li>'Step 3 completed'</li><li>'Step 4 completed'</li><li>'Step 5 completed'</li><li>'Step 6 completed'</li></ol>



all in one block:


```R
# plan
plan(multicore)

command_set <- vector(mode = "list", length = 50)
t1 <- proc.time()

for (i in 1:50){
  command_set[[i]] <- future(
    # but code for each iteration with curly brackets
    {slow_function(i)}
  ) 
}

# evaluate
output <- value(command_set)

# ouput is here
head(unlist(output))


# timestampe after loop 
t2 <- proc.time()


print("Elapsed time for future example:")
# length of time
time_Elapsed <- t2[[3]] - t1[[3]]
round(time_Elapsed, 2)
```


<style>
.list-inline {list-style: none; margin:0; padding: 0}
.list-inline>li {display: inline-block}
.list-inline>li:not(:last-child)::after {content: "\00b7"; padding: 0 .5ex}
</style>
<ol class=list-inline><li>'Step 1 completed'</li><li>'Step 2 completed'</li><li>'Step 3 completed'</li><li>'Step 4 completed'</li><li>'Step 5 completed'</li><li>'Step 6 completed'</li></ol>



    [1] "Elapsed time for future example:"



11.85


## 2. Tidyverse example using the furrr package

To change [tidyverse code](https://www.tidyverse.org/){target=_blank}, all that is needed is to set up the parallelization (plan object) and then replace the map object with future_map. The evaluation step is completed with the future_map object. 

### Setup
This example uses the gapminder dataset and creates a linear model for each country. The datafame becomes a nested dataframe by country so we can use the purrr package to iterate over each country and compute the linear model. See this link to the [purrr package](https://purrr.tidyverse.org/){target=_blank} and the [repurrrsive](https://github.com/jennybc/repurrrsive){target=_blank} for more details. 


```R
# create a nested dataframe with each country as a row 
# we will iterate using the map function from the purrr package
head(gap_simple)


country_nested <- 
  gap_simple %>%
  group_by(country) %>%
  nest() %>%
  ungroup() # this is necessary or furrr will be slow



# linear model we will run for each country. 
# in the real world, don't do this!! Always check model assumptions!
custom_model <- 
  function(data){
    # just to slow it down so its more obvious it's in parallel
    Sys.sleep(.2)
    lm(lifeExp ~ pop + gdpPercap + year, data = data)
  }



# Sequential form (normal purrr)
t1 <- proc.time()
model_done <- 
    country_nested %>%
    mutate(lm_obj = map(data, custom_model))

t2 <- proc.time()
print("Elapsed time for sequential purrr  example:")
# length of time
time_Elapsed <- t2[[3]] - t1[[3]]
round(time_Elapsed, 2)
```


<table class="dataframe">
<caption>A tibble: 6 × 6</caption>
<thead>
```bash
<tr><th scope=col>country</th><th scope=col>continent</th><th scope=col>year</th><th scope=col>lifeExp</th><th scope=col>pop</th><th scope=col>gdpPercap</th></tr>
<tr><th scope=col>&lt;fct&gt;</th><th scope=col>&lt;fct&gt;</th><th scope=col>&lt;int&gt;</th><th scope=col>&lt;dbl&gt;</th><th scope=col>&lt;int&gt;</th><th scope=col>&lt;dbl&gt;</th></tr>
```
</thead>
<tbody>
```bash
<tr><td>Afghanistan</td><td>Asia</td><td>1952</td><td>28.801</td><td> 8425333</td><td>779.4453</td></tr>
<tr><td>Afghanistan</td><td>Asia</td><td>1957</td><td>30.332</td><td> 9240934</td><td>820.8530</td></tr>
<tr><td>Afghanistan</td><td>Asia</td><td>1962</td><td>31.997</td><td>10267083</td><td>853.1007</td></tr>
<tr><td>Afghanistan</td><td>Asia</td><td>1967</td><td>34.020</td><td>11537966</td><td>836.1971</td></tr>
<tr><td>Afghanistan</td><td>Asia</td><td>1972</td><td>36.088</td><td>13079460</td><td>739.9811</td></tr>
<tr><td>Afghanistan</td><td>Asia</td><td>1977</td><td>38.438</td><td>14880372</td><td>786.1134</td></tr>
```
</tbody>
</table>



    [1] "Elapsed time for sequential purrr  example:"



28.92


###  Rerunning tidyverse in parallel via the furrr package 
To change the code to work with tidyverse, replace the map object with future_map.


```R
# using furrr package

# use all cores available
plan(multicore) 

# this time we do no not need to the evauluation step as it is included in the future_map function.
t1 <- proc.time()
model_done <- 
    country_nested %>%
    mutate(lm_obj = future_map(data, custom_model)) # switched map to future_map!

# switch back to one core
plan(sequential)   
t2 <- proc.time()

print("Elapsed time for furrr example:")
time_Elapsed <- t2[[3]] - t1[[3]]
round(time_Elapsed, 2)

```

    [1] "Elapsed time for furrr example:"



4.17


Example: model output for Bosnia and Herzegovina (proof the future code provided output). 



```R
model_done$country[[13]]
model_done$lm_obj[[13]]
```


Bosnia and Herzegovina
<details>
```bash
<summary style=display:list-item;cursor:pointer>
	<strong>Levels</strong>:
</summary>
<style>
.list-inline {list-style: none; margin:0; padding: 0}
.list-inline>li {display: inline-block}
.list-inline>li:not(:last-child)::after {content: "\00b7"; padding: 0 .5ex}
</style>
<ol class=list-inline><li>'Afghanistan'</li><li>'Albania'</li><li>'Algeria'</li><li>'Angola'</li><li>'Argentina'</li><li>'Australia'</li><li>'Austria'</li><li>'Bahrain'</li><li>'Bangladesh'</li><li>'Belgium'</li><li>'Benin'</li><li>'Bolivia'</li><li>'Bosnia and Herzegovina'</li><li>'Botswana'</li><li>'Brazil'</li><li>'Bulgaria'</li><li>'Burkina Faso'</li><li>'Burundi'</li><li>'Cambodia'</li><li>'Cameroon'</li><li>'Canada'</li><li>'Central African Republic'</li><li>'Chad'</li><li>'Chile'</li><li>'China'</li><li>'Colombia'</li><li>'Comoros'</li><li>'Congo, Dem. Rep.'</li><li>'Congo, Rep.'</li><li>'Costa Rica'</li><li>'Cote d\'Ivoire'</li><li>'Croatia'</li><li>'Cuba'</li><li>'Czech Republic'</li><li>'Denmark'</li><li>'Djibouti'</li><li>'Dominican Republic'</li><li>'Ecuador'</li><li>'Egypt'</li><li>'El Salvador'</li><li>'Equatorial Guinea'</li><li>'Eritrea'</li><li>'Ethiopia'</li><li>'Finland'</li><li>'France'</li><li>'Gabon'</li><li>'Gambia'</li><li>'Germany'</li><li>'Ghana'</li><li>'Greece'</li><li>'Guatemala'</li><li>'Guinea'</li><li>'Guinea-Bissau'</li><li>'Haiti'</li><li>'Honduras'</li><li>'Hong Kong, China'</li><li>'Hungary'</li><li>'Iceland'</li><li>'India'</li><li>'Indonesia'</li><li>'Iran'</li><li>'Iraq'</li><li>'Ireland'</li><li>'Israel'</li><li>'Italy'</li><li>'Jamaica'</li><li>'Japan'</li><li>'Jordan'</li><li>'Kenya'</li><li>'Korea, Dem. Rep.'</li><li>'Korea, Rep.'</li><li>'Kuwait'</li><li>'Lebanon'</li><li>'Lesotho'</li><li>'Liberia'</li><li>'Libya'</li><li>'Madagascar'</li><li>'Malawi'</li><li>'Malaysia'</li><li>'Mali'</li><li>'Mauritania'</li><li>'Mauritius'</li><li>'Mexico'</li><li>'Mongolia'</li><li>'Montenegro'</li><li>'Morocco'</li><li>'Mozambique'</li><li>'Myanmar'</li><li>'Namibia'</li><li>'Nepal'</li><li>'Netherlands'</li><li>'New Zealand'</li><li>'Nicaragua'</li><li>'Niger'</li><li>'Nigeria'</li><li>'Norway'</li><li>'Oman'</li><li>'Pakistan'</li><li>'Panama'</li><li>'Paraguay'</li><li>'Peru'</li><li>'Philippines'</li><li>'Poland'</li><li>'Portugal'</li><li>'Puerto Rico'</li><li>'Reunion'</li><li>'Romania'</li><li>'Rwanda'</li><li>'Sao Tome and Principe'</li><li>'Saudi Arabia'</li><li>'Senegal'</li><li>'Serbia'</li><li>'Sierra Leone'</li><li>'Singapore'</li><li>'Slovak Republic'</li><li>'Slovenia'</li><li>'Somalia'</li><li>'South Africa'</li><li>'Spain'</li><li>'Sri Lanka'</li><li>'Sudan'</li><li>'Swaziland'</li><li>'Sweden'</li><li>'Switzerland'</li><li>'Syria'</li><li>'Taiwan'</li><li>'Tanzania'</li><li>'Thailand'</li><li>'Togo'</li><li>'Trinidad and Tobago'</li><li>'Tunisia'</li><li>'Turkey'</li><li>'Uganda'</li><li>'United Kingdom'</li><li>'United States'</li><li>'Uruguay'</li><li>'Venezuela'</li><li>'Vietnam'</li><li>'West Bank and Gaza'</li><li>'Yemen, Rep.'</li><li>'Zambia'</li><li>'Zimbabwe'</li></ol>
```
</details>



    
    Call:
    lm(formula = lifeExp ~ pop + gdpPercap + year, data = data)
    
    Coefficients:
    (Intercept)          pop    gdpPercap         year  
     -4.950e+02    4.916e-06   -5.017e-04    2.757e-01  



## 3. Bayesian example using multiple nodes via the future.batchtools package
This example runs multiple Bayesian models in parallel by submitting each model (iteration) to a separate node. This is very useful because each model can already run in parallel, thus CARC enables you to run all of your Bayesian parallelized models at once. the future.batchtools packages will use information in the batchtools.torque.tmpl file in your current directory to submit jobs. This file can be modified to change parameters such as length of walltime, number of cores, etc (see file below). 

These Bayesian models are joint-species distribution models (jSDMs) which fit the distributions of bird species and determines how it relates to their habitat, phylogeny, and traits. The code will run four Bayesian models that differ in their thinning and then write the models to file. The data and model objects are pre-built and loaded with the hmsc_setup.RData file and are derived from the [bird example](https://www2.helsinki.fi/en/researchgroups/statistical-ecology/hmsc){target=_blank} from their book. 

For the sequential version, I already set up it using future. You can check your future code by running it sequentially by using plan(sequential).

### batchtools.torque.tmpl

batchtools.torque.tmpl file needed in the same directory or specify the path to it. You will need this file created prior to running this example.

## Setup 


```R
# bring in data for model
install.packages("coda")
install.packages("devtools") # if not yet installed
library(devtools)
install_github("hmsc-r/HMSC")
library(coda)
library(Hmsc)
load("hmsc_setup.RData")

# Setting up the model
studyDesign = data.frame(Route = XData$Route)
rL = Hmsc::HmscRandomLevel(sData=xy)
XFormula = ~ hab + poly(clim,degree = 2,raw = TRUE)
TrFormula = ~Migration + LogMass

# parameters for bayesian models. 
nChains = 4
nParallel = 4 
samples = 10 
```

    Updating HTML index of packages in '.Library'
    
    Making 'packages.html' ...
     done
    
    also installing the dependencies ‘askpass’, ‘credentials’, ‘sys’, ‘zip’, ‘gitcreds’, ‘ini’, ‘fastmap’, ‘highr’, ‘markdown’, ‘xfun’, ‘diffobj’, ‘rematch2’, ‘clipr’, ‘curl’, ‘gert’, ‘gh’, ‘rprojroot’, ‘whisker’, ‘yaml’, ‘processx’, ‘mime’, ‘openssl’, ‘cachem’, ‘xopen’, ‘commonmark’, ‘knitr’, ‘Rcpp’, ‘stringr’, ‘xml2’, ‘brio’, ‘praise’, ‘ps’, ‘waldo’, ‘usethis’, ‘callr’, ‘desc’, ‘httr’, ‘memoise’, ‘pkgbuild’, ‘pkgload’, ‘rcmdcheck’, ‘remotes’, ‘roxygen2’, ‘rstudioapi’, ‘rversions’, ‘sessioninfo’, ‘testthat’
    
    
    Updating HTML index of packages in '.Library'
    
    Making 'packages.html' ...
     done
    
    Loading required package: usethis
    
    Downloading GitHub repo hmsc-r/HMSC@HEAD
    


    RcppArmad... (NA    -> 0.10.6.0.0) [CRAN]
    matrixStats  (NA    -> 0.60.0    ) [CRAN]
    conquer      (NA    -> 1.0.2     ) [CRAN]
    MatrixModels (NA    -> 0.5-0     ) [CRAN]
    SparseM      (NA    -> 1.81      ) [CRAN]
    fansi        (0.4.2 -> 0.5.0     ) [CRAN]
    gridExtra    (NA    -> 2.3       ) [CRAN]
    dotCall64    (NA    -> 1.0-1     ) [CRAN]
    plyr         (NA    -> 1.8.6     ) [CRAN]
    quantreg     (NA    -> 5.86      ) [CRAN]
    mcmc         (NA    -> 0.9-7     ) [CRAN]
    maps         (NA    -> 3.3.0     ) [CRAN]
    viridis      (NA    -> 0.6.1     ) [CRAN]
    spam         (NA    -> 2.7-0     ) [CRAN]
    truncnorm    (NA    -> 1.0-8     ) [CRAN]
    statmod      (NA    -> 1.4.36    ) [CRAN]
    sp           (NA    -> 1.4-5     ) [CRAN]
    pROC         (NA    -> 1.17.0.1  ) [CRAN]
    pracma       (NA    -> 2.3.3     ) [CRAN]
    MCMCpack     (NA    -> 1.5-0     ) [CRAN]
    FNN          (NA    -> 1.1.3     ) [CRAN]
    fields       (NA    -> 12.5      ) [CRAN]
    BayesLogit   (NA    -> 2.1       ) [CRAN]
    ape          (NA    -> 5.5       ) [CRAN]
    abind        (NA    -> 1.4-5     ) [CRAN]


    Installing 25 packages: RcppArmadillo, matrixStats, conquer, MatrixModels, SparseM, fansi, gridExtra, dotCall64, plyr, quantreg, mcmc, maps, viridis, spam, truncnorm, statmod, sp, pROC, pracma, MCMCpack, FNN, fields, BayesLogit, ape, abind
    
    Updating HTML index of packages in '.Library'
    
    Making 'packages.html' ...
     done
    


    ✔  checking for file ‘/tmp/Rtmp7wOOGo/remotes4e4e7a4c3c6e/hmsc-r-HMSC-940f41c/DESCRIPTION’ (389ms)
    ─  preparing ‘Hmsc’:
    ✔  checking DESCRIPTION meta-information
    ─  installing the package to process help pages
    ─  saving partial Rd database (19.7s)
    ─  checking for LF line-endings in source and make files and shell scripts
    ─  checking for empty or unneeded directories
    ─  building ‘Hmsc_3.0-12.tar.gz’
       
    


```R
#!/bin/bash

## Job name:
#PBS -q default
#PBS -l nodes=1:ppn=4
#PBS -l walltime=0:20:00
#PBS -N x_big_model_test_parallel_4_cores_4_chains
#PBS -j oe
#PBS -m ae


start=`date +%s`

cd $PBS_O_WORKDIR

# load R
module load r-4.0.4-gcc-10.2.0-python3-dghog6f

#Rscript -e '.libPaths("~/R/Jupyter")' -e 'batchtools::doJobCollection("<%= uri %>")'
Rscript -e 'batchtools::doJobCollection("<%= uri %>")'
end=`date +%s`

runtime=$((end-start))

echo "Runtime was $runtime seconds"
```


```R
# set up that it will submit pbs scripts for each model
plan(sequential)

m = Hmsc(Y=Y, XData = XData, XFormula=XFormula, 
         phyloTree = phyloTree, TrData = TrData, 
         TrFormula = TrFormula,
         distr="probit", studyDesign=studyDesign, 
         ranLevels=list(Route=rL))
y <- list()
t1 <- proc.time()
# running 4 models, each with a different thinning value
for (thin in c(2,3,4,5)){
  y[[thin]] <- future({
  transient = 50*thin
  m = sampleMcmc(m, thin = thin, samples = samples, transient = transient,
                 nChains = nChains, initPar = "fixed effects",
                 nParallel = nParallel)
                 
  # write model outputs to file               
  filename=file.path(paste0("Big_model_torque_chains_",as.character(nChains),"_samples_",as.character(samples),"_thin_",as.character(thin)))
  save(m,file=filename)
  }, seed = TRUE)
}
# evaluate expression
#y <- value(y) 

t2 <- proc.time()
time_Elapsed <- t2[[3]] - t1[[3]]
round(time_Elapsed, 2)
```


875.1


###  Rerunning bayesian models in parallel using future.batchtools package 



```R
# set up that it will submit pbs scripts for each model
# calls upon the batchtools.torque.tmpl file to set the parameters for each job

plan(batchtools_torque)

m = Hmsc(Y=Y, XData = XData, XFormula=XFormula, 
         phyloTree = phyloTree, TrData = TrData, 
         TrFormula = TrFormula,
         distr="probit", studyDesign=studyDesign, 
         ranLevels=list(Route=rL))

y <- list()
t1 <- proc.time()
# running 4 models, each with a different thinning value
for (thin in c(2,3,4,5)){
  y[[thin]] <- future({
  transient = 50*thin
  m = sampleMcmc(m, thin = thin, samples = samples, transient = transient,
                 nChains = nChains, initPar = "fixed effects",
                 nParallel = nParallel)
                 
  # write model outputs to file               
  filename=file.path(paste0("Big_model_torque_chains_",as.character(nChains),"_samples_",as.character(samples),"_thin_",as.character(thin)))
  save(m,file=filename)
  }, seed = TRUE)
}
# evaluate expression
y <- value(y) 

t2 <- proc.time()
time_Elapsed <- t2[[3]] - t1[[3]]
round(time_Elapsed, 2)


```

    setting updater$Gamma2=FALSE due to specified phylogeny matrix
    
    setting updater$Gamma2=FALSE due to specified phylogeny matrix
    
    setting updater$Gamma2=FALSE due to specified phylogeny matrix
    
    setting updater$Gamma2=FALSE due to specified phylogeny matrix
    



375.03


<a id=’section_4’></a>
## Appendix: Testing speedup with Future

For some circumstances, writing your code in parallel with future package can make it slower! To illustrate this problem, I created a function called speedup_calc that allows you to play with the parameters and see what scenarios running the code in parallel. Essentially, it runs a for loop for different combinations of iterations, cores, and length of iterations (wait_time). Also, you can see the cost of setting up the code with the "evaluate" argument. When it is done, it will generate two plots (unless evaluate == FALSE), a plot showing how long it took for each iteration and core combination to complete, and a plot showing the speedup when the number of cores is increased. 

Arguments:

\* iterations requires a value (total iteratations) or a vector. 

\* cores requires numeric a vector of cores you want to run. 

\* wait_time requires a numeric value for how to wait for each iterative step. 

\* evaluate requires a TRUE or FALSE. Allows you to turn off evaluation if you only want to measure setup time. It will also skip the speedup plot since setup time cannot be calculated for the serial version. 


speedup_calc function:


```R
speedup_calc <- 
    function(iterations, cores, wait_time, evaluate){
        
    try(if(cores[1] != 1) stop("Need to run the core as 1"))

    # set up arrays for storing for-loop data
        times <- array(dim = c(length(iterations), length(cores)))
        row.names(times) <- iterations
        
        
       
    # running the first set outside of the future loop. 
    # skips it if since there is no set up time for serial version. 
    if (evaluate == TRUE){    
        
        j <- 1
        for (i in 1:length(iterations)){

            current_step <- iterations[i]
                    ### time point 1
            t1 <- proc.time()
            setup <- vector(mode = "list", length = length(iterations))

                 for (k in 1:current_step){

                        Sys.sleep(wait_time)
                    }

            
            ### time point
            t2 <- proc.time()

            ## calculate difference in time and convert to minutes. Write to array
            times[i, j] <- (t2[[3]] - t1[[3]]) 

                }


    }
        
        # loop that first run through the number of cores
        for (j in 2:length(cores)){

            plan(multicore, workers = cores[j])
            # figure out time it taakes for each iteration for j number of cores
            for (i in 1:length(iterations)){

                current_step <- iterations[i]
                ### time point 1
                t1 <- proc.time()
                setup <- vector(mode = "list", length = length(iterations))

                    for (k in 1:current_step){

                        setup[[k]] <- future({Sys.sleep(wait_time)})
                    }
                
                #  Allows you to run it without evaluating the code
                if (evaluate == TRUE){evaluated <-  value(setup)}
               
                ### time point
                t2 <- proc.time()

                ## calculate difference in time and convert to minutes. Write to array
                times[i, j] <- (t2[[3]] - t1[[3]]) 
            }

            
        }
        
        
        ### clean and plot all of the data
        options(repr.plot.width = 5, repr.plot.height = 2)  
        # loop is finished so now cleanaing up data
        times <- as.data.frame(times)
        colnames(times) <- cores
        times$iterations <- row.names(times)
        
        times$iterations <- factor(as.character(iterations), levels = iterations)
        # plot time it takes for each core/iteratiaons combo
   
    if (evaluate == FALSE){
        time_plot <- 
            times %>%
            pivot_longer(cols = 1:length(cores), names_to = "Cores", values_to = "Seconds") %>%
            filter(Cores != "1") %>%
            ggplot(aes(x = iterations, y = Seconds, group =  Cores, color = Cores)) + 
            geom_line() + geom_point() + 
            ggtitle("Time Needed to Set Up Task")
        print(time_plot)
        }
        
     
        
    if (evaluate == TRUE){   
        
         time_plot <- 
            times %>%
            pivot_longer(cols = 1:length(cores), names_to = "Cores", values_to = "Seconds") %>%
            ggplot(aes(x = iterations, y = Seconds, group =  Cores, color = Cores)) + 
            geom_line() + geom_point() + 
            ggtitle("Time Needed to Complete Task")
        print(time_plot)
        
        # speedup calcs
        speedup_df <- 1/ (times[,1:length(cores)] /times[,1] )
        speedup_df$iterations <- iterations

    
        speedup_plot <- 
            speedup_df %>%
            pivot_longer(cols = 1:length(cores), names_to = "Cores", values_to = "Speedup") %>%
            mutate(Cores = as.numeric(Cores)) %>%
            ggplot(aes(x = Cores, y = Speedup, group =  iterations, color = iterations)) + 
            geom_line() + geom_point() + 
            ggtitle("Speedup Calculations")

        print(speedup_plot)
        }
        
    }

```

### Scenario 1: Cost of setup for many iterations 
This scenario plots how much time the setup takes. The cost for 100 iterations or more can be timely. The line with one core is omitted becauase it doesn't have setup time.


```R
speedup_calc(iterations = seq(from = 20, 100, by = 20), cores = c(1,2, 4, 8), wait_time = .25, evaluate = FALSE)
```


    
![png](../assets/images/quickbytes/parallel-r-future_files/parallel-r-future_41_0.png)
    


### Scenario 2: running previous example
Given the cost due to the setup time, parallelization offer very little improvement. 


```R
speedup_calc(iterations = seq(from = 20, 100, by = 20), cores = c(1,2, 4, 6, 8), wait_time = .25, evaluate = TRUE)
```


    
![png](../assets/images/quickbytes/parallel-r-future_files/parallel-r-future_43_0.png)
    



    
![png](../assets/images/quickbytes/parallel-r-future_files/parallel-r-future_43_1.png)
    


### Scenario 3: Moderate speedup with a slower iteration
When you incease the wait_time to 1 second, the increase in cores does provide a benefit. 


```R
speedup_calc(iterations = seq(from = 100, 300, by = 100), cores = c(1,2,4,6, 8), wait_time = 1, evaluate = TRUE)

```


    
![png](../assets/images/quickbytes/parallel-r-future_files/parallel-r-future_45_0.png)
    



    
![png](../assets/images/quickbytes/parallel-r-future_files/parallel-r-future_45_1.png)
    


### Scenario 4: Best scenario for speedup
The best case for future will be running the code with few iterations but very slow functions. 


```R
speedup_calc(iterations = seq(from = 10, 30, by = 10), cores = c(1,2,4,6, 8), wait_time = 15, evaluate = TRUE)
```


    
![png](../assets/images/quickbytes/parallel-r-future_files/parallel-r-future_47_0.png)
    



    
![png](../assets/images/quickbytes/parallel-r-future_files/parallel-r-future_47_1.png)
    


## Conclusion
The future package will offer the best performance when there are fewer iterations but each iteration is very slow. With your code, try running one iteration to see how long it takes and then use that estimate with speedup_calc. Depending on the wait_time and number of iterations, the function will give an estimate on if parallelization is worthwhile (caveat: the code will be slower if you have a large global environment).

## Video walkthrough

**Parallel R with Future** — from the [CARC video tutorials](../training/videos.md):

<iframe class="carc-video" src="https://www.youtube-nocookie.com/embed/G5xGfF151Co" title="Parallel R with Future" loading="lazy" allow="accelerometer; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen></iframe>

<p class="carc-provenance" markdown>Migrated from [UNM-CARC QuickBytes](https://github.com/UNM-CARC/QuickBytes/blob/master/Parallel_R_with_Future.ipynb){target=_blank} (last source update 2021-08-11). Spotted a problem? [Open an issue or pull request](https://github.com/UNM-CARC/QuickBytes){target=_blank}.</p>

---8<--- https://unm-carc.github.io/docs/software/paraview/

---
title: "ParaView remote visualization"
description: "Run the ParaView server on CARC compute nodes and connect from your desktop client."
type: Guide
tags:
  - Visualization
  - ParaView
generated:
  by: "claude/fable-5"
  at: "2026-08-29T00:00:00Z"
sources:
  - id: quickbytes
    resource: "https://github.com/UNM-CARC/QuickBytes/blob/master/paraview.md"
    title: "UNM-CARC QuickBytes: paraview.md"
    author: "team:unm-carc"
    last_modified: "2024-06-09T01:00:00-06:00"
---

# ParaView remote visualization

ParaView is an open-source, multi-platform data analysis and visualization application. ParaView users can quickly build visualizations to analyze their data using qualitative and quantitative techniques. The data exploration can be done interactively in 3D or programmatically using ParaView's batch processing capabilities.

ParaView was developed to analyze extremely large datasets using distributed memory computing resources. It can be run on supercomputers to analyze datasets of petascale size.

These steps will help you setup Paraview to work as a client/server mode, being your laptop/desktop computer a client and the cluster a server. Be sure that the ParaView version installed on your local computer matches the same one that is installed on the CARC clusters.

To see a current list of paraview versions installed on CARC clusters login to the cluster and run

    module spider paraview

* Downloads: https://www.paraview.org/download/
* ParaView User's Guide: https://docs.paraview.org/en/latest/UsersGuide/index.html

## Hopper Cluster Connection

The most common approach to use ParaView on Hopper is through the Client-Server mode support by ParaView, which requires an installation of ParaView on your local computer (Client). There are two methods to connect to Paraview Server (PVSERVER):

### Method 1: Direct Connection (Off-Campus)

The process to connecto to ParaView is, in one terminal you will ask Hopper to assign you compute nodes, where you will run the ParaView server. Once the ParaView server is listening for connections, you will open an ssh tunnel in another terminal window (This process is from your local computer to one of the compute nodes you were assigned). Then, you will tell the ParaView client on your computer to connect to the tunnel and so to the compute nodes at CARC, where it will perform the rendering.

#### Terminal 1: Login to Hopper and allocate resources

#### 1. Login to Hopper

    ssh username@hopper.alliance.unm.edu
  
#### 2. Allocating 2 nodes : Total 64 cores for 30 minutes in the "General" queue/partition.

    salloc --nodes 1 --exclusive --partition general --time 00:30:00

#### 3. Load Module

    module load paraview/5.11.1

#### 4. Running pvserver (this command will allow a connection between your local computer and Hopper).

    mpiexec -np 32 pvserver --mpi --force-offscreen-rendering --server-port=11111

![](../assets/images/quickbytes/paraview-img1.png)

#### Terminal 2: Hopper SSH Tunneling
The hopper### corresponds to the compute node allocated by slurm, and do not forget to change your username. 

    ssh -L 11111:hopper###:11111 username@hopper.alliance.unm.edu

#### ParaView 5.11.1 Client and Setup Server Configuration

1. File --> Connect 
2. On the "Choose Server Configuration" window: 
* Click on "Add Server"
* Name: Hopper
* Server Type: "Client / Server"
* Host: localhost
* Port: 11111

![](../assets/images/quickbytes/paraview-img2.png)

3. Click on "Configure"
4. Startup Type: Manual
5. Click on "Save"

You can then click connect. It will take about 30 seconds to connect and let you start using paraview normally. Once connected, you can test it worked by going to view > Memory Inspector. You should see the following:

![](../assets/images/quickbytes/paraview-img3.png)

NOTE: When you are finished make sure to end the interactive job on the compute nodes. You can do this by exiting "Exit" the compute node or the "scancel" command on the cluster head node.

### Method 2: Reverse Connection (UNM On-Campus)

This process allows you to connect to Hopper service node. This process requires to know your localhost IP address "local_host_IP". Check your firewall setting if you are having firewall connectivity issues.

#### Terminal 1: Login to Hopper and allocate resources

#### 1. Login to Hopper

    ssh username@hopper.alliance.unm.edu
  
#### 2. Allocating 1 node : Total 32 cores for 30 minutes in the "General" queue/partition.

    salloc --nodes 1 --exclusive --partition general --time 00:30:00

#### 3. Load Module

    module load paraview/5.11.1

#### 4. Running pvserver (this command will allow a connection between your local computer and Hopper).

    mpiexec -np 32 pvserver --mpi --force-offscreen-rendering --rc --client-host=My_Public_IP

#### Opening ParaView 5.11.1 Client and Setup Server Configuration

Note: To Verify, Client - Server setup, go to "View" and select "Memory Inspector"

1. File --> Connect
2. On the "Choose Server Configuration" window:
* Click on "Add Server"
* Name: Hopper RC
* Server Type: "Client / Server (Reverse Connection)"
* Host: localhost
* Port: 11111

3. Click on "Configure"
4. Startup Type: Manual
5. Click on "Save"

NOTE: When you are finished make sure to end the interactive job on the compute nodes. You can do this by exiting "Exit" the compute node or the "scancel" command on the cluster head node.

## ParaView executables
ParaView comes with several executables that serve different purposes. These are: paraview, pvpython, pvbatch, pvserver, pvdataserver and pvrenderserver. To learn more about this executables, https://docs.paraview.org/en/latest/UsersGuide/introduction.html#paraview-executables. 

*This quickbyte was validated on 6/9/2024*

<p class="carc-provenance" markdown>Migrated from [UNM-CARC QuickBytes](https://github.com/UNM-CARC/QuickBytes/blob/master/paraview.md){target=_blank} (last source update 2024-06-09). Spotted a problem? [Open an issue or pull request](https://github.com/UNM-CARC/QuickBytes){target=_blank}.</p>

---8<--- https://unm-carc.github.io/docs/software/pytorch-classifier/

---
title: "PyTorch image classifier walkthrough"
description: "End-to-end example: train an image classifier with PyTorch on a CARC GPU node."
type: Tutorial
tags:
  - Python
  - GPU
  - Machine learning
  - PyTorch
status: draft
generated:
  by: "claude/fable-5"
  at: "2026-08-29T00:00:00Z"
sources:
  - id: quickbytes
    resource: "https://github.com/UNM-CARC/QuickBytes/blob/master/PyTorch_Classifier_Xena%20.ipynb"
    title: "UNM-CARC QuickBytes: PyTorch_Classifier_Xena .ipynb"
    author: "team:unm-carc"
    last_modified: "2023-09-20T15:17:43-06:00"
---

# PyTorch image classifier walkthrough

This notebook serves as an extension to the PyTorch tutorial that will get you started with basic image classification. It is based on the following tutorial (https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html#) and includes modifications that you will need to get the tutorial working.


```python
import torch
```

To use torchvision to your notebook, you'll need to first go back to your terminal and run the command 'conda install torchvision'. Once the installation is finished, run the below codeblock to import it.


```python
import torchvision
```


```python
import torchvision.transforms as transforms
```


```python
transform = transforms.Compose(
    [transforms.ToTensor(),
     transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])

batch_size = 4

trainset = torchvision.datasets.CIFAR10(root='./data', train=True,
                                        download=True, transform=transform)
trainloader = torch.utils.data.DataLoader(trainset, batch_size=batch_size,
                                          shuffle=True, num_workers=2)

testset = torchvision.datasets.CIFAR10(root='./data', train=False,
                                       download=True, transform=transform)
testloader = torch.utils.data.DataLoader(testset, batch_size=batch_size,
                                         shuffle=False, num_workers=2)

classes = ('plane', 'car', 'bird', 'cat',
           'deer', 'dog', 'frog', 'horse', 'ship', 'truck')

```

    Files already downloaded and verified


    /users/mdunlavy/.local/lib/python3.9/site-packages/torch/utils/data/dataloader.py:478: UserWarning: This DataLoader will create 2 worker processes in total. Our suggested max number of worker in current system is 1, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.
      warnings.warn(_create_warning_msg(


    Files already downloaded and verified


To use matplotlib, you'll need to first go back to your terminal and run the command 'conda install matplotlib'. You'll then be able to use the plot function for the below training and classification. 


```python
import matplotlib.pyplot as plt
import numpy as np

# functions to show an image


def imshow(img):
    img = img / 2 + 0.5     # unnormalize
    npimg = img.numpy()
    plt.imshow(np.transpose(npimg, (1, 2, 0)))
    plt.show()


# get some random training images
dataiter = iter(trainloader)
images, labels = next(dataiter)

# show images
imshow(torchvision.utils.make_grid(images))
# print labels
print(' '.join(f'{classes[labels[j]]:5s}' for j in range(batch_size)))
```


    
![png](../assets/images/quickbytes/pytorch-classifier_files/pytorch-classifier_8_0.png)
    


    plane deer  car   cat  



```python
import torch.nn as nn
import torch.nn.functional as F


class Net(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(3, 6, 5)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(6, 16, 5)
        self.fc1 = nn.Linear(16 * 5 * 5, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)

    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = torch.flatten(x, 1) # flatten all dimensions except batch
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x


net = Net()
```


```python
import torch.optim as optim

criterion = nn.CrossEntropyLoss()
optimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9)
```


```python
for epoch in range(2):  # loop over the dataset multiple times

    running_loss = 0.0
    for i, data in enumerate(trainloader, 0):
        # get the inputs; data is a list of [inputs, labels]
        inputs, labels = data

        # zero the parameter gradients
        optimizer.zero_grad()

        # forward + backward + optimize
        outputs = net(inputs)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()

        # print statistics
        running_loss += loss.item()
        if i % 2000 == 1999:    # print every 2000 mini-batches
            print(f'[{epoch + 1}, {i + 1:5d}] loss: {running_loss / 2000:.3f}')
            running_loss = 0.0

print('Finished Training')
```

    /users/mdunlavy/.local/lib/python3.9/site-packages/torch/nn/functional.py:718: UserWarning: Named tensors and all their associated APIs are an experimental feature and subject to change. Please do not use them for anything important until they are released as stable. (Triggered internally at  /pytorch/c10/core/TensorImpl.h:1156.)
      return torch.max_pool2d(input, kernel_size, stride, padding, dilation, ceil_mode)


    [1,  2000] loss: 2.211
    [1,  4000] loss: 1.886
    [1,  6000] loss: 1.695
    [1,  8000] loss: 1.596
    [1, 10000] loss: 1.533
    [1, 12000] loss: 1.478
    [2,  2000] loss: 1.428
    [2,  4000] loss: 1.370
    [2,  6000] loss: 1.379
    [2,  8000] loss: 1.342
    [2, 10000] loss: 1.326
    [2, 12000] loss: 1.294
    Finished Training



```python
PATH = './cifar_net.pth'
torch.save(net.state_dict(), PATH)
```


```python
dataiter = iter(testloader)
images, labels = next(dataiter)

# print images
imshow(torchvision.utils.make_grid(images))
print('GroundTruth: ', ' '.join(f'{classes[labels[j]]:5s}' for j in range(4)))
```


    
![png](../assets/images/quickbytes/pytorch-classifier_files/pytorch-classifier_13_0.png)
    


    GroundTruth:  cat   ship  ship  plane



```python
net = Net()
net.load_state_dict(torch.load(PATH))
```




    <All keys matched successfully>




```python
outputs = net(images)
```


```python
_, predicted = torch.max(outputs, 1)

print('Predicted: ', ' '.join(f'{classes[predicted[j]]:5s}'
                              for j in range(4)))
```

    Predicted:  cat   plane plane plane



```python
correct = 0
total = 0
# since we're not training, we don't need to calculate the gradients for our outputs
with torch.no_grad():
    for data in testloader:
        images, labels = data
        # calculate outputs by running images through the network
        outputs = net(images)
        # the class with the highest energy is what we choose as prediction
        _, predicted = torch.max(outputs.data, 1)
        total += labels.size(0)
        correct += (predicted == labels).sum().item()

print(f'Accuracy of the network on the 10000 test images: {100 * correct // total} %')
```

    Accuracy of the network on the 10000 test images: 52 %



```python
# prepare to count predictions for each class
correct_pred = {classname: 0 for classname in classes}
total_pred = {classname: 0 for classname in classes}

# again no gradients needed
with torch.no_grad():
    for data in testloader:
        images, labels = data
        outputs = net(images)
        _, predictions = torch.max(outputs, 1)
        # collect the correct predictions for each class
        for label, prediction in zip(labels, predictions):
            if label == prediction:
                correct_pred[classes[label]] += 1
            total_pred[classes[label]] += 1


# print accuracy for each class
for classname, correct_count in correct_pred.items():
    accuracy = 100 * float(correct_count) / total_pred[classname]
    print(f'Accuracy for class: {classname:5s} is {accuracy:.1f} %')
```

    Accuracy for class: plane is 70.6 %
    Accuracy for class: car   is 68.9 %
    Accuracy for class: bird  is 43.7 %
    Accuracy for class: cat   is 27.8 %
    Accuracy for class: deer  is 29.6 %
    Accuracy for class: dog   is 28.6 %
    Accuracy for class: frog  is 85.9 %
    Accuracy for class: horse is 62.1 %
    Accuracy for class: ship  is 62.0 %
    Accuracy for class: truck is 50.0 %



```python
device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')

# Assuming that we are on a CUDA machine, this should print a CUDA device:

print(device)
```


```python
net.to(device)
```




    Net(
      (conv1): Conv2d(3, 6, kernel_size=(5, 5), stride=(1, 1))
      (pool): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
      (conv2): Conv2d(6, 16, kernel_size=(5, 5), stride=(1, 1))
      (fc1): Linear(in_features=400, out_features=120, bias=True)
      (fc2): Linear(in_features=120, out_features=84, bias=True)
      (fc3): Linear(in_features=84, out_features=10, bias=True)
    )




```python
inputs, labels = data[0].to(device), data[1].to(device)
```

<p class="carc-provenance" markdown>Migrated from [UNM-CARC QuickBytes](https://github.com/UNM-CARC/QuickBytes/blob/master/PyTorch_Classifier_Xena%20.ipynb){target=_blank} (last source update 2023-09-20). Spotted a problem? [Open an issue or pull request](https://github.com/UNM-CARC/QuickBytes){target=_blank}.</p>

---8<--- https://unm-carc.github.io/docs/software/pytorch/

---
title: "PyTorch on CARC GPUs"
description: "Install and run GPU-enabled PyTorch on CARC clusters."
type: Guide
tags:
  - Python
  - GPU
  - Machine learning
  - PyTorch
status: draft
generated:
  by: "claude/fable-5"
  at: "2026-08-29T00:00:00Z"
sources:
  - id: quickbytes
    resource: "https://github.com/UNM-CARC/QuickBytes/blob/master/PyTorch_1.9_Xena.md"
    title: "UNM-CARC QuickBytes: PyTorch_1.9_Xena.md"
    author: "team:unm-carc"
    last_modified: "2023-09-20T15:19:14-06:00"
---

# PyTorch on CARC GPUs

### SSH in to the cluster
To connect to a CARC cluster (Hopper in this example), you will need to use the secure shell command below with your username in place of $USERNAME. This will prompt you for your password. When typing your password, you will not get any visual feedback. If you have issues with connecting to the machine, please reach out to the CARC helpdesk. 
```
ssh $USERNAME@hopper.alliance.unm.edu
```

### Create the Condarc File
To create the Condarc file that you will need, use your favorite text editor to create an empty file '.condarc'. For this tutorial I will use nano, which is an easy beginning text editor.
```
nano .condarc
```
The above code creates a blank file using the nano text editor. 

In the condarc file you just created, paste the following then change the $USERNAME prompt to your username.
```
auto_activate_base: false
channels:
  - conda-forge
  - defaults
envs_dirs:
  - /users/$USERNAME/.conda/envs
pkgs_dirs:
  - /users/$USERNAME/.conda/pkgs
```
If you are using nano, notice that at the bottom of the screen below, it lists options for how users can interact with the open file. 

<img width="719" alt="Screenshot 2023-08-29 at 11 01 54 PM" src="https://github.com/UNM-CARC/QuickBytes/assets/130007104/c647a39d-c6d8-437f-917d-f9e1cee8d7e5">

To save the lines of code you just pasted into the file, hold down the control key and press 'O'. This "writes out" or saves the text. You can then exit the text editor by holding down the control key and pressing 'X'. 

To confirm that the file has been edited and contains the correct code, run the command. 

```
cat .condarc 
```

The following text should appear under your command line. 
```
auto_activate_base: false
channels:
  - conda-forge
  - defaults
envs_dirs:
  - /users/{YOUR USERNAME HERE}/.conda/envs
pkgs_dirs:
  - /users/{YOUR USERNAME HERE}/.conda/pkgs
```
If you do not see this text appear, walk through the creation of the condarc file again.

## Navigate to the PyTorch1.9-K40-Compatible directory

Run the following code to change directories into the PyTorch1.9-K40-Compatible directory. 
```
cd /projects/shared/pytorch/PyTorch1.9-K40-Compatible

```
Assuming you have a default command line prompt, this will change your command prompt to reflect the updated directory. If you are not in the PyTorch1.9-K40-Compatible directory, you will run into issues in later steps. To confirm that you are in the correct directory, print your working directory with the following command.
```
pwd

```

### Create the Conda Environment
Now that you are in the correct directory, you'll create your python environment. To create a python environment, you first need to load the miniconda3 module with the following command: (No output should print from this command)
```
module load miniconda3
```
You can then create the python environment by running the following command: 
```
conda env create -f torch-1.9.0+cu11.1-K40.yml 
```
When asked to proceed, enter "y"

## Activate the Conda Environment

Once the environment is created, you will need to activate it before you start the pytorch installation. Activate the environment by running the following: 
```
conda activate pytorch-1.9-cuda-11-K40
```
This command will add a prefix to your command prompt that indicates which environment you are in. Your command line prompt should take the form of: 
```
(pytorch-1.9-cuda-11-K40)[(username)@(hostname) PyTorch1.9-K40-Compatible]$
```
## Pip installation 
To install pytorch, run the following command:
```
pip3 install --user torch-1.9.0+cu11.1-cp39-cp39-linux_x86_64.whl
```
After the installation is complete, we can confirm that the torch install was sucessful in JupyterHub. Follow the link below to the CARC website. 
```
http://carc.unm.edu
Navigate to Systems > JupyterHub Cluster Links > Hopper
```
<img width="1265" alt="Screenshot 2023-09-05 at 9 36 45 PM" src="https://github.com/UNM-CARC/QuickBytes/assets/130007104/eaeb6530-4627-4927-99c2-63b6cb34686f">

Next, log in to JupyterHub with your CARC username and password. If you are logged in without being prompted to select a server, click on the control panel button in the upper right corner. Then select "Stop My Server", then select "Start Server"

You will be prompted to choose a server. For this tutorial, I will choose a server option with GPUs. 
```
Example:
1 hour, 2 GPUs, 16 cores, 60 GB RAM
```
Create a new notebook by selecting new > Python [conda env:.conda-pytorch-1.9-cuda-11-K40]


<img width="1184" alt="Screenshot 2023-09-05 at 9 41 11 PM" src="https://github.com/UNM-CARC/QuickBytes/assets/130007104/1e1505a7-a610-4ed4-a13a-e77d0a922738">


To test that the installation was successful, run the following code.
```
import torch
torch.cuda.device_count()
```
With 2 GPUs, you should get an output of 2.

<p class="carc-provenance" markdown>Migrated from [UNM-CARC QuickBytes](https://github.com/UNM-CARC/QuickBytes/blob/master/PyTorch_1.9_Xena.md){target=_blank} (last source update 2023-09-20). Spotted a problem? [Open an issue or pull request](https://github.com/UNM-CARC/QuickBytes){target=_blank}.</p>

---8<--- https://unm-carc.github.io/docs/software/r-packages/

---
title: "Installing R packages"
description: "Install R packages into your user library on CARC systems."
type: Guide
tags:
  - R
generated:
  by: "claude/fable-5"
  at: "2026-08-29T00:00:00Z"
sources:
  - id: quickbytes
    resource: "https://github.com/UNM-CARC/QuickBytes/blob/master/R_at_CARC/installing_packages.md"
    title: "UNM-CARC QuickBytes: R_at_CARC/installing_packages.md"
    author: "team:unm-carc"
    last_modified: "2019-12-11T11:32:27-07:00"
---

# Installing R packages

## Installing interactively

This is the way that most of us are probably comfortable with installing packages with R since it is exactly the same way you install packages on your laptop. However, because the head node of CARC systems is a shared resource it is best practice to not compile binaries for R for an extended period of time because it can result in overhead on the head node. To avoid this we can request a compute node for interactive use, or even better, one of our debug nodes. The actual call to Slurm, our job scheduler, is explained in more depth later; for now, to request an interactive node type the following at the command prompt:

```
yourusername@hopper$ srun --time=01:00:00 --ntasks=8 --pty bash
```
Which will request a compute node and log you in once it is ready. Load a R software module with your preferred method and start a R session. If this is your first time installing a R package for one of the major versions you will be prompted to use a personal library.

```
R version 3.6.0 (2019-04-26) -- "Planting of a Tree"
Copyright (C) 2019 The R Foundation for Statistical Computing
Platform: x86_64-pc-linux-gnu (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.


> install.packages("ape", dependencies=T, repos="http://cran.r-project.org")
Warning in install.packages("ape", dependencies = T, repos = "http://cran.r-project.org",  :
  'lib = "/opt/spack/opt/spack/linux-centos7-x86_64/gcc-7.3.0/r-3.6.0-7akol5ts7ebhhvoqz2tf6ghmc32hng7g/rlib/R/library"' is not writable
Would you like to use a personal library instead? (yes/No/cancel) 
yes
Would you like to create a personal library
'~/R/x86_64-pc-linux-gnu-library/3.6'
to install packages into? (yes/No/cancel)
yes
```
Go ahead and say `yes` to both questions and install packages normally. You only need to specify a personal library the first time you use a new major version. 

## Installing packages using a script

Installing packages in a script is a little more complicated so it might be easier to just do it interactively. However, it is possible. 

If you haven't created a personal library yet either interactively or through a script you first need to do that. The following commands in an R script will take care of this for you:

```
# First create the directory, .libPaths() will not append your library list unless the directory exists. 
dir.create(Sys.getenv("R_LIBS_USER", recursive=T, mode="0777"))

#Now append your library path with your newly created local library
.libPaths(c(Sys.getenv("R_LIBS_USER"), .libPaths()))

#The above steps are only necessary the first time you are installing packages. Remove or comment out if you have already created a persional library.

#Now install packages normally
> install.packages("ape", dependencies=T, lib=Sys.getenv("R_LIBS_USER"), repos="http://cran.r-project.org")
```

The first two lines are only necessary when you have not created a personal library for that major version of R yet, otherwise you just need to specify the repos you are downloading packages from and specify your personal library as the install location. You shouldn't need to specify the library since you have appended your library path, but it doesn't hurt to be explicit.

<p class="carc-provenance" markdown>Migrated from [UNM-CARC QuickBytes](https://github.com/UNM-CARC/QuickBytes/blob/master/R_at_CARC/installing_packages.md){target=_blank} (last source update 2019-12-11). Spotted a problem? [Open an issue or pull request](https://github.com/UNM-CARC/QuickBytes){target=_blank}.</p>

---8<--- https://unm-carc.github.io/docs/software/r-pbs-jobs/

---
title: "R batch jobs with PBS (retired)"
description: "Historical instructions for submitting R jobs with PBS/Torque, which CARC has replaced with Slurm."
type: Guide
tags:
  - R
  - PBS
  - Legacy
status: deprecated
generated:
  by: "claude/fable-5"
  at: "2026-08-29T00:00:00Z"
sources:
  - id: quickbytes
    resource: "https://github.com/UNM-CARC/QuickBytes/blob/master/R_at_CARC/PBS_job_submission.md"
    title: "UNM-CARC QuickBytes: R_at_CARC/PBS_job_submission.md"
    author: "team:unm-carc"
    last_modified: "2019-12-03T10:18:11-07:00"
---

# R batch jobs with PBS (retired)

!!! warning "Please note"
    CARC schedulers now run Slurm. See [R on CARC systems](r-usage.md) and [Example Slurm scripts](../running-jobs/example-slurm-scripts.md) instead.

## The PBS script
In order to actually submit a job to the cluster you need a PBS script that lists all of the resources you are requesting, the software you want to use, and the commands to that software. So pretend we have an R script called "my_script.R" that you want to run on Wheeler, the PBS script would look something like this:

```
#!/bin/bash

#PBS -N my_r_job
#PBS -l walltime=48:00:00
#PBS -l nodes=1:ppn=8
#PBS -j oe
#PBS -V
#PBS -m abe
#PBS -M my_name@unm.edu

cd $PBS_O_WORKDIR

module load r-3.6.0-gcc-7.3.0-python2-7akol5t

Rscript my_script.R
```
This PBS script would then be submitted to the job queue with 

```
yourusername@wheeler-sn$ qsub my_pbs_script.pbs
```

A PBS script is just a bash script that combines flags for the job scheduler (PBS), and bash commands. We can break it down by sections. 

### PBS directives

If you are unfamiliar with bash scripting the first line `#!/bin/bash/` is called the "hashbang" or "shebang" and is directing your shell on what should be used to interpret the following code. This isn't actually necessary for a PBS script but is just common practice. 

The next chunk of lines are the flags to `qsub` asking for specific resources. The `#` here are important because they represent comments to bash but are interpreted by `qsub`.

```
## The -N flag specifies the name of your job and will be what shows on the queue and the prefix for all 
## PBS specific output.
#PBS -N my_r_job

## The -l flag specifies the actual computational resources you want and can take many arguments. In this case we are 
## asking for 48 hours of walltime on the cluster with one node and all cpu's on that node, which for Wheeler is 8.
#PBS -l walltime=48:00:00
#PBS -l nodes=1:ppn=8

## The -j flag joins your standard out (stdout) and standard error (stderr) into one file. I do this just so there is 
## less output and clutter resulting from a job.
#PBS -j oe

## The -V flag exports user environmental variables from the head node to the compute node. Not always necessary.
#PBS -V

## The -m and -M flags specifiy how you want the scheduler to control mailing information about your job. The -m abe 
## is saying that you want emails for Abort, Begin, and End, and the -M flag is a comma separated list of who
## to send mail to. 
#PBS -m abe
#PBS -M my_name@unm.edu
```

At the very least you should always specify the walltime, nodes, and processors per node for each job. For an exhaustive list of `qsub` commands you can type `man qsub` on the head node of any CARC system. 

### Calling your code
The rest of the PBS script are usually bash commands (or whatever shell you prefer to use, which can be specified with the -S flag) that tells the compute node where your data is, loads the software you want, and then executes your job. 

```
## There are many job specific variables that are created when PBS starts a job, including $PBS_O_WORKDIR. The
## following line moves to the directory where the qsub command was executed.
cd $PBS_O_WORKDIR

## This loads the R software environment.
module load r-3.6.0-gcc-7.3.0-python2-7akol5t

## If you are using a conda environment instead of an R module installed by CARC you would have the following
module load anaconda3
source activate my_r_env

## This line is where you are actually running your R script. The older way of using R CMD BATCH is deprecated
## and Rscript is the preferred way for launching an R batch job.
Rscript my_script.R
```

<p class="carc-provenance" markdown>Migrated from [UNM-CARC QuickBytes](https://github.com/UNM-CARC/QuickBytes/blob/master/R_at_CARC/PBS_job_submission.md){target=_blank} (last source update 2019-12-03). Spotted a problem? [Open an issue or pull request](https://github.com/UNM-CARC/QuickBytes){target=_blank}.</p>

---8<--- https://unm-carc.github.io/docs/software/r-usage/

---
title: "R on CARC systems"
description: "Load R, run scripts in batch jobs, and use R interactively on CARC clusters."
type: Guide
tags:
  - R
generated:
  by: "claude/fable-5"
  at: "2026-08-29T00:00:00Z"
sources:
  - id: quickbytes
    resource: "https://github.com/UNM-CARC/QuickBytes/blob/master/R_usage.md"
    title: "UNM-CARC QuickBytes: R_usage.md"
    author: "team:unm-carc"
    last_modified: "2019-03-20T13:46:58-06:00"
---

# R on CARC systems

R Programming in HPC
====================

## What is R?
R is a programming language and a software environment for statistical computing and graphics techniques. This is widely used now a days by statisticians and data miners for developing software tools required for data analysis. The R language is primarily derived from the S language developed at Bell Laboratories in 1975. R provides various tools and techniques for linear and nonlinear modelling, statistical tests, time series analysis, classification, clustering etc. More history and documentation of R are available at this [link](https://cran.r-project.org/manuals.html){target=_blank}
## How to run R?
First login to one of the CARC machines via SSH.

`ssh -X user@machine_name.alliance.unm.edu`

Once logged into the machine, you have to load the module which has R program files. 

`module load r-3.5.0-gcc-4.8.5-python2-khqxja7`

After loading the R module, begin R programming by typing

`R`

This will shows

```bash

R version 3.5.0 (2018-04-23) -- "Joy in Playing"
Copyright (C) 2018 The R Foundation for Statistical
Computing
Platform: x86_64-pc-linux-gnu (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

  Natural language support but running in an English locale

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

```
This means the R module is loaded and you are ready to use R for your research
## Running a sample script example.R 
Let us look a sample script, example.R which will print "Hello World"
The script looks like 

```bash
# Program to print Hello World

print('Hello World')
```
In order to execute this script, type

`Rscript example.R`

This will give an output of

`[1] "Hello World"`

If you want to generate samples from a normal distribution, you can use the `rnorm()` function. Let's make some change to the example.R script. 

```bash
# Program to print Hello World and to generate normal random numbers

print('Hello World')

#Calling the rnorm() function to generate three random numbers from a 
normal distribution with mean 5 and a standard  deviation of 5

rnorm(3,mean=5,sd=5)
```

Execute the script again and the look at the new output.

```bash
[1] "Hello World"
[1] 3.744463 7.954163 3.363275
```

<p class="carc-provenance" markdown>Migrated from [UNM-CARC QuickBytes](https://github.com/UNM-CARC/QuickBytes/blob/master/R_usage.md){target=_blank} (last source update 2019-03-20). Spotted a problem? [Open an issue or pull request](https://github.com/UNM-CARC/QuickBytes){target=_blank}.</p>

---8<--- https://unm-carc.github.io/docs/software/singularity/

---
title: "Singularity / Apptainer containers"
description: "Build, pull, and run software containers on CARC clusters."
type: Guide
tags:
  - Containers
  - Singularity
generated:
  by: "claude/fable-5"
  at: "2026-08-29T00:00:00Z"
sources:
  - id: quickbytes
    resource: "https://github.com/UNM-CARC/QuickBytes/blob/master/singularity-markdown-version.md"
    title: "UNM-CARC QuickBytes: singularity-markdown-version.md"
    author: "team:unm-carc"
    last_modified: "2020-03-26T09:59:47-06:00"
---

# Singularity / Apptainer containers

## Introduction

Have you ever installed software on your own system to process data, but then find it is not so easy to setup the same environment on a CARC cluster when you want to process larger data sets? CARC staff will work with you to get your software running, but occasionally we run across fundamental incompatibilities. Docker allows us to circumvent those incompatibilities.

Docker and Singularity are free tools that allow you to run software with dependencies on its environment that are difficult to satisfy natively on the CARC clusters. For example, your software might depend on a particular flavor of linux that differs from the one installed on the cluster you want to use. Perhaps your software needs to make global changes to the operating system that are incompatible with the needs of the center or other users. Docker allows us to deploy software with these requirements.

In this guide you will learn how to setup your software to run in a docker container, and how to convert the container to singularity and run it at CARC. The R software package party.


## What is Docker?

Docker allows you to setup a virtual environment for your program that you can configure however you like. You can choose the underlying operating system (so long as it is linux based), install any packages you need, and make any other changes the root user could make. The custom OS environment is stored in a docker image file that can be loaded by any docker installation. Docker has online repositories with many pre-built images available to download.

When loaded, a docker image provides a container that allows the software to have complete control of its environment without effecting the host operating system.

If you have used a virtual machine (such as virtualbox, or vmware) the description of docker images will sound familiar. The main difference is that docker just containerizes the environment but still uses the host operating system's kernel. This means there is very little performance impact.

## What is Singularity?

Singularity is able to convert and run docker images into a secure form suitable for multiuser machines such as the CARC clusters.

## Installing Docker

Docker runs on Microsoft Windows, Apple OS X, and Linux. All the systems at CARC are linux based so you will need to create and configure a linux based docker container. OS X and Linux will allow you to do this. Windows 10 can as well but you need to install the Windows Subsystem for Linux and a linux distribution.

Docker can be downloaded from www.docker.com.

## Running Docker

To start docker under Windows or OS X just launch the program like you would any other. To start the docker daemon under linux enter:

`sudo systemctl start docker`

## Creating a Docker Image

### Pulling an existing base docker image

For this example we are going to use an existing docker image that has the CentOS linux distribution preconfigured. We could just as easily use Ubuntu by replacing "centos" with "ubuntu" below.

```bash
docker pull centos
Using default tag: latest Trying to pull
repository docker.io/library/centos ...
latest: Pulling from docker.io/library/centos
a02a4930cb5d: Pull complete
Digest:sha256:184e5f35598e333bfa7de10d8fb1cebb5ee4df5bc0f970bf2b1e7c7345136426
Status: Downloaded newer image for docker.io/centos:latest
```

We can now issue a command that runs inside the docker container and returns output. Below we run the `ls -la` command to get a file listing inside the container.

```
docker run centos ls
anaconda-post.log
bin
dev
etc
home
lib
lib64
media
mnt
opt
proc
root
run
sbin
srv
sys
tmp
usr
var
```

### Customizing the docker image

Now we can install the program we need inside the docker container and any dependencies it needs. It is convenient to do this interactively inside the container. Configuring the container in interactive mode allows us to do everything we need to do before saving the changes to a docker image. If we issued commands one-by-one with run we would lose the state of the container after each command.

```
docker run -it centos

[root@a264d0f7b8c9 /]#
```

The long string after `@` is the name of the container we are running in. Notice it is not the same name as the docker image.

In a new terminal we can list the running docker containers and see that our container is running.

```
docker container list
CONTAINER ID  IMAGE   COMMAND      CREATED             STATUS  		PORTS  NAMES
a264d0f7b8c9  centos  "/bin/bash"  About a minute ago  Up About a minute  striking_mahavira
```

### Example: Installing the "party" R package

As an example, we will install party. Party is an R package for recursive partitioning. It produces regression trees for machine learning. But you can install whatever tools you wish. This is the beauty of docker, it gives you control over your environment.

```
[root@a264d0f7b8c9 /]# yum update 
[root@a264d0f7b8c9 /]# yum -y install epel-release 
[root@a264d0f7b8c9 /]# yum install R
```

Once R has finished installing:

```
[root@a264d0f7b8c9 /]# R

R version 3.5.0 (2018-04-23) -- "Joy in Playing"
Copyright (C) 2018 The R Foundation for Statistical Computing
Platform: x86_64-pc-linux-gnu (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

Natural language support but running in an English locale

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

>install.packages("party",deps=YES)
```

## Commit customised container to image file

Once party has finished installing, our Docker container is ready. Now we need to commit those changes to a docker image file so we can load the image that has R and party installed for future use.

Exit R:

```
[root@a264d0f7b8c9 /]# > quit()
```

Exit the docker container:

In general:

```
[root@a264d0f7b8c9 /]# exit
$ sudo docker commit
```

For our example:

```
[root@a264d0f7b8c9 /]# exit
$ sudo docker commit a264d0f7b8c9 r_party
```

Now we have a docker image saved.

## Running Commands using the Docker Image

Now that we have a docker image we can run it on our own machines as before with:

```
docker run r_party ls
```

But now we can also execute R commands such as: 

```
docker run r_party Rscript
```

For our party example we can write the following R script and save it to a file called test_party.R:

```
library("party")
set.seed(290875)
### honest (i.e., out-of-bag) cross-classification of
### true vs. predicted classes
data("mammoexp", package = "TH.data")
table(mammoexp$ME, predict(cforest(ME ~ ., data = mammoexp,
control = cforest_unbiased(ntree = 50)),
OOB = TRUE))
### fit forest to censored response
if (require("TH.data") && require("survival")) {
data("GBSG2", package = "TH.data")
bst <- cforest(Surv(time, cens) ~ ., data = GBSG2,
control = cforest_unbiased(ntree = 50))
### estimate conditional Kaplan-Meier curves
treeresponse(bst, newdata = GBSG2[1:2,], OOB = TRUE)
party:::prettytree(bst@ensemble[[1]], names(bst@data@get("input")))
```

And run it using the docker image with:

```
$ docker run -v < path to test_party.R folder > :/mnt r_party Rscript /mnt/test_party.R
```

# Singularity

## Converting Docker Images to Singularty Images

Once we are happy with the docker image we created we will convert it to a singularity image so we can use it on the CARC clusters.

We do the conversion by using a docker image provided to us that contains the necessary tools:

```
$ docker pull singularityware/docker2singularity
$ docker run -v /var/run/docker.sock:/var/run/docker.sock -v /tmp:/output --privileged -t --rm singularityware/docker2singularity r_party
```

This will produce a singularity image in the /tmp directory that we can upload to a CARC cluster using our favorite file transfer program.

## Running a Singularity Image at CARC

First login to a CARC cluster head node.

Next we will load the singularity module:

```
$ module load singularity-2.4.1-intel-17.0.4-sjwoqj4 $
```

The syntax for executing Singularity images are similar to those we used for docker:

```
$ singularity exec r_party.simg Rscript test_party.R
```

## Mapping Directories

```
$ singularity -B $PBS_O_WORKDIR:/mnt exec r_party.simg Rscript test_party.R
```
The Above command maps the directory that that PBS script was submitted from, `$PBS_O_WORKDIR`, to the `/mnt` location within the `r_party.simg` singularity image and then executes the `test_party.R` script. 

# Version Issues

If you recieve the following when you try to execute your singularity image there may be a mismatch between the version you used to create the image and the singularity version you loaded at CARC. 

ERROR  : Failed to mount image in (read only): Invalid argument

ABORT  : Retval = 255

We have several modules with different singularity versions. Enter the following command to see them all: 
```module avail singularity```

<p class="carc-provenance" markdown>Migrated from [UNM-CARC QuickBytes](https://github.com/UNM-CARC/QuickBytes/blob/master/singularity-markdown-version.md){target=_blank} (last source update 2020-03-26). Spotted a problem? [Open an issue or pull request](https://github.com/UNM-CARC/QuickBytes){target=_blank}.</p>

---8<--- https://unm-carc.github.io/docs/software/spark/

---
title: "Apache Spark"
description: "Launch Apache Spark clusters inside Slurm allocations for large-scale data analysis."
type: Tutorial
tags:
  - Spark
  - Big data
  - Parallel
generated:
  by: "claude/fable-5"
  at: "2026-08-29T00:00:00Z"
sources:
  - id: quickbytes
    resource: "https://github.com/UNM-CARC/QuickBytes/blob/master/spark_tutorial.md"
    title: "UNM-CARC QuickBytes: spark_tutorial.md"
    author: "team:unm-carc"
    last_modified: "2026-07-07T11:02:40-06:00"
---

# Apache Spark

Apache Spark is a distributed computing framework for processing large data sets, generally easier to program than something like MPI. You write a single Python (or Java/Scala) program that coordinates parallel work across many worker processes.

This tutorial assumes you're comfortable with Slurm, modules, and HPC basics, but new to Spark. Each step explains *why* it's needed — Spark's main complication on a shared HPC cluster isn't writing Spark code, it's standing up your own personal Spark cluster inside Slurm's allocation system. The one file you need from this repo, `slurm-spark-submit`, is at <https://github.com/Graviton28/QuickBytes/tree/master/spark> — everything else below you create yourself with a `cat` command as you go.

---

## Why Spark Needs Special Handling Here

Most jobs you run on Easley are a single program Slurm starts and stops. Spark expects to run as its own long-lived cluster — one master process plus one or more workers, coordinating over the network. But Slurm only gives you nodes temporarily, and which nodes you get changes every time. The `slurm-spark-submit` script below exists to solve exactly this: every time you get an allocation, it builds a correctly-configured Spark cluster on whatever nodes you were given, automatically.

---

## The Spark Model, Briefly

- You write **one program** that runs on a **driver/master** node.
- The driver splits work across **worker** processes on other nodes.
- Data lives in **RDDs** (distributed collections) or, more commonly, **DataFrames** (RDDs with named columns, SQL-like operations, and native CSV/JSON/Parquet support).

The one rule that explains most Spark behavior:

- **Transformations** (`map`, `filter`, `groupBy`) are lazy — Spark just records what it'll need to do, then runs it in parallel across workers once forced to.
- **Actions** (`collect`, `count`, `show`) trigger that execution. `collect()` specifically pulls results back to the single driver process — collecting something huge can blow out driver memory even if it was fine distributed across workers.

## Word Count Example

```python
from pyspark.sql import SparkSession
from operator import add
import sys

spark = SparkSession.builder.appName("WordCount").getOrCreate()

# Transformations: lazy, distributed, nothing executes yet
lines = spark.read.text(sys.argv[1]).rdd.map(lambda r: r[0])
words = lines.flatMap(lambda line: line.split())
counts = words.map(lambda word: (word, 1)).reduceByKey(add)

# Action: this is where execution actually happens
output = counts.collect()

# Plain Python now, running only on the driver
for word, count in sorted(output, key=lambda x: -x[1])[:20]:
    print(f"{word}: {count}")
```

---

## Step 1: Load the Correct Spark Module

```bash
module load spark/3.5.1-kn2k
```

Easley has two Spack builds of Spark 3.5.1: one with Hadoop support (`kn2k`) and one without (`sewd`). Spark's standalone cluster mode needs Hadoop's jars on the classpath even if you never touch HDFS — without them you get `NoClassDefFoundError: org/slf4j/Logger` immediately on startup. **The generic `module load spark` loads the broken `sewd` build by default.** Always specify the version.

## Step 2: Fix SPARK_HOME

```bash
export SPARK_HOME=$SPARK_ROOT
```

Spark's own scripts (`start-master.sh`, `spark-submit`, etc.) expect `$SPARK_HOME`. The Easley module only sets `$SPARK_ROOT`. Skip this and nothing Spark-related can find its own installation.

## Step 3: Match Driver and Worker Python Versions

```bash
module load miniforge3
conda create -n spark-env python=3.11 numpy scipy pandas matplotlib pyarrow -y
conda activate spark-env
export PYSPARK_PYTHON=~/.conda/envs/spark-env/bin/python
export PYSPARK_DRIVER_PYTHON=~/.conda/envs/spark-env/bin/python
```

PySpark runs your code on both the driver and every worker, and they must run the same minor Python version. Easley's system Python is 3.9; if your conda env is 3.11, you'll hit:
```
PySparkRuntimeError: [PYTHON_VERSION_MISMATCH] Python in worker has
different version (3, 9) than that in driver 3.11
```
`PYSPARK_DRIVER_PYTHON` controls the driver's interpreter, `PYSPARK_PYTHON` controls the workers' — you need both set, not just one.

---

## Step 4: What `slurm-spark-submit` Actually Does

Spark standalone clusters normally expect fixed, known hostnames — you SSH to one node and start a master, then SSH to every other node to start workers pointed at it. That doesn't work when your nodes change every allocation. The script automates this:

1. Loads the module and fixes the environment variables from Steps 1–3.
2. Asks Slurm for the current node list: `scontrol show hostnames "$SLURM_JOB_NODELIST"`.
3. Starts the master directly on the first node.
4. SSHes into every allocated node and starts a worker — explicitly passing `SPARK_HOME`, `JAVA_HOME`, `SPARK_DIST_CLASSPATH`, and `PYSPARK_PYTHON` into each session, since a fresh `ssh` session doesn't inherit your shell's environment. Skip this and every worker hits the same classpath and Python errors all over again.
5. Redirects all Spark logs/work/pid directories to `/tmp/spark-$SLURM_JOB_ID/`, since the shared Spark install directory is read-only.
6. If you pass it a script, runs it with `spark-submit` and tears the cluster down afterward (for use inside `sbatch`). If not, it leaves the cluster running for interactive use and prints the master URL.

---

## Step 5: Interactive Use

```bash
salloc --nodes=1 --ntasks-per-node=1 --cpus-per-task=4 --mem=16G --time=00:30:00 --partition=general
module load spark/3.5.1-kn2k
./slurm-spark-submit
```

Note the printed `MASTER_URL` (e.g. `spark://easley002:7077`) — it changes every allocation since it depends on which node you got. Connect with:

```bash
export PYSPARK_PYTHON=~/.conda/envs/spark-env/bin/python
export PYSPARK_DRIVER_PYTHON=~/.conda/envs/spark-env/bin/python
pyspark --master spark://easley002:7077
```

```python
>>> sc.parallelize(range(100)).sum()
```
Expected output:
```
4950
```
This forces an actual distributed computation and round-trip, confirming master, worker, networking, and Python compatibility all work — not just that processes exist.

---

## Step 6: Batch Jobs

This needs three files: the word-count program, a small input file, and the Slurm submission script. Create all three:

```bash
cat > wordcount.py <<'EOF'
#!/usr/bin/env python3
"""
wordcount.py — Simple Spark word count example.
Compatible with Spark 3.5+ and Python 3.9+.

Usage:
    spark-submit --master <master_url> wordcount.py <input_file>
"""

import sys
from operator import add
from pyspark.sql import SparkSession


def main():
    if len(sys.argv) < 2:
        print(f"Usage: {sys.argv[0]} <input_file>", file=sys.stderr)
        sys.exit(1)

    input_path = sys.argv[1]

    spark = (
        SparkSession.builder
        .appName("WordCount")
        .getOrCreate()
    )
    spark.sparkContext.setLogLevel("WARN")

    lines = spark.read.text(input_path).rdd.map(lambda r: r[0])
    words = lines.flatMap(lambda line: line.split())
    counts = words.map(lambda word: (word, 1)).reduceByKey(add)

    output = counts.collect()
    for word, count in sorted(output, key=lambda x: -x[1])[:20]:
        print(f"{word}: {count}")

    spark.stop()


if __name__ == "__main__":
    main()
EOF
```

```bash
cat > big.txt <<'EOF'
the quick brown fox jumps over the lazy dog the fox the
the quick brown fox jumps over the lazy dog the fox the
the quick brown fox jumps over the lazy dog the fox the
the quick brown fox jumps over the lazy dog the fox the
EOF
```

```bash
cat > wordcount.sh <<'EOF'
#!/bin/bash
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
#SBATCH --cpus-per-task=4
#SBATCH --mem=16G
#SBATCH --time=00:10:00
#SBATCH --output=wordcount_%j.log
#SBATCH --partition=general

module load spark/3.5.1-kn2k
export SPARK_HOME=$SPARK_ROOT

SCRATCHDIR="/tmp/spark-${SLURM_JOB_ID}"
mkdir -p "$SCRATCHDIR"
cp "$SLURM_SUBMIT_DIR/wordcount.py" "$SCRATCHDIR/"
cp "$SLURM_SUBMIT_DIR/big.txt"      "$SCRATCHDIR/"
cd "$SCRATCHDIR"

LOGFILE="wordcount_${SLURM_JOB_ID}.log"

bash "$SLURM_SUBMIT_DIR/slurm-spark-submit" \
    wordcount.py big.txt > "$LOGFILE" 2>&1

cp "$LOGFILE" "$SLURM_SUBMIT_DIR/"
EOF
```

Standard pattern: copy inputs to local `/tmp` scratch for faster I/O, `$SLURM_SUBMIT_DIR` tracks where you originally ran `sbatch` from since the script `cd`s away from it, and results get copied back at the end. The log is named with `$SLURM_JOB_ID` so repeated runs don't overwrite each other's output. Submit and monitor as usual:
```bash
sbatch wordcount.sh
squeue --me
```

---

## Step 7: Multi-Node Scaling

```bash
salloc --nodes=2 --ntasks-per-node=1 --cpus-per-task=4 --mem=16G --time=00:30:00 --partition=general
./slurm-spark-submit
```

Nothing else changes — the script re-queries Slurm's node list every run, so it scales from 1 to N nodes with no edits.

One quirk on this Spack build: master/worker `.out` logs in `/tmp/spark-$SLURM_JOB_ID/logs/` may show only the Java launch line with nothing after it. That's a logging config quirk, not a failure. Verify health with process/port checks instead:
```bash
ps aux | grep -E "Master|Worker" | grep -v grep
ss -tlnp | grep 7077
```

To confirm work actually spread across nodes, not just that processes exist:
```python
>>> sc.parallelize(range(1000), 32).mapPartitions(lambda it: [sum(1 for _ in it)]).collect()
```
Expected output:
```
[31, 31, 31, 32, ...]
```
32 nonzero partition counts means the data was actually split and processed across your workers.

---

## Step 8: DataFrames and Plotting

Create `monthly_counts.py`, which generates a small synthetic dataset directly in Spark (no external file needed), aggregates it there, then brings only the small result back to Pandas to plot:

```bash
cat > monthly_counts.py <<'EOF'
import matplotlib
matplotlib.use("PDF")  # no display on compute nodes
import matplotlib.pyplot as plt
import pandas as pd
from pyspark.sql import SparkSession
from pyspark.sql import functions as F

spark = SparkSession.builder.appName("MonthlyCounts").getOrCreate()

# Synthetic dataset: 5,000 rows, each with a random date in 2024
df = spark.range(5000).select(
    F.date_add(F.lit("2024-01-01"), (F.rand(seed=42) * 365).cast("int")).alias("Date")
)

monthly = (
    df.withColumn("Month", F.month("Date"))
    .groupBy("Month").count()
    .orderBy("Month")
    .collect()          # only action in the chain — result is tiny by now
)

pdf = pd.DataFrame(monthly, columns=["month", "record_count"])
pdf.plot(figsize=(20, 10), kind="line", x="month", y="record_count")
plt.savefig("monthly-counts.pdf")
EOF
```

The grouping/counting happens distributed across all workers before anything leaves the cluster; only the final handful of rows gets pulled into Pandas/Matplotlib, which are single-machine tools never meant to handle the full raw dataset directly.

Run it against the cluster you already have up from Step 5 or 7:

```bash
spark-submit --master spark://<node>:7077 monthly_counts.py
ls -la monthly-counts.pdf
```

Since the dataset is generated with a fixed seed, the counts are deterministic — grouping by `Month` should come out near `5000/12 ≈ 417` per month every run.

---

## Known Issues on Easley

| Symptom | Cause | Fix |
|---|---|---|
| `NoClassDefFoundError: org/slf4j/Logger` | Default `module load spark` loads the Hadoop-less `sewd` build | `module load spark/3.5.1-kn2k` |
| Spark scripts can't find their install | Module sets `$SPARK_ROOT`, not `$SPARK_HOME` | `export SPARK_HOME=$SPARK_ROOT` |
| `PYTHON_VERSION_MISMATCH` | conda env Python ≠ system Python (3.9) used by workers | Set both `PYSPARK_PYTHON` and `PYSPARK_DRIVER_PYTHON` |
| Worker/master logs look empty | Logging config quirk in this build | Check `ps aux` / `ss -tlnp` instead of logs |

---

## Further Reading

- **Spark Streaming** — same transformation/action model applied to continuously-arriving data in time-windowed batches.
- **Spark ML** — built-in distributed ML algorithms using the same partitioning model.
- **Spark SQL** — run actual SQL against DataFrames via `spark.sql(...)`.
- Official docs: <https://spark.apache.org/docs/3.5.1/>

<p class="carc-provenance" markdown>Migrated from [UNM-CARC QuickBytes](https://github.com/UNM-CARC/QuickBytes/blob/master/spark_tutorial.md){target=_blank} (last source update 2026-07-07). Spotted a problem? [Open an issue or pull request](https://github.com/UNM-CARC/QuickBytes){target=_blank}.</p>

---8<--- https://unm-carc.github.io/docs/software/tensorflow-multi-gpu/

---
title: "Multi-GPU TensorFlow"
description: "Distribute TensorFlow training across multiple GPUs on a CARC node."
type: Tutorial
tags:
  - Python
  - GPU
  - Machine learning
  - TensorFlow
generated:
  by: "claude/fable-5"
  at: "2026-08-29T00:00:00Z"
sources:
  - id: quickbytes
    resource: "https://github.com/UNM-CARC/QuickBytes/blob/master/multiGPU_tensorflow_tutorial.md"
    title: "UNM-CARC QuickBytes: multiGPU_tensorflow_tutorial.md"
    author: "team:unm-carc"
    last_modified: "2026-06-24T10:55:01-06:00"
---

# Multi-GPU TensorFlow

The relatively recent mainstream availability of complex algorithms and computationally efficient hardware has created a platform for innovations never before available to the scientific computing world. Since the development of early computer systems, computing times have been drastically reduced, making more complex computations feasible. The continuous cycle of improvements in computation speed and hardware, driving ever more complex computation goals, can be seen in how hardware has scaled to meet those goals. A key part of this cycle is the use of GPUs in high-performance computing for machine learning and deep learning algorithms.

Most complex computing strategies can be simplified into basic linear algebra operations such as addition, multiplication, subtraction, and inversion. Of these, matrix multiplication and inversion are the most computationally expensive.

Most matrix operations are performed sequentially on the CPU, resulting in computation time that scales with the size of the matrix by a factor of &theta;(n<sup>3</sup>). As a result, the time required for computation is proportional to matrix size, constrained further by limited cache memory and RAM. This same problem persists with multicore or distributed systems due to those same resource thresholds. A GPU, on the other hand, is composed of several thousand cores, providing several GBs of computational memory compared to the MBs available in CPU cache. This configuration enables parallelism across GPU cores and a much higher data bandwidth, massively reducing computation time as the device scales its performance with data size.

These gains in computation time give researchers good reason to move computationally heavy operations from CPUs to GPUs, particularly where CPU-based operations don't scale with data at a constant rate. This benefits computationally heavy domains such as machine learning, deep learning, linear algebra, optimization, and data structures broadly.

### CARC Benchmarks

To illustrate the CPU-vs-GPU performance gap, here are benchmarks run on a previous CARC GPU system using intensive linear algebra operations — matrix multiplication and matrix inversion — comparing CPU-only execution to GPU-accelerated execution.

The CPU version was deployed on a multicore processor with 16 cores and 64GB of RAM, using NumPy arrays in Python. The GPU version was deployed on an NVIDIA Tesla K40 with 11GB of GPU memory, using TensorFlow. The CPU implementations were tested with two different NumPy builds: `mkl_mul` refers to multiplication using NumPy compiled with Intel's Math Kernel Library (MKL), while `nomkl_mul` refers to NumPy without MKL. MKL-based NumPy was installed in a Conda environment, whereas NumPy installed via pip does not integrate MKL.

That system had nodes with both single- and dual-GPU configurations. A dual-GPU node offered 2×11GB of GPU memory, allowing larger batch sizes and roughly double the cores for faster training of larger, more complex models. `gpu_mul` corresponds to multiplication on a single-GPU node, and `dualgpu_mul` corresponds to multiplication on a dual-GPU node. A similar benchmark was run for matrix inversion.

![](../assets/images/quickbytes/matrix_inverse.png)

Fig 1. Time for matrix inversion vs. size of matrix N

![](../assets/images/quickbytes/matrix_multiplication.png)

Fig 2. Time for matrix multiplication vs. size of matrix N

The implementation code for these benchmarks can be found [here](https://github.com/ceodspspectrum/CARC_WORK/tree/master/master){target=_blank}.

### TensorFlow Basics

TensorFlow is an open-source deep learning library originally developed by Google. It provides primitives for defining functions over tensors and automatically computing their derivatives. A tensor represents any multidimensional array of numbers, similar in spirit to a NumPy array.

**Comparing NumPy and TensorFlow**

Both libraries store data in N-dimensional arrays — NumPy's `ndarray` and TensorFlow's `tf.Tensor`. However, NumPy doesn't support automatic differentiation or GPU acceleration. For workloads that need either of those — like training neural networks — TensorFlow's GPU support and built-in autograd typically make it the better choice, especially as data dimensionality grows.

**NumPy vs. TensorFlow: Matrix Addition**

***NumPy:***
```python
import numpy as np

a = np.zeros((2, 2))
b = np.zeros((2, 2))
np.sum(b, axis=0)
a.shape
np.reshape(b, (1, 4))
```

***TensorFlow (2.x, eager execution):***
```python
import tensorflow as tf

a = tf.zeros((2, 2))
b = tf.ones((2, 2))
tf.reduce_sum(b, axis=1)
a.shape
tf.reshape(b, (1, 4))
```

Unlike older versions of TensorFlow, TensorFlow 2.x uses **eager execution** by default — operations run and return values immediately, just like NumPy, with no separate "session" step required.

For example, in NumPy:
```python
a = np.zeros((2, 2))
print(a)
```
This immediately prints the value of `a`. In modern TensorFlow, the same is true:
```python
a = tf.zeros((2, 2))
print(a)
```
This also prints the value of `a` right away — no `.eval()` or session needed. (Older TensorFlow 1.x code required wrapping everything in a `tf.Session()` and explicitly calling `.eval()` or `sess.run()` to get a value; that pattern is obsolete in TensorFlow 2.x.)

**TensorFlow Variables**

Like other programming languages, TensorFlow uses a `Variable` object to store and update parameters that change during training (e.g. model weights). In TensorFlow 2.x, variables are initialized immediately when created — no separate initialization step is needed:

```python
import tensorflow as tf

W = tf.Variable(tf.zeros((2, 2)), name="weights")
R = tf.Variable(tf.random.normal((2, 2)), name="random_weights")

print(W)
print(R)
```

**Converting NumPy Data to a Tensor**

```python
import numpy as np
import tensorflow as tf

a = np.zeros((3, 3))
t_a = tf.convert_to_tensor(a)
print(t_a)
```

**Functions and Custom Operations**

Older TensorFlow code used `tf.placeholder` to define inputs that were filled in later via a `feed_dict`. In TensorFlow 2.x, you simply write a regular Python function — optionally decorated with `@tf.function` for performance — and call it directly with your data:

```python
import tensorflow as tf

@tf.function
def multiply(input1, input2):
    return tf.multiply(input1, input2)

result = multiply(7.0, 2.0)
print(result)
```

This replaces the old pattern of defining `tf.placeholder` variables and feeding them through a `tf.Session()`.

### Learning More TensorFlow

Rather than reproduce a full general-purpose TensorFlow walkthrough here, we recommend going straight to the source: the [official TensorFlow tutorials](https://www.tensorflow.org/tutorials){target=_blank} maintained by Google. These are kept up to date with the current TensorFlow API and run as ready-to-use Jupyter/Colab notebooks with no local setup required.

A good starting point is the [TensorFlow 2 quickstart for beginners](https://www.tensorflow.org/tutorials/quickstart/beginner){target=_blank}, which walks through loading a dataset, building a simple Keras model, and training/evaluating it. For a deeper, lower-level walkthrough, the [quickstart for experts](https://www.tensorflow.org/tutorials/quickstart/advanced){target=_blank} covers the same task using TensorFlow's more customizable API.

> **Note:** if you've used TensorFlow before and your code still uses `tf.Session()`, `tf.placeholder`, or `tf.initialize_all_variables()` — that's the TensorFlow 1.x API, fully superseded by eager execution in TensorFlow 2.x. See Google's [Effective TensorFlow 2](https://www.tensorflow.org/guide/effective_tf2){target=_blank} guide and the official [migration guide](https://www.tensorflow.org/guide/migrate){target=_blank} if you need to update older code.

*This quickbyte was validated on 6/22/2026*

<p class="carc-provenance" markdown>Migrated from [UNM-CARC QuickBytes](https://github.com/UNM-CARC/QuickBytes/blob/master/multiGPU_tensorflow_tutorial.md){target=_blank} (last source update 2026-06-24). Spotted a problem? [Open an issue or pull request](https://github.com/UNM-CARC/QuickBytes){target=_blank}.</p>

---8<--- https://unm-carc.github.io/docs/software/tensorflow/

---
title: "TensorFlow on CARC GPUs"
description: "Install and run GPU-enabled TensorFlow on CARC clusters."
type: Guide
tags:
  - Python
  - GPU
  - Machine learning
  - TensorFlow
generated:
  by: "claude/fable-5"
  at: "2026-08-29T00:00:00Z"
sources:
  - id: quickbytes
    resource: "https://github.com/UNM-CARC/QuickBytes/blob/master/Tensorflow_documentation.md"
    title: "UNM-CARC QuickBytes: Tensorflow_documentation.md"
    author: "team:unm-carc"
    last_modified: "2019-03-06T12:37:10-07:00"
---

# TensorFlow on CARC GPUs

The relatively recent mainstream availability of complex algorithms and computationally efficient hardware is creating a platform for new innovations never before available to the scientific computing world. Since the development of computer systems, computing times have been drastically reduced, making more complex computations feasible. The continuous cycle of improvements in computation speeds and hardware leading to ever more complex computations can be seen in the scaling of hardware to meet these more complex computation goals. A resolution to this cycle can be found in the utilization of GPU's in high-performance computing for Machine learning and deep learning algorithms.
Most of the complex computing strategies can be simplified into basic linear algebra operations such as addition, multiplication, subtraction, inversion and such. Out of the listed operations, matrix multiplication and inversion are the most computationally expensive operations.

Most matrix operations are performed sequentially on the CPU resulting in computation time that scales with the size of the matrix as a factor &theta;(n<sup>3</sup>). Hence, computation cycles and duration of time to be allocated towards computation is proportional to the size of matrices under the constrained hardware with limited cache memory and RAM. The same problem still exists with multicore systems or distributed systems due to the threshold on the resources mentioned. On the other hand, a GPU is composed of several thousand cores, combining to provide the user with several GBs of computational memory compared to the MBs provided by CPU cache memory. The distributed computing this configuration provides enables parallelism across GPU cores and allows a super fast flow of data resulting from incredibly high bandwidth. This distribution across multiple cores amounts to massively reduced computation time as the device is able to scale its performance with data size.

The enormous gains in computation time should give researchers a valid reason to switch from CPUs to GPUs for computationally heavy operations where the CPU-based operations do not scale with the data at a constant rate. Utilization of this methodology will provide enormous benefits for the computationally heavy domains such as machine learning, deep learning, linear algebra, optimization, data structures, etc. To illustrate this statement, we've included some benchmarks run on a previous CARC GPU system using intensive linear algebra operations, i.e., matrix multiplication and matrix-inversion, on a CPU only compared to CPU utilizing the GPU as well.

The CPU version was deployed on a multicore processor with 16 cores and 64GB of RAM with using numpy arrays in python. The GPU Version was deployed on NVIDIA Tesla K40 with 11GB of GPU memory using Tensorflow.  Here the CPU implementations were carried with two different types of numpy compilations. mkl_mul stands for multiplication operation carried with numpy compiled with math kernel library. Nomkl_mul stands for the numpy without math kernel library. mkl based numpy was installed in an anaconda enviroment using conda to install numpy whereas the numpy installed with pip doesn't integrate math kernel library. That system had nodes with a single GPU and dual GPUs. Dual GPU node offers users 2x11GB of computational GPU memory which allows the use of a larger batch size and double the number of cores for faster implementation of highly complex models with a large number of parameters. The GPU_mul corresponds to multiplication operations utilizing single GPU node and dualgpu_mul corresonds to the one utilizing a dual GPU node. Another interesting benchmark was performed for inversion operations similar to those done for multiplication

![](https://raw.githubusercontent.com/ceodspspectrum/CARC_WORK/master/download2.png)

Fig 1. Time for Matrix Inversion vs size of Matrix N


![](https://raw.githubusercontent.com/ceodspspectrum/CARC_WORK/master/download1.png)

Fig 2. Time for Matrix Multiplication vs size of Matrix N

The implementations can be found [here.](https://github.com/ceodspspectrum/CARC_WORK/tree/master/master){target=_blank}


Tensorflow is an open source deep learning library provided by Google. It provides primitives for functions definitions on tensor and a mechanism to compute their derivatives automatically. It uses a tensor to represent any multidimensional array of numbers.

**Comparision between Numpy and Tensorflow**

TensorFlow's computational housing is a tensor, similar to Numpy's housing of data in Ndarray's making both of them N-d array libraries.

However, Numpy does not offer a method to create tensor functions and automatically compute derivatives, nor does it support GPU implementation. Thus, for processing data of higher dimensions,Tensorflow outperforms Numpy arrays due largely to its GPU implementations.

**Numpy vs Tensorflow Implementations**

***Numpy Implementation of Matrix Addition***

```python
	import numpy as np
	a=np.zeros((2,2))
	b=np.zeros((2,2))
	np.sum(b,axis=0)
	a.shape
	np.reshape(b,(1,3))
```



***Tensorflow Implementation of Matrix Addition***

```python
	import tensorflow as tf
	tf.InteractiveSession()
	a=tf.zeros((2,2))
	b=tf.ones((2,2))
	tf.reduce_sum(b,reduction_indices=1).eval()
	a.get_shape()
	tf.reshape(b,(1,3)).eval()
```

It is important to note that tensorflow requires explicit evaluation, i.e, tensorflow computation defines a computational graph which only gets initialized with values after a session has been evaluated.

Numpy for example

```python
	a=np.zeros((2,2)) ; print(a)
```

will immediately give the value of "a".

However, for tensorflow:

```python
	a=tf.zeros((2,2))
	print(a)
```

will not return the value of "a" until it is evaluated with

```python
       print(ta.eval())
```

So It is important to understand how tensorflow works and initializes the environment.

Tensorflow uses a "session object" which encapsulates the environment in which the tensors are evaluated.

A  Tensorflow (tf) session for performing multiplication is demonstrated below:

```python
	a= tf.constant(9999999)
	b=tf.constant(111111111)
	c=a*b
	with tf.Session() as sess:
	     print(sess.run(c))
	     print(c.eval())
```

Tensorflow firstly structures the program, creates a graph integrating the variables, and uses session to exectute the process.

*** Tensorflow Variables ***
Similar to other programming language variables, tensorflow uses a variable object to store and update the parameters. They are stored in memory buffers that contain tensors. TensorFlow variables must be initialized before they have values! This is in contrast with constant tensors:

```python
	W=tf.Variable(tf.zeros((2,2)), name="weights")
	R=tf.Variable(tf.random_normal((2,2)), name="Random_weights")

	with tf.Session() as sess:
	        sess.run(tf.initialize_all_variables())
		print(sess.run(W))
		print(sess.run(R))
```

Converting numpy data to tensor:

```python
	a=np.zeros((3,3))
	t_a=tf.convert_to_tensor(a)
	with tf.Session() as sess:
		print(sess.run(t_a))
```

For scalable variables for performing operations we can use `tf.placeholder` which defines a placeholder and provides entry points for the data to be viewed in a computational graph.  `feed_dict` is used in the below example to map from `tf.placeholder` variables to data (np arrays, list, etc).


```python
	input1= tf.placeholder(tf.float32)
	input2 = tf.placeholder(tf.float32)
	output = tf.multiply(input1, input2)
	with tf.Session() as sess:
		print(sess.run([output], feed_dict={input1:[7.], input2:[2.]}))
```

<p class="carc-provenance" markdown>Migrated from [UNM-CARC QuickBytes](https://github.com/UNM-CARC/QuickBytes/blob/master/Tensorflow_documentation.md){target=_blank} (last source update 2019-03-06). Spotted a problem? [Open an issue or pull request](https://github.com/UNM-CARC/QuickBytes){target=_blank}.</p>

---8<--- https://unm-carc.github.io/docs/tutorials/beast/

---
title: "Bayesian phylogenetics with BEAST"
description: "Run BEAST Bayesian evolutionary analyses on CARC clusters."
type: Tutorial
tags:
  - Bioinformatics
  - Phylogenetics
generated:
  by: "claude/fable-5"
  at: "2026-08-29T00:00:00Z"
sources:
  - id: quickbytes
    resource: "https://github.com/UNM-CARC/QuickBytes/blob/master/Beast_at_CARC.md"
    title: "UNM-CARC QuickBytes: Beast_at_CARC.md"
    author: "team:unm-carc"
    last_modified: "2019-10-15T10:51:40-06:00"
---

# Bayesian phylogenetics with BEAST

Bayesian Evolutionary Analysis by Sampling Trees [(BEAST)](https://beast.community/index.html){target=_blank} is a software package that performs
phylogenetic tree analysis with user specified molecular clock models using the widely popular Bayesian Markov chain Monte Carlo 
(MCMC) methods. BEAST has its origins in modeling pathogen evolution in near real time but is also popular for other phylogenetic 
applications. BEAST is a well documented and flexible tool for modeling phylogenetics. Using BEAST at CARC offers more power for 
rigorous computations.

## Generating BEAST imput files: BEAUti

BEAST uses .xml files which contain sequences and model parameters. Because BEAST is capable of incorporating a diverse range of 
meta data and specific time modeling parameters, the graphical user interface [BEAUTi](https://beast.community/first_tutorial){target=_blank} 
allows users to upload nexus files and create .xml files with ease. Make sure that the version of beast in the module you load 
matches the version of BEAUTi used to generate the .xml files. 

## Running BEAST on CARC systems

Once a .xml file is generated, beast can be easily run on CARC. An example .pbs script is as follows: 

```
#!/bin/bash

#PBS -q default
#PBS -N BEASTjob
#PBS -l nodes=1:ppn=8
#PBS -l walltime=24:00:00
#PBS -j oe

cd $PBS_O_WORKDIR

module load beast2-2.5.2-intel-19.0.4-hcnoysj

beast my_data.xml
```

The output should be a job log (joined with any potential error file), and a .tree file for your downstream analysis. For more assistance 
with BEAST at CARC please email help@carc.unm.edu.

<p class="carc-provenance" markdown>Migrated from [UNM-CARC QuickBytes](https://github.com/UNM-CARC/QuickBytes/blob/master/Beast_at_CARC.md){target=_blank} (last source update 2019-10-15). Spotted a problem? [Open an issue or pull request](https://github.com/UNM-CARC/QuickBytes){target=_blank}.</p>

---8<--- https://unm-carc.github.io/docs/tutorials/gatk/

---
title: "Variant calling with GATK"
description: "A genomics variant-calling workflow using GATK best practices on CARC systems."
type: Tutorial
tags:
  - Bioinformatics
  - Genomics
generated:
  by: "claude/fable-5"
  at: "2026-08-29T00:00:00Z"
sources:
  - id: quickbytes
    resource: "https://github.com/UNM-CARC/QuickBytes/blob/master/GATK_QuickByte.md"
    title: "UNM-CARC QuickBytes: GATK_QuickByte.md"
    author: "team:unm-carc"
    last_modified: "2023-01-18T15:28:37-07:00"
---

# Variant calling with GATK

This QuickByte outlines how to run a pipeline based on Genome Analysis Toolkit v4 (GATK4) best practices, a common pipeline for processing genomic data from Illumina platforms. Major modifications from “true” best practices are done to facilitate using this pipeline for both model and non-model organisms. Additionally, we show how to best parallelize these steps on CARC. Here we outline the steps for a single sample without parallelization, then with parallelization for specific steps, and finally provide an example of a fully parallelized script. Extensive documentation (including other Best Practices pipelines) can be found [here](https://gatk.broadinstitute.org/hc/en-us/sections/360007226651-Best-Practices-Workflows){target=_blank}. Specifically, the Best Practices informing this pipeline are the [data pre-processing workflow](https://gatk.broadinstitute.org/hc/en-us/articles/360035535912-Data-pre-processing-for-variant-discovery){target=_blank} and the [germline short variant discovery workflow](https://gatk.broadinstitute.org/hc/en-us/articles/360035535932-Germline-short-variant-discovery-SNPs-Indels-){target=_blank}. We aim to give you sample commands to emulate these scripts workflows, which will also allow you to easily modify the pipeline.

The goal of this pipeline is to output Single Nucleotide Polymorphisms (SNPs) and optionally indels for a given dataset. This same pipeline can be used for humans, model organisms, and non-model organisms. Spots that can leverage information from model organisms are noted, but those steps can be bypassed. Because sample size and depth of coverage are often lower in non-model organisms, filtering recommendations and memory requirements will vary. Note that this assumes you are using paired-end data and will differ slightly if you use unpaired.

The basic steps are aligning and processing raw reads into binary alignment map (BAM) files, optionally getting descriptive metrics about the samples’ sequencing and alignment, calling variants to produce genomic variant call format (GVCF) files, genotyping those GVCFs to produce VCFs, and filtering those variants for analysis.

For CARC users, we have provided some test data to run this on from a paper on [the conservation genomics of sagegrouse](https://academic.oup.com/gbe/article/11/7/2023/5499175){target=_blank}. It is two sets of gzipped fastq files per species (i.e. eight total, 4 read and 4 read 2), a file with adapter sequences to trim, and a reference genome. They are located at /projects/shared/tutorials/GATK/. Copy them into your space like "cp /projects/shared/tutorials/quickbytes/GATK/* ~/path/to/directory". A .pbs script for running the pipeline (seen below) is also included, but you may learn more by running each step individually. The whole process with the script with 4 nodes takes about 5.5 hours. If you run this script, note that it should output a filtered VCF file that's ~350 Mb.

Please note that you must cite any program you use in a paper. At the end of this, we have provided citations you would include for the programs we ran here.

## Table of Contents ##

- [Preliminaries](#prelim)

- [The Pipeline](#pipeline)

  - [Trimming reads](#trimming)
	
  - [Alignment and pre-processing](#align)
	
  - [Summary statistics](#sumstat)
	
  - [Calling Variants](#haplo)
	
  - [Consolidating and genotyping](#geno)
	
  - [Selecting variants](#select)
	
- [Scatter-gather parallel](#parallel)

- [Sample Scatter-gather PBS script](#script)

- [Trobuleshooting](#tshoot)

- [Citations](#cite)	

<a name="prelim"/>

## Preliminary stuff ##

### Module and directories ###

We will be using conda to make an environment to load within our PBS script. First, if you haven’t already, set up conda as follows:

```bash
module load miniconda3-4.7.12.1-gcc-4.8.5-lmtvtik
# can also use other conda modules
conda init bash
<exit and re-log or restart shell>
```

This following line will create an environment and install the most recent versions of what we need. We assume you run this before starting up your job.

```bash
conda create -n gatk-env -c bioconda -c conda-forge gatk4 bwa samtools picard trimmomatic
```

Alternatively, you can load these as modules, but they may not be the most recent versions:

```bash
module load bwa-0.7.17-intel-18.0.2-7jvpfu2
module load samtools-1.9-gcc-7.3.0-tnzvvzt
module load picard-2.20.8-gcc-4.8.5-3yh2dzv
module load gatk-4.1.4.1-gcc-4.8.5-python3-fqiavji
module load trimmomatic-0.36-gcc-4.8.5-q3gx4rj
```

If you are parallelizing (see “Scatter-gather Parallel” and sample PBS script), you'll need this:

```bash
module load parallel-20170322-gcc-4.8.5-2ycpx7e
source $(which env_parallel.bash)
```

The directories we will need (other than the home directory) are a raw_reads directory for the demultiplexed reads and the following for various intermediate files to go into. Alternatively, if you don’t want to move around all your reads, just replace the path in the BWA call with that path. Note that a few of these are only used with scatter-gather parallelization (reccomended for larger datasets).

```bash
mkdir clean_reads
mkdir alignments
# next three are only if you get optional metrics
mkdir alignments/alignment_summary
mkdir alignments/insert_metrics
mkdir alignments/depth
mkdir alignments/dedup_temp
mkdir bams
mkdir gvcfs
mkdir combined_vcfs
mkdir analysis_vcfs
# scatter-gather only
mkdir combined_vcfs/intervals
mkdir gvcfs/combined_intervals
```

We will be using a few variables throughout this that we can set now. These are shortcuts for the path to our working directory and reference.

```bash
src=$PBS_O_WORKDIR
reference=$src/reference
```

### Sample Names ###

To keep our script short, and outputs easy to understand, we will use consistent sample names for each step, and keep the sample names in a file. We assume this file is named “sample_list”. The file should have one sample name per line. with a single blank line at the end. The one for the tutorial dataset looks like:

```bash
GRSG_JHWY140
GRSG_JHWY142
GUSG_GGS1
GUSG_GGS2
```
	
We will use this sample list in two ways. The first way is loops, and second is GNU parallel. You can see some examples in the PBS script at the end of the document. Here's a basic demonstration of how to us the list in a loop:

```bash
while read sample; do
	RunTask -input ${sample}
done < $src/sample_list
```

And this is what GNU parallel looks like (note it's different for BWA, as we need to specify a specific number of jobs). Remember, we need to use env_parallel if we are using conda.
	
```bash
cat $src/sample_list | env_parallel --sshloginfile $PBS_NODEFILE \
	'RunTask -input {}.file'
```

For clarity, in most cases the commands are written as they would be for a for loop (i.e. with $sample instead of {}).

### Demultiplexing ###

Because it is not covered by best practices, and is often done by the sequencing center, we will not go into the details of demultiplexing here. We recommend you use Illumina’s software [bcl2fastq](https://support.illumina.com/sequencing/sequencing_software/bcl2fastq-conversion-software.html){target=_blank} if you have the data in .bcl format, and [saber](https://github.com/najoshi/sabre){target=_blank} if it has already been converted to fastq format and it does not have dual combinatorial barcodes. **We'll assume these reads will be in the raw_reads folder with the name SAMPLE_1.fastq.gz (or 2 for read 2).**

<a name="pipeline"/>

## The Pipeline ##

<a name="trimming"/>

### Trimming Reads ###

Although not a part of GATK's best practices, it is common practice to trim your reads before using them in analyses. We'll use trimmomatic for this. Trimmomatic performs very poorly with its internal thread command, so we'll use GNU parallel to run it in the final script. Note that trimmomatic doesn't have many command line flags, so we'll name variables ahead of time to keep them straight:

```bash
# note this assumes the provided fasta file is in your working directory.
adapters=$src/TruSeq3-PE.fa

read1=$src/raw_reads/${sample}_1.fastq.gz
read2=$src/raw_reads/${sample}_2.fastq.gz
paired_r1=$src/clean_reads/${sample}_paired_R1.fastq.gz
paired_r2=$src/clean_reads/${sample}_paired_R2.fastq.gz
unpaired_r1=$src/clean_reads/${sample}_unpaired_R1.fastq.gz
unpaired_r2=$src/clean_reads/${sample}_unpaired_R2.fastq.gz
# the minimum read length accepted, we do the liberal 30bp here
min_length=30
trimmomatic PE -threads 1 \
	$read1 $read2 $paired_r1 $unpaired_r1 $paired_r2 $unpaired_r2 \
	ILLUMINACLIP:${adapters}:2:30:10:2:True \
	LEADING:3 TRAILING:3 MINLEN:${min_length}
```
	
If you don't have access to the CARC directory with the adapters file, it can be found in the conda install/spack package. The exact path will vary, but they'll be something like this:
	
```bash
# spack
adapters=/opt/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/trimmomatic-0.36-q3gx4rjeruluf75uhcdfkjoaujqnjhzf/bin/TruSeq3-SE.fa
# conda
adapters=~/.conda/pkgs/trimmomatic-0.39-1/share/trimmomatic-0.39-1/adapters/TruSeq3-PE.fa
```

If you're using the spack module, you call trimmomatic using java:

```bash
java -jar /opt/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/trimmomatic-0.36-q3gx4rjeruluf75uhcdfkjoaujqnjhzf/bin/trimmomatic-0.36.jar PE ...
```

<a name="align"/>

### Alignment and Pre-processing ###

This section prepares BAM files for variant calling. First, we need to index our reference and make a sequence dictionary. We'll index two ways, one for bwa and one for GATK:

```bash
bwa index -p $reference ${reference}.fa
samtools faidx ${reference}.fa -o ${reference}.fa.fai
picard CreateSequenceDictionary \
```
       		R=${reference}.fa \
       		O=${reference}.dict

Then, we need to align demultiplexed reads to a reference. For this step, we will use the Burrough-Wheeler Aligner’s (BWA) mem algorithm. Another common option is [Bowtie](http://bowtie-bio.sourceforge.net/bowtie2/index.shtml){target=_blank}. One important flag here is the -R flag, which is the read group and sample ID for a given sample. We assume that these samples are in the same read group. We can get a node's worth of parallelization with the -t command (it can't work across nodes). Therefore, in the sample script at the end we will show you how to further parallelize BWA. The base command looks like this:

```bash
bwa mem \
	-t [# threads] -M \
	-R "@RG\tID:${sample}\tPL:ILLUMINA\tLB:${sample}\tSM:${sample}" \
	$reference \
	$src/clean_reads/${sample}_paired_R1.fastq.gz \
	$src/clean_reads/${sample}_paired_R2.fastq.gz \
	> $src/alignments/${sample}.sam
```

The next step is to mark PCR duplicates to remove bias, sort the file, and convert it to the smaller BAM format for downstream use. GATK’s new MarkDuplicatesSpark performs all these tasks, but needs a temporary directory to store intermediate files. Note that although we aren't formally using Spark for parallelization, the line " --conf 'spark.executor.cores=8'" still speed it up, and makes (change the number of cores if the cores per node are higher than 8):

```bash
gatk MarkDuplicatesSpark \
	-I $src/alignments/${sample}.sam \
	-M $src/bams/${sample}_dedup_metrics.txt \
	--tmp-dir $src/alignments/dedup_temp \
	-O $src/bams/${sample}_dedup.bam \
	 --conf "spark.executor.cores=8"
```

We recommend combining these steps per sample for efficiency and smoother troubleshooting. One issue is that we do not want large SAM files piling up. This can either be done by piping BWA output directly to MarkDuplicatesSpark or removing the SAM file after each loop. In case you want to save the SAM files, we did the latter (this isn’t a bad idea if you have the space, in case there is a problem with generating BAM files). If you are doing base recalibration, you can also add “rm ${sample}\_debup.bam” to get rid of needless BAM files. Later in the pipeline, we assume you did base recalibration, so will use the {sample}\_recal.bam file. If you did not use base recalibration, use {sample}\_dedup.bam file in its place.

#### Base Quality Score Recalibration (model organisms)

An optional step (and one not taken in the tutorial) is to recalibrate base call scores. This applies machine learning to find where quality scores are over or underestimated based on things like read group and cycle number of a given base. This is recommended, but is rarely possible for non-model organisms, as a file of known polymorphisms is needed. Note, however, that it can take a strongly filtered VCF from the end of the pipeline, before running the entire pipeline again (but [others haven’t found much success with this method](https://evodify.com/gatk-in-non-model-organism/){target=_blank}). Here is how it looks, with the first line indexing the input VCF file if you haven't already.

```bash
gatk IndexFeatureFile -I $src[name-of-known-sites].vcf

gatk BaseRecalibrator \
	-I $src/bams/${sample}_dedup.bam \
	-R ${reference}.fa \
	--known-sites $src/[name-of-known-sites].vcf \
	-O $src/bams/${sample}_recal_data.table

gatk ApplyBQSR \
	-I $src/bams/${sample}_dedup.bam \
	-R ${reference}.fa \
	--bqsr-recal-file $src/bams/${sample}_recal_data.table \
	-O $src/bams/${sample}_recal.bam
```

<a name="sumstat"/>

### Collect alignment and summary statistics (optional)

This step is optional, and is not part of GATK’s best practices, but is good to have. It will output important stats for assessing sample quality. Picard’s “CollectAlignmentSummaryMetrics” gives several helpful statistics about the alignment for a given sample. Picard’s “CollectInsertSizeMetrics” gives information about the distribution of insert sizes. Samtools’s “depth” gives information about the read depth of the sample. Note that "depth" has huge output files, so it may be best to skip it until needed.

```bash
picard CollectAlignmentSummaryMetrics \
	R=${reference}.fa \
	I=$src/bams/${sample}_recal.bam \
	O=$src/alignments/alignment_summary/${sample}_alignment_summary.txt

picard CollectInsertSizeMetrics \
	INPUT=$src/bams/${sample}_recal.bam \
	OUTPUT=$src/alignments/insert_metrics/${sample}_insert_size.txt \
	HISTOGRAM_FILE=$src/${sample}_insert_hist.pdf

samtools depth \
	-a $src/bams/${sample}_recal.bam \
	> $src/alignments/depth/${sample}_depth.txt
```

<a name="haplo"/>

### Calling variants with HaplotypeCaller

The simplest way is individually going through BAM files and calling SNPs on them using HaplotypeCaller in GVCF mode (“-ERC GVCF” flag), resulting in GVCFs as output.

```bash
gatk HaplotypeCaller \
	-R ${reference}.fa \
	-I $src/bams/${sample}_recal.bam \
	-O $src/gvcfs/${sample}_raw.g.vcf.gz \
	-ERC GVCF
```

One issue with HaplotypeCaller is that it takes a long time, but is not programmed to be parallelized by default. We can use GNU parallel to solve that problem in two ways. If you have many small inputs and don't want to do scatter-gather parallel, you can run one instance of HaplotypeCaller per core. Note that we restrict the memory such that each job can only max out the core it's on (you'll want to change from 6g based on the machine you're running this on):

```bash
cat $src/sample_list | env_parallel --sshloginfline $PBS_NODEFILE \
	'gatk --java-options "-Xmx6g" HaplotypeCaller \
	-R ${reference}.fa \
	-I $src/bams/{}_recal.bam \
	-O $src/gvcfs/${}_raw.g.vcf.gz \
	-ERC GVCF'
```

If you are dealing with large files, HaplotypeCaller may take longer than your walltime. The Scatter-gather Parallel section will outline how to fix that by breaking the job (and the next section) into multiple intervals.

<a name="geno"/>

### Consolidating and Genotyping ###

This next step has two options, GenomicsDBImport and CombineGVCFs. GATK recommends GenomicsDBImport, as it is more efficient for large datasets, but it performs poorly on references with many contigs. CombineGVCFs can take a long time for large datasets, but is easier to use. Note that GenomicsDBImport must have intervals (generally corresponding to contigs or chromosomes) specified. GenomicsDBImport can take a file specifying GVCFs, but because CombineGVCFs cannot take this input, we just make a list of samples to combine programmatically and plug it in. Here is how we generate that command:

```bash
gvcf_names=""
while read sample; do
	gvcf_names="${gvcf_names}-V ${src}/gvcfs/${sample}_raw.g.vcf.gz "
done < $src/sample_list
```

If you do use GenomicsDBImport, or want to genotype contigs/chromosomes independently, we'll need intervals for it to work with (the same used for scatter-gather parallelization). Also, you'll need to pre-make a temp directory for holding files:

```bash
mkdir gendb_temp
cut -f 1 ${reference}.fa.fai > $src/intervals.list	
```

For GenomicsDBImport, you'll need to get rid of the directory you use for the database (here genomic_database) if you already made it:

```bash
gatk GenomicsDBImport \
	${gvcf_names} \
	--genomicsdb-workspace-path $src/genomic_database \
	--tmp-dir $src/gendb_temp \
	-L $src/intervals.list
```

And an example for CombineGVCFs is:

```bash
gatk CombineGVCFs \
	-R ${reference}.fa \
	${gvcf_names} \
	-O $src/combined_vcfs/combined_gvcf.g.vcf.gz
```

The next step is to genotype the combined (cohort) GVCF file. Here’s a sample command for GenomicsDBImport:

```bash
gatk GenotypeGVCFs \
	-R ${reference}.fa \
	-V gendb://$src/genomic_database \
	-O $src/combined_vcfs/combined_vcf.vcf.gz
```
	
And one for CombineGVCFs:

```bash
gatk GenotypeGVCFs \
	-R ${reference}.fa \
	-V $src/combined_vcfs/combined_gvcf.g.vcf.gz \
	-O $src/combined_vcfs/combined_vcf.vcf.gz
```

<a name="select"/>

### Selecting and filtering variants

This first step is optional, but here we separate out indels and SNPs. Note that we don’t use indels down the line, but similar filters can be applied.

```bash
gatk SelectVariants \
	-R ${reference}.fa \
	-V $src/combined_vcfs/combined_vcf.vcf.gz \
	-select-yype SNP \
	-O $src/combined_vcfs/raw_snps.vcf.gz

gatk SelectVariants \
	-R ${reference}.fa \
	-V $src/combined_vcfs/combined_vcf.vcf.gz \
	-select-type INDEL \
	-O $src/combined_vcfs/raw_indel.vcf.gz
```

Here are some good sample filters. The “DP_filter” is depth of coverage (you will probably want to change this), “Q_filter” is quality score, “QD_filter” is quality by depth (avoids artificial inflation of calls), "MQ_filter" is a mapping quality filter, and “FS_filter” is a strand bias filter (higher value means higher bias). Note that DP is better for low depth samples, while QD is better for high depth. More info can be found on [GATK’s website](https://gatk.broadinstitute.org/hc/en-us/articles/360035890471-Hard-filtering-germline-short-variants){target=_blank}.

```bash
gatk VariantFiltration \
	-R ${reference}.fa \
	-V $src/combined_vcfs/raw_snps.vcf.gz \
	-O $src/analysis_vcfs/filtered_snps.vcf  \
	-filter "DP < 4" --filter-name "DP_filter" \
	-filter "QUAL < 30.0" --filter-name "Q_filter" \
	-filter "QD < 2.0" --filter-name "QD_filter" \
	-filter "FS > 60.0" --filter-name "FS_filter" \
	-filter "MQ < 40.0" --filter-name "MQ_filter"
```

This will give us our final VCF! Note that the filtered SNPs are still included, just with a filter tag. You can use something like SelectVariants' "exclude-filtered" flag or [VCFtools’](http://vcftools.sourceforge.net/){target=_blank} “--remove-filtered-all” flag to get rid of them.

<a name="parallel"/>

## Scatter-gather Parallel

Scatter-gather is the process of breaking a job into intervals (i.e. contigs or scaffolds in a reference) and running HaplotypeCaller, CombineGVCFs, and GenotypeGVCFs on each interval in parallel. Then, at the end, all the invervals are gathered together with GatherGVCFs. This results in a massive speed-up due to the parallelization. This is fully implemented in the sample script below, with each step outlined here. The output of GatherVcfs is the same as what comes from GenotypeGVCFs in the non-parallel version. Here is how we run HaplotypeCaller, note that this is only one sample, see the sample script for running this on all samples:
	
```bash
# make our interval list
cut -f 1 ${reference}.fa.fai > $src/intervals.list

while read sample; do
	mkdir ${src}/gvcfs/${sample}
	cat $src/intervals.list | env_parallel --sshloginfile $PBS_NODEFILE \
		'gatk --java-options "-Xmx6g" HaplotypeCaller \
		-R ${reference}.fa \
		-I $src/bams/${sample}_recal.bam \
		-O $src/gvcfs/${sample}/${sample}_{}_raw.g.vcf.gz \
		-L {} \
		-ERC GVCF'
done < $src/sample_list
```
	
You'll run then run CombineGVCFs. For each interval, you'll make a list of GVCF file paths for each sample you're including (the while loop below).

```bash
cat $src/intervals.list | env_parallel --sshloginfile $PBS_NODEFILE \
	'interval_list=""
	# loop to generate list of sample-specific intervals to combine
	while read sample; do
		interval_list="${interval_list}-V ${src}/gvcfs/${sample}/${sample}_{}_raw.g.vcf.gz "
	done < $src/sample_list
	gatk --java-options "-Xmx6g" CombineGVCFs \
		-R ${reference}.fa \
		${interval_list} \
		-O $src/gvcfs/combined_intervals/{}_raw.g.vcf.gz'
```
			
Next, you run GenotypeGVCFs to get VCFs to gather afterwards. No fancy lists needed!

```bash
cat $src/intervals.list | env_parallel --sshloginfile $PBS_NODEFILE \
	'gatk --java-options "-Xmx6g" GenotypeGVCFs \
		-R ${reference}.fa \
		-V $src/gvcfs/combined_intervals/{}_raw.g.vcf.gz \
		-O $src/combined_vcfs/intervals/{}_genotyped.vcf.gz'
```
			
If you have many samples, it may be best to use GenomicsDBImport. It is very similar, with both that step and the genotyping below. Note that the directory for --genomicsdb-workspace-path can't exist (unless you're updating it):

```bash
cat $src/intervals.list | env_parallel --sshloginfile $PBS_NODEFILE \
	'mkdir $src/gendb_temp/{}
 	interval_list=""
```
    		# loop to generate list of sample-specific intervals
```bash
	while read sample; do
		interval_list="${interval_list}-V ${src}/gvcfs/${sample}/${sample}_{}_raw.g.vcf.gz "
	done < $src/sample_list
	# run make the genomics databases
	gatk --java-options "-Xmx6g" GenomicsDBImport \
		 ${interval_list} \
		--genomicsdb-workspace-path $src/genomics_databases/{} \
		--tmp-dir $src/gendb_temp/{} \
		-L {}'

cat $src/intervals.list | env_parallel --sshloginfile $PBS_NODEFILE \
	'gatk --java-options "-Xmx6g" GenotypeGVCFs \
	-R ${reference}.fa \
	-V gendb://$src/genomics_databases/{} \
	-O $src/combined_vcfs/intervals/{}_genotyped.vcf.gz'
```

The final (gather) step uses GatherVcfs, for which we'll make a file containing the paths to all input genotyped VCFs (generated in the while loop). Note the first line initializes a blank file for the gather list. After we make the gathered VCF, we need to index it for future analyses.

```bash
> $src/combined_vcfs/gather_list
while read interval; do
	echo ${src}/combined_vcfs/intervals/${interval}_genotyped.vcf.gz >> \
```
       			$src/combined_vcfs/gather_list
```bash
done < $src/chromosomes.list

gatk GatherVcfs \
```
    		-I $src/combined_vcfs/gather_list \
    		-O combined_vcfs/combined_vcf.vcf.gz
		
```bash
gatk IndexFeatureFile \
	-I $src/combined_vcfs/raw_snps.vcf.gz
```

**NOTE THAT THIS FILE STILL NEEDS TO HAVE VARIANTS SELECTED AND FILTERED, SEE "Selecting and filtering variants" ABOVE**

<a name="script"/>

## Sample PBS Script ##

Here is a sample PBS script combining everything we have above, with as much parallelization as possible. One reason to break up steps like we did is for improved checkpointing (without having to write code checking if files are already present). Once you are finished running a block of code, you can just comment it out. Similarly, if you can only get part way through your sample list, you can copy it and remove samples that have already completed a given step.

To convert this to Slurm, replace $PBS_O_WORKDIR with $SLURM_SUBMIT_DIR and refer to [this conversion guide](../running-jobs/pbs-to-slurm.md) for the rest.
	
```bash
#!/bin/bash

#PBS -q default
#PBS -l nodes=4:ppn=8
#PBS -l walltime=10:00:00
#PBS -N gatk_tutorial
#PBS -m ae
#PBS -M youremail@school.edu

# the PBS lines are for the default queue, using 4 nodes, and has a conservative 10 hour wall time
# it is named "gatk_tutorial" and sends an email to "youremail@school.edu" when done

# load your conda environment
module load miniconda3-4.7.12.1-gcc-4.8.5-lmtvtik
eval "$(conda shell.bash hook)"
conda activate gatk-env

# load GNU parallel, get env_parallel
module load parallel-20170322-gcc-4.8.5-2ycpx7e
source $(which env_parallel.bash)

src=$PBS_O_WORKDIR
# this is "sagegrouse_reference" in the tutorial
reference=${src}/reference

# indexing reference
bwa index -p $reference ${reference}.fa
samtools faidx ${reference}.fa -o ${reference}.fa.fai
picard CreateSequenceDictionary \
```
       		R=${reference}.fa \
       		O=${reference}.dict
	
```bash
# Trimming section
adapters=~/.conda/pkgs/trimmomatic-0.39-1/share/trimmomatic-0.39-1/adapters/TruSeq3-PE.fa
cat $src/sample_list | env_parallel --sshloginfile $PBS_NODEFILE \
	'read1=$src/raw_reads/{}_1.fastq.gz
	read2=$src/raw_reads/{}_2.fastq.gz
	paired_r1=$src/clean_reads/{}_paired_R1.fastq.gz
	paired_r2=$src/clean_reads/{}_paired_R2.fastq.gz
	unpaired_r1=$src/clean_reads/{}_unpaired_R1.fastq.gz
	unpaired_r2=$src/clean_reads/{}_unpaired_R2.fastq.gz
	# the minimum read length accepted, we do the liberal 30bp here
	min_length=30
	trimmomatic PE -threads 1 \
		$read1 $read2 $paired_r1 $unpaired_r1 $paired_r2 $unpaired_r2 \
		ILLUMINACLIP:${adapters}:2:30:10:2:True \
		LEADING:3 TRAILING:3 MINLEN:${min_length}'

# Section for alignment and marking duplicates. 
# Note we parallelize such that BWA uses exactly one node.
# Then, we have a number of jobs equal to the number of nodes requested.

cat $src/sample_list | env_parallel -j 1 --sshloginfile $PBS_NODEFILE \
	'bwa mem \
		-t 8 -M \
		-R "@RG\tID:{}\tPL:ILLUMINA\tLB:{}\tSM:{}" \
		$reference \
		$src/clean_reads/{}_paired_R1.fastq.gz \
		$src/clean_reads/{}_paired_R2.fastq.gz \
		> $src/alignments/{}.sam
	gatk MarkDuplicatesSpark \
		-I $src/alignments/{}.sam \
		-M $src/bams/{}_dedup_metrics.txt \
		--tmp-dir $src/alignments/dedup_temp \
		-O $src/bams/{}_dedup.bam \
		 --conf "spark.executor.cores=8"
	rm $src/alignments/{}.sam'

# Collecting metrics in parallel
# Remember to change from _recal to _dedup if you can’t do base recalibration.
# Also, depth will take A LOT of room up, so you may not want to run it until you know what to do with it.

cat $src/sample_list | env_parallel --sshloginfile $PBS_NODEFILE \
	'picard CollectAlignmentSummaryMetrics \
		R=${reference}.fa \
		I=$src/bams/{}_dedup.bam \
		O=$src/alignments/alignment_summary/{}_alignment_summary.txt
	picard CollectInsertSizeMetrics \
		INPUT=$src/bams/{}_dedup.bam \
		OUTPUT=$src/alignments/insert_metrics/{}_insert_size.txt \
		HISTOGRAM_FILE=$src/alignments/insert_metrics/{}_insert_hist.pdf
	samtools depth \
		-a $src/bams/{}_dedup.bam \
		> $src/alignments/depth/{}_depth.txt'

# Scatter-gather HaploType Caller, probably the most likely to need checkpoints.
# This can take a lot of different forms, this one is best for large files.
# Go to the HaplotypeCaller section for more info. 

cut -f 1 ${reference}.fa.fai > $src/intervals.list

while read sample; do
	mkdir ${src}/gvcfs/${sample}
	cat $src/intervals.list | env_parallel --sshloginfile $PBS_NODEFILE \
		'gatk --java-options "-Xmx6g" HaplotypeCaller \
		-R ${reference}.fa \
		-I $src/bams/${sample}_dedup.bam \
		-O $src/gvcfs/${sample}/${sample}_{}_raw.g.vcf.gz \
		-L {} \
		-ERC GVCF'
done < $src/sample_list

# Run CombineGVCFs per interval, each step combines all samples into one interval-specific GVCF
cat $src/intervals.list | env_parallel --sshloginfile $PBS_NODEFILE \
	'interval_list=""
	# loop to generate list of sample-specific intervals to combine
	while read sample; do
		interval_list="${interval_list}-V ${src}/gvcfs/${sample}/${sample}_{}_raw.g.vcf.gz "
	done < $src/sample_list
	gatk --java-options "-Xmx6g" CombineGVCFs \
		-R ${reference}.fa \
		${interval_list} \
		-O $src/gvcfs/combined_intervals/{}_raw.g.vcf.gz'

# Run GenotypeGVCFs on each interval GVCF
cat $src/intervals.list | env_parallel --sshloginfile $PBS_NODEFILE \
	'gatk --java-options "-Xmx6g" GenotypeGVCFs \
		-R ${reference}.fa \
		-V $src/gvcfs/combined_intervals/{}_raw.g.vcf.gz \
		-O $src/combined_vcfs/intervals/{}_genotyped.vcf.gz'

# Make a file with a list of paths for GatherVcfs to use
> $src/combined_vcfs/gather_list
while read interval; do
	echo ${src}/combined_vcfs/intervals/${interval}_genotyped.vcf.gz >> \
```
       			$src/combined_vcfs/gather_list
```bash
done < $src/intervals.list

# Run GatherVcfs
gatk GatherVcfs \
```
    		-I $src/combined_vcfs/gather_list \
    		-O $src/combined_vcfs/combined_vcf.vcf.gz

```bash
# Index the gathered VCF
gatk IndexFeatureFile \
	-I $src/combined_vcfs/combined_vcf.vcf.gz

# Select and filter variants
gatk SelectVariants \
	-R ${reference}.fa \
	-V $src/combined_vcfs/combined_vcf.vcf.gz \
	-select-type SNP \
	-O $src/combined_vcfs/raw_snps.vcf.gz

gatk SelectVariants \
	-R ${reference}.fa \
	-V $src/combined_vcfs/combined_vcf.vcf.gz \
	-select-type INDEL \
	-O $src/combined_vcfs/raw_indel.vcf.gz

gatk VariantFiltration \
	-R ${reference}.fa \
	-V $src/combined_vcfs/raw_snps.vcf.gz \
	-O $src/analysis_vcfs/filtered_snps.vcf  \
	-filter "DP < 4" --filter-name "DP_filter" \
	-filter "QUAL < 30.0" --filter-name "Q_filter" \
	-filter "QD < 2.0" --filter-name "QD_filter" \
	-filter "FS > 60.0" --filter-name "FS_filter" \
	-filter "MQ < 40.0" --filter-name "MQ_filter"
```

<a name="tshoot"/>

## Troubleshooting ##

If you need help troubleshooting an error, make sure to let us know the size of your dataset (number of individuals and approximate number of reads should suffice, unless coverage varies between individuals), GATK version, node details, and any error messages output.

<a name="cite"/>

## Citations ##

Bolger, A. M., Lohse, M., & Usadel, B. (2014). Trimmomatic: a flexible trimmer for Illumina sequence data. Bioinformatics, 30(15), 2114–2120. https://doi.org/10.1093/bioinformatics/btu170

Li, H., & Durbin, R. (2009). Fast and accurate short read alignment with Burrows-Wheeler transform. Bioinformatics, 25(14), 1754–1760. https://doi.org/10.1093/bioinformatics/btp324

Li, H., Handsaker, B., Wysoker, A., Fennell, T., Ruan, J., Homer, N., … Durbin, R. (2009). The Sequence Alignment/Map format and SAMtools. Bioinformatics, 25(16), 2078–2079. https://doi.org/10.1093/bioinformatics/btp352

McKenna, A., Hanna, M., Banks, E., Sivachenko, A., Cibulskis, K., Kernytsky, A., … DePristo, M. A. (2010). The genome analysis toolkit: A MapReduce framework for analyzing next-generation DNA sequencing data. Genome Research, 20(9), 1297–1303. https://doi.org/10.1101/gr.107524.110

Picard toolkit. (2019). Broad Institute, GitHub Repository. https://doi.org/http://broadinstitute.github.io/picard/

Tange, O. (2018). GNU Parallel 2018 [Computer software]. https://doi.org/10.5281/zenodo.1146014.

<p class="carc-provenance" markdown>Migrated from [UNM-CARC QuickBytes](https://github.com/UNM-CARC/QuickBytes/blob/master/GATK_QuickByte.md){target=_blank} (last source update 2023-01-18). Spotted a problem? [Open an issue or pull request](https://github.com/UNM-CARC/QuickBytes){target=_blank}.</p>

---8<--- https://unm-carc.github.io/docs/tutorials/metabarcoding/

---
title: "Metabarcoding analysis"
description: "Process environmental DNA metabarcoding data on CARC clusters."
type: Tutorial
tags:
  - Bioinformatics
  - Ecology
generated:
  by: "claude/fable-5"
  at: "2026-08-29T00:00:00Z"
sources:
  - id: quickbytes
    resource: "https://github.com/UNM-CARC/QuickBytes/blob/master/Metabarcoding.md"
    title: "UNM-CARC QuickBytes: Metabarcoding.md"
    author: "team:unm-carc"
    last_modified: "2022-09-29T17:30:23-06:00"
---

# Metabarcoding analysis

Metabarcoding is the process of using next-generation sequencing platforms (Illumina, PacBio, or Oxford Nanopore) to sequence amplicons and determine the ecological community that is present. The most common applications are microbiome analyses to study the community of bacteria or fungi. Given that metabarcoding relies upon PCR, biases do occur (e.g. primer biases, variation in loci copy number, incomplete lineage sorting, etc.). However, other techniques such as metagenomic sequencing cannot fully assemble larger genomes such as fungal genomes and fail to capture all of the species present in high diversity samples. Thus, metabarcoding remains the best option to characterize microbial communities. 

This tutorial is designed to give you an example of how to take the sequences you get from the sequencing facility and generate:

1. a fasta file of all of the unique sequences known as the representative sequence file (abbreviated to rep-seq) 
2. a table with the abundance of each of the representative sequences for every sample. Often referred to as an OTU/ASV table.  


One point of contention for metabarcoding projects is how to define which sequences are unique. The traditional view was to  cluster any sequences that diverged by less than 3% of sequence similarity into a species known as an OTU (operational taxon unit). However, other people reject the clustering step as being arbitrary and define any divergence as noteworthy. This approach is known as ASVs, (amplicon sequence variants). While the debate between clustering to OTUs or using ASVs remains contentious, for most community analyses, each will give you the same biological answer. The right choice will depend on your question and your system. For example, fungal metabarcoding studies use a highly variable region called the internal transcribed spacer (ITS) which is known to diverge by 3% within an individual (they have many copies of the ITS region) and among members of the same species. Thus, clustering to OTUs is more logical for fungal taxa to avoid oversplitting species. However, if you were using a conserved gene such as the 18S, ASV's might give you a better approximation of species.

 Final note: make sure you are runing the commands in a pbs script or on an interactive node!
 
### Different pipelines ###
There are many different pipelines to process metabarcoding samples. For this tutorial we will focus on the three main ones:
1. QIIME2 (using DADA2)
2. Mothur
3. USEARCH




### Steps for each pipeline ###
For eachof them, they will follow these key steps:
1. install
      * how to set up the environments. 
3. join forward and reverse reads
      * Illumina sequencing produces forward and reverse reads for every sequence.
5. filter reads (remove chimeras)
      * The merged reads will need to be trimmed of extraneous sequence, poor quality sequences will need be removed, and chimera, the generation of DNA sequences           from disparate organisms due to errors in the PCR process, will also need to be removed. 
7. create OTUs/ASVs
      * The unique sequences will be determined based on the chosen algorithm. 
9. Creation of OTU/ASV table
      * The abundances of the OTUs or ASVs will be tabulated per sample to create the table. 
11. determine Taxonomy for OTUs/ASVs
      * The taxonomy of each of the OTUs/ASVs will be inferred by comparing the sequences against commonly used databases.



## Data for tutorial ##
For each of the pipelines we will use 16S bacterial of the V4 region provided by the Mothur pipeline. 

### create folder structure ###
```
mkdir metabarcoding
cd metabarcoding
src=~/metabarcoding

mkdir data
cd data

# dataset is called miseqsopdata
wget https://mothur.s3.us-east-2.amazonaws.com/wiki/miseqsopdata.zip
unzip miseqsopdata.zip

cd miseqsopdata
# gzip all fastq files to save space.
gzip *fa

# move back to main folder
cd $src
```







## QIIME2 (using DADA2) pipeline ##
For QIIME2, every file created is either uses a .qsv or .qsa extension. the .qsv is a zip file that contains data and the metadata. The .qsv are visualizations that can be viewed by uploading the file to https://view.qiime2.org/. QIIME2 prefers to create ASVs using the DADA2 method, so this tutorial will do that.


### install ###
We will create a conda environment called qiime2-2021.4. 
```
   # Load the conda module:
   module load miniconda3

   #download the yml
   wget https://data.qiime2.org/distro/core/qiime2-2021.4-py38-linux-conda.yml
   
   # create conda environment called qiime2-2020.8
   conda env create -n qiime2-2021.4 --file qiime2-2021.4-py38-linux-conda.yml
   
   # delete yml
   rm qiime2-2020.8-py36-linux-conda.yml
   
```


### join forward and reverse reads ###


```
conda activate qiime2-2021.4
cd $scr

mkdir qiime2_tutorial
cd qiime2_tutorial

# merge reads
qiime tools import \
--type 'SampleData[PairedEndSequencesWithQuality]' \
--input-path $src/data/MiSeq_SOP/fastqs/ \
--input-format CasavaOneEightSingleLanePerSampleDirFmt \
--output-path $src/qiime2_tutorial/demux-paired-end.qza

# summary figures online
# open visual at this link: https://view.qiime2.org/
qiime demux summarize \
  --i-data  $src/qiime2_tutorial/demux-paired-end.qza \
  --o-visualization $src/qiime2_tutorial/demux.qzv
```


### filter reads (remove chimeras) and create ASVs ###
This step does all of the filtering and creation of ASVs at once. 
```
# following what we see in the visualization we will trim the reads and denoise the reads. 
# this will also create the rep seq qiime file wiht the ASVs. 
# This method denoises paired-end sequences, dereplicates them, and filters chimeras.
qiime dada2 denoise-paired \
  --i-demultiplexed-seqs  $src/qiime2_tutorial/demux-paired-end.qza \
  --p-trunc-len-f 240 \
  --p-trunc-len-r 160 \
  --o-representative-sequences $src/qiime2_tutorial/rep-seqs-dada2.qza \
  --o-table  $src/qiime2_tutorial/table-dada2.qza \
  --o-denoising-stats  $src/qiime2_tutorial/stats-dada2.qza \
  --p-n-threads 0 # use all available cores

# create rep-seq file.
qiime feature-table tabulate-seqs \
  --i-data $src/qiime2_tutorial/rep-seqs-dada2.qza \
  --o-visualization $src/qiime2_tutorial/rep-seqs.qzv

# create summary table 
qiime feature-table summarize \
  --i-table $src/qiime2_tutorial/table-dada2.qza \
  --o-visualization $src/qiime2_tutorial/table.qzv 

```

### Creation of ASV table ###
```
  # export rep seq sequences.
  # it is exported in the rep-seqs wiht 
 qiime tools export \
  --input-path $src/qiime2_tutorial/rep-seqs-dada2.qza \
  --output-path $src/qiime2_tutorial/rep-seqs
     
# creates OTU table
qiime tools export \
  --input-path $src/qiime2_tutorial/table-dada2.qza \
   --output-path $src/qiime2_tutorial/exported-feature-table
```






## Mothur pipeline ##
Mothur uses a unique syntax in which each command begins is structured liek this: mothur "#command here(parameters_here=X)". Mothur can do ASVs but the preference of the its creator is to use OTUS so we will do that here. 

### install ###

```
 # load miniconda
 module load miniconda3-4.7.12.1-gcc-4.8.5-lmtvtik
   
      
conda env create -n mothur
conda install -n mothur -c bioconda mothur

 
 
```

### join forward and reverse reads ###

```
conda activate mothur
cd $src
mkdir mothur_tutorial 

cd mothur_tutorial
# copy files into mothur folder
scp $src/data/MiSeq_SOP/fastqs/*gz .

# create a list of the files
# output
#     1. stability.files
mothur "#make.file(inputdir=., type=gz, prefix=stability)"


# join forward and reverse reads. Gives you a count of reads of how many reads are assembled for each sample
# output
#     1. stability.trim.contigs.fasta
#     2. stability.scrap.contigs.fasta
#     3. stability.contigs.report
#     4. stability.contigs.groups
mothur "#make.contigs(file=stability.files, processors=8)"

# summary stats on the merged reads
mothur "#summary.seqs(fasta=stability.trim.contigs.fasta)"

```



### filter reads, remove primer region, and remove chimeras ###
```
# removes reads that are longer than 275 bases. Likely to be errors
# output files
#        1. stability.contigs.pick.groups
#        2. stability.trim.contigs.good.fasta
#        3. stability.trim.contigs.bad.accnos
#        4. stability.contigs.good.groups
mothur "#screen.seqs(fasta=stability.trim.contigs.fasta, group=stability.contigs.groups, maxambig=0, maxlength=275)"



# find unique sequences in dataset. This is to reduce the size of the dataset and reduce redundancies. 
# output files
#        1.stability.trim.contigs.good.names
         2. stability.trim.contigs.good.unique.fasta
mothur "#unique.seqs(fasta=stability.trim.contigs.good.fasta)"


# count up how many reads are match the good names. 
# output file
#        1. stability.trim.contigs.good.count_table
mothur "#count.seqs(name=stability.trim.contigs.good.names, group=stability.contigs.good.groups)"


# now going to trim unique reads to silva dataset. 

# download reference database to trim reads
wget https://mothur.s3.us-east-2.amazonaws.com/wiki/silva.bacteria.zip
unzip silva.bacteria.zip


# trim to the V4 variable region of the silva bacteria dataset. This is the reigon we sequenced
# output file
#        1. silva.bacteria/silva.bacteria.pcr.fasta
mothur "#pcr.seqs(fasta=silva.bacteria/silva.bacteria.fasta, start=11894, end=25319, keepdots=F, processors=8)"

# align unique reads to Silva reference
# output files. 
#        1. stability.trim.contigs.good.unique.align
#        2. stability.trim.contigs.good.unique.align.report
mothur "#align.seqs(fasta=stability.trim.contigs.good.unique.fasta, reference=silva.bacteria/silva.bacteria.pcr.fasta)"


# summary of where the sequences align
# output file
#        1. stability.trim.contigs.good.unique.summary
mothur "#summary.seqs(fasta=stability.trim.contigs.good.unique.align, count=stability.trim.contigs.good.count_table)"

# trim sequences that extend beyond the Silva alignment (overhangs) and remove gap only columns. 
# output file
#        1. stability.filter
#        2. stability.trim.contigs.good.unique.filter.fasta
mothur "#filter.seqs(fasta=stability.trim.contigs.good.unique.align, vertical=T, trump=.)"


# find all unique sequences again in case there are some sequences that are now identical.
# creates:
#     1. stability.trim.contigs.good.unique.filter.count_table
#     2. stability.trim.contigs.good.unique.filter.unique.fasta
mothur "#unique.seqs(fasta=stability.trim.contigs.good.unique.filter.fasta, count=stability.trim.contigs.good.count_table)"



# pre-clustering to clean up sequencing errors. Differences of two nucleotides will be clustered. 
# creates an output file for each sample so it can create a lot of files.
# output files
#        1. stability.trim.contigs.good.unique.filter.unique.precluster.fasta
#        2. stability.trim.contigs.good.unique.filter.unique.precluster.count_table
#        3. stability.trim.contigs.good.unique.filter.unique.precluster.F3D0.map
#        4. stability.trim.contigs.good.unique.filter.unique.precluster.F3D1.map
#        5. stability.trim.contigs.good.unique.filter.unique.precluster.F3D141.map
#        6. stability.trim.contigs.good.unique.filter.unique.precluster.F3D142.map
#        7. stability.trim.contigs.good.unique.filter.unique.precluster.F3D143.map
#        8. stability.trim.contigs.good.unique.filter.unique.precluster.F3D144.map
#        9. stability.trim.contigs.good.unique.filter.unique.precluster.F3D145.map
#        10. stability.trim.contigs.good.unique.filter.unique.precluster.F3D146.map
#        11. stability.trim.contigs.good.unique.filter.unique.precluster.F3D147.map
#        12. stability.trim.contigs.good.unique.filter.unique.precluster.F3D148.map
#        13. stability.trim.contigs.good.unique.filter.unique.precluster.F3D149.map
#        14. stability.trim.contigs.good.unique.filter.unique.precluster.F3D150.map
#        15. stability.trim.contigs.good.unique.filter.unique.precluster.F3D2.map
#        16. stability.trim.contigs.good.unique.filter.unique.precluster.F3D3.map
#        17. stability.trim.contigs.good.unique.filter.unique.precluster.F3D5.map
#        18. stability.trim.contigs.good.unique.filter.unique.precluster.F3D6.map
#        19. stability.trim.contigs.good.unique.filter.unique.precluster.F3D7.map
#        20. stability.trim.contigs.good.unique.filter.unique.precluster.F3D8.map
#        21. stability.trim.contigs.good.unique.filter.unique.precluster.F3D9.map
#        22. stability.trim.contigs.good.unique.filter.unique.precluster.Mock.map
mothur "#pre.cluster(fasta=stability.trim.contigs.good.unique.filter.unique.fasta, count=stability.trim.contigs.good.unique.filter.count_table, diffs=2)"



# remove chimeras
Output files:
#     1. stability.trim.contigs.good.unique.filter.unique.precluster.denovo.vsearch.pick.count_table
#     2. stability.trim.contigs.good.unique.filter.unique.precluster.denovo.vsearch.chimeras
#     3. stability.trim.contigs.good.unique.filter.unique.precluster.denovo.vsearch.accnos
mothur "#chimera.vsearch(fasta=stability.trim.contigs.good.unique.filter.unique.precluster.fasta, count=stability.trim.contigs.good.unique.filter.unique.precluster.count_table, dereplicate=t)"

# remove chimeras from fasta
# output file
# 1. stability.trim.contigs.good.unique.filter.unique.precluster.pick.fasta
mothur "#remove.seqs(fasta=stability.trim.contigs.good.unique.filter.unique.precluster.fasta, accnos=stability.trim.contigs.good.unique.filter.unique.precluster.denovo.vsearch.accnos)"


# downloading files to classifiy sequences and remove non-bacterial reads
wget https://mothur.s3.us-east-2.amazonaws.com/wiki/trainset18_062020.pds.tgz
tar zxvf  trainset18_062020.pds.tgz
 

# remove sequences that are not bacteria. 
# output
#     1. stability.trim.contigs.good.unique.filter.unique.precluster.pick.pds.wang.taxonomy
#     2. stability.trim.contigs.good.unique.filter.unique.precluster.pick.pds.wang.tax.summary
mothur "#classify.seqs(fasta=stability.trim.contigs.good.unique.filter.unique.precluster.pick.fasta, count=stability.trim.contigs.good.unique.filter.unique.precluster.denovo.vsearch.pick.count_table, reference=trainset18_062020.pds/trainset18_062020.pds.fasta, taxonomy=trainset18_062020.pds/trainset18_062020.pds.tax, cutoff=80)"


# removing lineages here
# output
#     1. stability.trim.contigs.good.unique.filter.unique.precluster.pick.pds.wang.pick.taxonomy
#     2. stability.trim.contigs.good.unique.filter.unique.precluster.pick.pds.wang.accnos
#     3. stability.trim.contigs.good.unique.filter.unique.precluster.denovo.vsearch.pick.pick.count_table
#     4. stability.trim.contigs.good.unique.filter.unique.precluster.pick.pick.fasta
mothur "#remove.lineage(fasta=stability.trim.contigs.good.unique.filter.unique.precluster.pick.fasta, count=stability.trim.contigs.good.unique.filter.unique.precluster.denovo.vsearch.pick.count_table, taxonomy=stability.trim.contigs.good.unique.filter.unique.precluster.pick.pds.wang.taxonomy, taxon=Chloroplast-Mitochondria-unknown-Archaea-Eukaryota)"

# get summary of taxonomy by counts
mothur "#summary.tax(taxonomy=stability.trim.contigs.good.unique.filter.unique.precluster.pick.pds.wang.pick.taxonomy, count=stability.trim.contigs.good.unique.filter.unique.precluster.denovo.vsearch.pick.pick.count_table)"


```


### create OTUs ###
```

# create file of distances between potential OTUs
# output
#     1. stability.trim.contigs.good.unique.filter.unique.precluster.pick.pick.dist
mothur "#dist.seqs(fasta=stability.trim.contigs.good.unique.filter.unique.precluster.pick.pick.fasta, cutoff=0.03)"

# create OTUs at 97% similarity.
# output 
#     1. stability.trim.contigs.good.unique.filter.unique.precluster.pick.pick.opti_mcc.list
#     2. stability.trim.contigs.good.unique.filter.unique.precluster.pick.pick.opti_mcc.steps
#     3. stability.trim.contigs.good.unique.filter.unique.precluster.pick.pick.opti_mcc.sensspec
mothur "#cluster(column=stability.trim.contigs.good.unique.filter.unique.precluster.pick.pick.dist, count=stability.trim.contigs.good.unique.filter.unique.precluster.denovo.vsearch.pick.pick.count_table)"

# Creates OTU table with only OTUs
# output
#     1.  stability.trim.contigs.good.unique.filter.unique.precluster.pick.pick.opti_mcc.shared
mothur "#make.shared(list=stability.trim.contigs.good.unique.filter.unique.precluster.pick.pick.opti_mcc.list, count=stability.trim.contigs.good.unique.filter.unique.precluster.denovo.vsearch.pick.pick.count_table, label=0.03)"

# create taxonomy for each OTU
#     1. stability.trim.contigs.good.unique.filter.unique.precluster.pick.pick.opti_mcc.0.03.cons.taxonomy
#     2. stability.trim.contigs.good.unique.filter.unique.precluster.pick.pick.opti_mcc.0.03.cons.tax.summary
mothur "#classify.otu(list=stability.trim.contigs.good.unique.filter.unique.precluster.pick.pick.opti_mcc.list, count=stability.trim.contigs.good.unique.filter.unique.precluster.denovo.vsearch.pick.pick.count_table, taxonomy=stability.trim.contigs.good.unique.filter.unique.precluster.pick.pds.wang.pick.taxonomy, label=0.03)"

# get representative sequences files
# output 
#     1. stability.trim.contigs.good.unique.filter.unique.precluster.pick.pick.opti_mcc.0.03.rep.names
#     2. stability.trim.contigs.good.unique.filter.unique.precluster.pick.pick.opti_mcc.0.03.rep.fasta
mothur "#get.oturep(column=stability.trim.contigs.good.unique.filter.unique.precluster.pick.pick.dist, list=stability.trim.contigs.good.unique.filter.unique.precluster.pick.pick.opti_mcc.list, name=stability.trim.contigs.good.names,  fasta=stability.trim.contigs.good.unique.filter.unique.precluster.pick.fasta)"
```





## USEARCH ##
USEARCH is a proprietary software developed by Robert Edgar. For this tutorial, we will use the free, 32-bit version. However, if you have multiple sequencing runs, you will likely need to upgrade to the paid, 64-bit version. USEARCH can do OTUs or ASVs. For this example I stuck with OTUs. Many of the commands can easily be changed to VSEARCH, an alternative that is open-source.

### install ###
```
cd $src

mkdir usearch_tutorial
cd usearch_tutorial


# download the program
wget https://www.drive5.com/downloads/usearch11.0.667_i86linux32.gz

gunzip usearch11.0.667_i86linux32

# change permissions so you can run it
chmod +x usearch11.0.667_i86linux32

# rename so its shorter
 mv usearch11.0.667_i86linux32 usearch11.0.667
 
 ```
 
 
 ### join forward and reverse reads ###
 ```
# need to unzip all the files first
cd $src/data/MiSeq_SOP/fastqs
gunzip *gz

cd $src/usearch_tutorial

# merges forward and reverse reads. 
./usearch11.0.667 -fastq_mergepairs $src/data/MiSeq_SOP/fastqs/*R1*.fastq -fastqout merged.fq -relabel @


 
 # we can use this step to guide how we will filter the reads. 
 # this will tell us the size of the reads and the error rate. 
 # given the error rate is well below 0.5 and most reaads are above 250 bp, i will use that and 250 as the shortest size for length. 
  ./usearch11.0.667 -fastx_info merged.fq -secs 5 -output reads_info.txt
  
  

 # filter reads based off the suggestions above. This creates a fasta file which can be used for clustering
 ./usearch11.0.667 -fastq_filter merged.fq -fastaout reads.fasta -fastq_maxee 0.5 -fastq_minlen 250
```

### Create OTUs #### 

```
# dereplicate sequences so we only have unique reads. 
# creates file with uniques.fasta which each read relabeled as uniq
./usearch11.0.667 -fastx_uniques reads.fasta -fastaout uniques.fasta -sizeout -relabel Uniq

# creates otus at 97% and removes chimeras via UPARSE
# warning: this will throw out singletons so change -minsize to 1. 
# creates uparse.txt - log of how the clustering went for each unique read.
# creates otus.fasta - representative OTUs. 
./usearch11.0.667 -cluster_otus uniques.fasta -otus otus.fasta -uparseout uparse.txt -relabel Otu -minsize 2


```


### create OTU table ###
 ```
# map back reads and create OTU table
# the otutab.txt is the OTU table 
# the map.txt shows you how each read is mapped to what OTU
./usearch11.0.667 -otutab merged.fq -otus otus.fasta -otutabout otutab.txt -mapout map.txt

```


### determine Taxonomy for OTUs ###
```
# downloaad rdp database
wget https://drive5.com/sintax/rdp_16s_v16.fa.gz
gunzip rdp_16s_v16.fa.gz

# Creates taxonomy file with OTU and the taxonomy in the reads.sintax file
./usearch11.0.667 -sintax otus.fasta -db rdp_16s_v16.fa -tabbedout reads.sintax -strand both -sintax_cutoff 0.8
```

<p class="carc-provenance" markdown>Migrated from [UNM-CARC QuickBytes](https://github.com/UNM-CARC/QuickBytes/blob/master/Metabarcoding.md){target=_blank} (last source update 2022-09-29). Spotted a problem? [Open an issue or pull request](https://github.com/UNM-CARC/QuickBytes){target=_blank}.</p>

---8<--- https://unm-carc.github.io/docs/tutorials/mpi-casa/

---
title: "Parallel CASA for radio astronomy"
description: "Run mpiCASA for parallel radio astronomy imaging on CARC clusters."
type: Tutorial
tags:
  - Astronomy
  - MPI
generated:
  by: "claude/fable-5"
  at: "2026-08-29T00:00:00Z"
sources:
  - id: quickbytes
    resource: "https://github.com/UNM-CARC/QuickBytes/blob/master/mpiCASA.md"
    title: "UNM-CARC QuickBytes: mpiCASA.md"
    author: "team:unm-carc"
    last_modified: "2021-03-08T15:51:32-07:00"
---

# Parallel CASA for radio astronomy

### A Bit About CASA

[CASA](https://casa.nrao.edu/){target=_blank} is the premier software for reducing radio data coming off of a variety of telescopes around the world, including the Jansky Very Large Array (VLA) and Atacama Large Millimeter Array (ALMA). 

### Getting Some Data to Play With

Going to use the new [Archive](data.nrao.edu). Find something small and have them download it manually, like a few GB at most.

### Getting Set Up

First off get some nodes

```bash
srun --partition singleGPU --nodes 2 --tasks-per-node 2 --pty bash
```

Do we want singleGPU? Can request more nodes obviously. tasks-per-node we don't need to set unless CASA demands slots

Might want to create aliases for casa and mpi casa, just to make things quick

```bash
alias casa='/path/to/casa/bin/casa'
alias mpicasa='/path/to/casa/bin/mpicasa'
```

Actually it doesn't like the alias when you run the command

2) It will create a nodefile for us at $PBS_NODEFILE
   If CASA doesn't need slots, we can use this, and all the nodes
   If CASA does need slots, we'll need to make a script that can reference $PBS_NUM_PPN and add slots

3) Run something like 'path_to_casa_bin/mpicasa -hostfile $PBS_NODEFILE path_to_casa_bin/casa <casa options>'
   Probably want to do --nogui and --log2term
   Puts us into a CASA environment

4) Run python script that does all the fun stuff
   Could run mpicasa call with '-c myscript.py'
   Can also do 'exec(open('./filename').read())'
       Shortcut as execfile 'filename.py'

<p class="carc-provenance" markdown>Migrated from [UNM-CARC QuickBytes](https://github.com/UNM-CARC/QuickBytes/blob/master/mpiCASA.md){target=_blank} (last source update 2021-03-08). Spotted a problem? [Open an issue or pull request](https://github.com/UNM-CARC/QuickBytes){target=_blank}.</p>

---8<--- https://unm-carc.github.io/docs/tutorials/msprime/

---
title: "Coalescent simulation with msprime"
description: "Simulate genealogical histories and genome sequences with msprime."
type: Tutorial
tags:
  - Bioinformatics
  - Population genetics
generated:
  by: "claude/fable-5"
  at: "2026-08-29T00:00:00Z"
sources:
  - id: quickbytes
    resource: "https://github.com/UNM-CARC/QuickBytes/blob/master/msprime_quickbyte.md"
    title: "UNM-CARC QuickBytes: msprime_quickbyte.md"
    author: "team:unm-carc"
    last_modified: "2021-04-28T13:01:51-06:00"
---

# Coalescent simulation with msprime

Many microevolutionary questions invoke population genetic processes to explain them, from population genetic summary statistics to selection on specific loci of interest. In turn, population genetic simulations are becoming increasingly important to publish in major journals to validate that empirical result are consistent with the processes authors say the findings represent. There are many programs used for these simulations, that fall into two general categories: forward (slow, simulate all individuals in the populations) and backward (fast, work from a number of samples backward to simulate genealogies). Prominent examples of these two are [SLiM](https://messerlab.org/slim/){target=_blank} and [msprime](https://msprime.readthedocs.io/en/stable/){target=_blank} respectively.

Here, we'll give a basic introduction to msprime, give a simple example of two diverging populations, and show how to parallelize replicates on CARC. The example is one I used to establish how F<sub>ST</sub>, [an estimate of population divergence](https://onlinelibrary.wiley.com/doi/10.1111/j.1558-5646.1984.tb05657.x){target=_blank}, changes over time based on population size. In particular, I used this to test if observed values of F<sub>ST</sub> between islands connected during the Pleistocene became panmictic at those times, or if there was still reduced gene flow between given islands (with the system being one worked on in the past in papers like [Smith & Filardi 2007](https://academic.oup.com/auk/article/124/2/479/5562749){target=_blank}).

This QuickByte describes msprime 1.0, which is a major update from the widely used earlier versions.

## How msprime Works ##

[The paper describing msprime](https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1004842){target=_blank} has been cited hundreds of time, but represents an extension of [the simulation program ms](https://academic.oup.com/bioinformatics/article/18/2/337/225783){target=_blank}, which has been cited a couple thousand times. The goal of this extension is to scale simulations up to the large sample sizes of individuals and loci used in modern day genomic studies. Msprime's fast speed and ease of analysis is achieved by by adding novel ways of keeping track of geneologies being analyzed through sparse trees and coalescence records, collectively refer to ass tree sequences.

## How to run msprime ##

Msprime is a python package, with reccomended download via conda. We reccomend you use miniconda for this, as we've had some bugs with installing this through anaconda in the past. We'll make an environment with it and a couple other important modules:

```bash
conda create -n msp1-env -c conda-forge msprime scikit-allel numpy
```

Then we'll activate the environment with "conda activate", but note that it needs to be "source activate" in a script:

```bash
conda activate msp1-env
```

In a script or in python command line, we'll import msprime as msp for convenience:

```python
import msprime as msp
```

The simulations themselves, at a base level, are very simple. All that's needed is a sample size of individuals "tracked" backwards. Note that these are assumed diploid unless ploidy is specified:

```bash
trees = msp.sim_ancestry(samples=10)
```
	
Mutations can be added using sim_mutations, ideally changing the default mutation rate (here to the per-year songbird mutation rate):

```bash
mutations = msp.sim_mutations(trees, rate=2.3e-9)
```

This is just one locus, but we gain a lot more information by adding sequence length and recombination rates.

```bash
trees = msp.sim_ancestry(samples=10, recombination_rate=1e-8, sequence_length=1e6)
```

Unlike the older versions of msprime, outside of pre-made demographic scenarios, you need to add things like populations and merge events to a demography object. Migration rates are also set there. First, we'll initialize the demography and add two populations:

```bash
demo = msp.Demography()
demo.add_population(name="pop1", initial_size=10000)
demo.add_population(name="pop2", initial_size=20000)
```
	
Next, we'll set up an ancestral population and add a population split time (the size is equal to the size before a bottleneck we implement below):

```bash
demo.add_population(name="anc_pop12", initial_size=50000)
demo.add_population_split(time=5700, derived=["pop1","pop2"], ancestral="anc_pop12")
```
	
Finally, we'll add symmetric migration (i.e. gene flow) between the two focal populations:

```bash
demo.set_symmetric_migration_rate(['pop1', 'pop2'], 0.001)
```

We can have the migration rate change over time too, here we increase the migration rate during the period between the split time (5700 generations ago) and 4000 generations go:

```bash
demo.add_symmetric_migration_rate_change(4000, ['pop1', 'pop2'], 0.01)
```
	
The final common demographic event is population size change, which we'll set to have a bottleneck three quarters of the way through the simulation. That is, we set the population size of population 1 to be three times the present population 2500 years:
	
```bash
demo.add_population_parameters_change(time=2500, initial_size=30000, population="pop1")
```

Finally, we need to sort these events, as we added them out of order for instructional purposes:

```bash
demo.sort_events()
```

A really helpful debugging feature is the print function for demography objects, it lists and explains the events and migration matrix!
	
```python
print(demo)
```

Bringing this all together, our simulation will look like:

```bash
trees = msp.sim_ancestry(samples={"pop1":10, "pop2":10},
                 demography=demo,
                 recombination_rate=1e-8,
                 sequence_length=1e7)
```
			 
And we'll add mutations like this, using a yearly rate of 2.3e-9 and a generation time of 2.55 years (same as the case study):

```bash
mutations = msp.sim_mutations(trees, rate=5.9e-9)
```

## Running MSPrime replicates on CARC ##

It is best practice to run many replicates of any simulation you run to assess the robustness of any estimates you make. You can use [GNU Parallel (specifically env_parallel)](../running-jobs/gnu-parallel.md) to run these simulations, and can add these replicates directly to an output file. The following examples your python simulation script takes population size (popsize) and population divergence time (divtime) and have an output file like "$popsize_$divtime.out" that the script writes to. First we'll run 30 replicates. Note that the "echo {}" just to deal with the parallel iterator, so GNU parallel doesn't append it to the end of our python call by default.

```bash
env_parallel --sshloginfile $PBS_NODEFILE \
	'echo {}; /path/to/python msprime_script.py --popsize 2000 --divtime 10000 \
	--output ./outputs/1000_10000.out' ::: {1..30}
```

You could also use GNU parallel to iterate over multiple parameter combinations, here we test multiple population sizes (2000, 3000, and 5000) and divergence times (1000, 5000, and 10000 generations). We'll assume the script has replicates coded into it.

```bash
env_parallel --sshloginfile $PBS_NODEFILE \
	'/path/to/python msprime_script.py --popsize {1} --divtime {2} \
	--output ./outputs/{1}_{2}' ::: 2000 3000 5000 ::: 1000 5000 10000
```

## Case study: Divergence between connected islands ##

Here we look at three islands interconnected at glacial maxima, modeled after Choiseul, Isabel, and Guadalcanal of the Solomon Islands. These islands are arranged from west to east, with Guadalcanal arguably not being fully connected to the rest. Our goal is to assess [F<sub>ST</sub>](https://onlinelibrary.wiley.com/doi/10.1111/j.1558-5646.1984.tb05657.x){target=_blank} between islands, for which which will use the scikit-allel package. Empirically, we find that F<sub>ST</sub> between Isabel and Choiseul is much higher than between Guadalcanal and Isabel, consistent with the slight break between those two islands. However, F<sub>ST</sub> increases faster after divergence with lower population size, so we want to know what density of birds on Guadalcanal would be required to produce this result given knowledge that a density of 25 birds/km<sup>2</sup> produced the empirical F<sub>ST</sub> between Isabel and Choiseul. We will test 5 densities for Guadalcanal (5, 10, 15, 20, 25, and 30), each with 30 replicates, holding the population density on Isabel as a constant 25 birds/km<sup>2</sup>.

First, we have to write our python script. We'll use the argparse module to hand our arguments. Note that we'll set Isabel as the first population and Guadalcanal as the second (in a more complete version we can take island names as input and have a function to determine their sizes). It will output the population size of the first (Isabel) and second (Guadalcanal) island along with the F<sub>ST</sub>.

```python
import os, sys, msprime as msp, numpy as np, allel, re, argparse

def main():
	# set up argparse
	parse = argparse.ArgumentParser(description = "Get simulation parameters")
	# two population densities for generality
	parse.add_argument("-d1", "--density1", type=float, help="Density of first population")
	parse.add_argument("-d2", "--density2", type=float, help="Density of second population")
	parse.add_argument("-o", "--output", type=str, help="Path to output file")
	args = parse.parse_args()

	# assign argparse values to variables
	dens1, dens2, outfile = args.density1, args.density2, args.output

	# calculate population sizes, with Isabel and Guadalcanal being 2999 and 5302 km<sup>2</sup> respectively
	size1 = dens1 * 2999
	size2 = dens2 * 5302

	# set up number of samples and demography
	samples = 30
	demography = msp.Demography()
	demography.add_population(name="pop1", initial_size=size1)
	demography.add_population(name="pop2", initial_size=size2)
	demography.add_population(name="anc_pop12", initial_size=size1+size2)
	demography.add_population_split(time=5700, derived=["pop1","pop2"], ancestral="anc_pop12")

	# run simulation for 10 megabases
	trees = msp.sim_ancestry(samples={"pop1":samples, "pop2":samples},
		demography=demography,
		recombination_rate=1e-8,
		sequence_length=1e7)

	# add mutations with a common estimate of mutation rate in birds
	mutations = msp.sim_mutations(trees, rate=5.9e-9)

	# get haplotypes from simulation
```
    		haplotypes = np.array(mutations.genotype_matrix())
    		genotypes = allel.HaplotypeArray(haplotypes).to_genotypes(ploidy=2)

```bash
	# calculate fst, assumes even sample size
```
    		fst = allel.stats.fst.average_weir_cockerham_fst(genotypes,[list(range(0,samples)),list(range(samples,samples*2))],10)[0]
		
```bash
	# write output to file
	output = open(outfile, "a")
```
    		output.write(str(int(size1))+"\t"+str(int(size2))+"\t"+str(fst)+"\n")
    		output.close()
	
```bash
if __name__ == '__main__':
	main()
```
	
Now that we have our scripted simulation, we'll write a PBS script to run it in parallel! We assumes you have a directory in your working directory called "output" and named your script "island_msp_twopop.py". We'll name files based on the denisty on Guadalcanal. We don't have it written in the script, but if you already have something in "output", this just appends to those files (i.e. "rm output\/*" beforehand). Note that this script excludes the header.
	
```bash
# prepare GNU parallel
module load parallel-20170322-gcc-4.8.5-2ycpx7e
source $(which env_parallel.bash)

# load our environment
module load miniconda3-4.7.12.1-gcc-4.8.5-lmtvtik
source activate msp1-env

# make a shortcut for our working directory, where we assume all scripts are located.	
dir=$PBS_O_WORKDIR

env_parallel --sshloginfile $PBS_NODEFILE \
	'echo {2}; python $dir/island_msp_twopop.py -d1 25 -d2 {1} -o $dir/output/density_{1}.out' \
	::: 3 4 5 10 15 20 25 30 ::: {1..30}
```
		
For interpretting these results, the empirical F<sub>ST</sub> value between Guadalcanal and Isabel is 0.077, and the best density match was 4 birds/km<sup>2</sup> (F<sub>ST</sub>=0.79). That means that to get the observed F<sub>ST</sub>, Guadalcanal would have to have 16% of the population density inferred on the other islands. Genetic diversity doesn't support this, suggesting that the gap between them formed an excess of population structure compared to the other islands!

## Citation ##

Below are citations for msprime and GNU parallel. Remember to cite programs you run whenever possible!

Kelleher, J., Etheridge, A. M., & McVean, G. (2016). Efficient Coalescent Simulation and Genealogical Analysis for Large Sample Sizes. PLoS Computational Biology, 12(5), 1004842. https://doi.org/10.1371/journal.pcbi.1004842

Tange, O. (2018). GNU Parallel 2018 [Computer software]. https://doi.org/10.5281/zenodo.1146014.

<p class="carc-provenance" markdown>Migrated from [UNM-CARC QuickBytes](https://github.com/UNM-CARC/QuickBytes/blob/master/msprime_quickbyte.md){target=_blank} (last source update 2021-04-28). Spotted a problem? [Open an issue or pull request](https://github.com/UNM-CARC/QuickBytes){target=_blank}.</p>

---8<--- https://unm-carc.github.io/docs/tutorials/orca/

---
title: "ORCA quantum chemistry"
description: "Run ORCA quantum chemistry calculations in parallel on CARC clusters."
type: Tutorial
tags:
  - Chemistry
status: draft
generated:
  by: "claude/fable-5"
  at: "2026-08-29T00:00:00Z"
sources:
  - id: quickbytes
    resource: "https://github.com/UNM-CARC/QuickBytes/blob/master/orca_wheeler_taos.md"
    title: "UNM-CARC QuickBytes: orca_wheeler_taos.md"
    author: "team:unm-carc"
    last_modified: "2020-07-15T06:51:46-06:00"
---

# ORCA quantum chemistry

### Submitting an Orca batch script

CARC clusters use Slurm (**S**imple **L**inux **U**tility for **R**esource **M**anagement) to submit jobs and manage resources. Slurm provides greater control over resource management and utilization which means one has to be more explicit in their submission script. Specifically, it is necessary to request sufficient memory for your task when submitting your job. Below is a sample script for submitting an Orca job named `orca_submission.sh`:

```bash
#!/usr/bin/bash

## Set your slurm flags here requesting resources.
#SBATCH --job-name=orca_test
#SBATCH --output=test.out
#SBATCH --ntasks=8
#SBATCH --cpus-per-task=1
## This depends on which queue you have access to.
#SBATCH --partition=my_partition
#SBATCH --mem-per-cpu=6GB
#SBATCH --mail-type=begin        # send email when job begins
#SBATCH --mail-type=end          # send email when job ends
#SBATCH --mail-type=fail         # send email if job fails
#SBATCH --mail-user=<YourNetID>@unm.edu

module load openmpi-3.1.5-gcc-5.4.0-f6ikvl6
module load orca/4.2.1

## Set your input and output file names
input_file=my_orca_input.inp
output_file=my_orca_output.log
prefix=$(echo $input_file | cut -f1 -d".")

# Set the scratch directory path
scratch_dir=/carc/scratch/$USER/

# Set the input and output paths on the scratch file system
mkdir $scratch_dir$prefix
TEMP_DIR=$scratch_dir$prefix
output_scratch_path=$TEMP_DIR/$output_file
input_scratch_path=$TEMP_DIR/$input_file

# Create directory for additional files
mkdir $SLURM_SUBMIT_DIR/$prefix
add_files_dir=$SLURM_SUBMIT_DIR/$prefix

# Copy the input file from the submission directory to the scratch directory
cp $SLURM_SUBMIT_DIR/$input_file $TEMP_DIR/

# Orca needs the full path when running in parallel
full_orca_path=$(which orca)

# Run Orca
$full_orca_path $input_scratch_path > $output_scratch_path

# Orca finished so copy the output file on scratch to the submission directory and clean the scratch directory
cp $output_scratch_path $SLURM_SUBMIT_DIR/$output_file
cp $TEMP_DIR/$prefix* $add_files_dir
rm $TEMP_DIR
```
Now you can simply submit your job to the queue with `sbatch orca_submission.sh`.

<p class="carc-provenance" markdown>Migrated from [UNM-CARC QuickBytes](https://github.com/UNM-CARC/QuickBytes/blob/master/orca_wheeler_taos.md){target=_blank} (last source update 2020-07-15). Spotted a problem? [Open an issue or pull request](https://github.com/UNM-CARC/QuickBytes){target=_blank}.</p>

---8<--- https://unm-carc.github.io/docs/tutorials/psmc/

---
title: "Demographic inference with PSMC"
description: "Infer population size history from diploid genomes using PSMC."
type: Tutorial
tags:
  - Bioinformatics
  - Population genetics
generated:
  by: "claude/fable-5"
  at: "2026-08-29T00:00:00Z"
sources:
  - id: quickbytes
    resource: "https://github.com/UNM-CARC/QuickBytes/blob/master/psmc_quickbyte.md"
    title: "UNM-CARC QuickBytes: psmc_quickbyte.md"
    author: "team:unm-carc"
    last_modified: "2021-08-16T10:52:42-06:00"
---

# Demographic inference with PSMC

The [pairwise sequentially Markovian coalescent model](https://www.nature.com/articles/nature10231){target=_blank} is a popular method of leveraging single high-quality diploid genomes to infer the demographic history of a lineage over thousands to hundreds of thousands of years. It can be a great exploratory tool for genomic data, and can help you understand and generate biogeographic and evolutionary hypotheses. It leverages heterozygosity information to estimate local times of most recent common ancestor across the genome, which is then used to reconstruct demographic "stairway plots".

It is implemented [by the authors of the original paper on GitHub](https://github.com/lh3/psmc){target=_blank}, but the documentation is difficult to understand and the method of calling variants is a bit outdated. Here I'll outline a simple pipeline for generating a consensus sequence using [high coverage (>18x) reads and sites with a depth of at least 10 reads](https://onlinelibrary.wiley.com/doi/10.1111/mec.13540){target=_blank} and a reference genome. Then, I'll go over how to run PSMC and perform bootstrapping. Note that nothing but the bootstrapping can work across different nodes, so if you find bootstrapping takes too long you can run it as a seperate job with more nodes.

The runtime and resource requirements will vary based on genome, but the only step that can work across nodes is bootstrap generation. Standard nodes will work for some samples, but nodes with more cores may be needed for others due to wall time limits.


## Installation and setup

Due to the inavailibility of PSMC on conda, high number of included utilities, and ease of installing locally, we suggest you install PSMC as shown below. You can install it anywhere, but we'll assume it's in the working directory you're using to run everything:

```bash
git clone https://github.com/lh3/psmc.git
cd psmc
make
cd utils
make
cd ../..
```

Then we'll install some dependencies with conda as below.

```bash
conda create -n psmc-env -c bioconda -c conda-forge picard bcftools samtools bwa
```

At the top of any scripts used for this, change to your working directory and activate the environment like:

```bash
cd $SLURM_SUBMIT_DIR
# if using PBS, 'cd $PBS_O_WORKDIR'
module load miniconda3/4.8.2-pilj
eval "$(conda shell.bash hook)"
conda activate psmc-env
```

Then, we'll make a subdirectory for future bootstraps with "mkdir boot". If you are testing multiple references for consistency, I suggest putting everything for each reference in its own subdirectory.

## Generating input

Generating input is essentially a simplified version of [GATK's widely used pipeline](gatk.md) using [bcftools](http://samtools.github.io/bcftools/bcftools.html){target=_blank} to leverage its simplicity and ability to generate a consensus FASTA file with heterozygosity. First, reads are alligned to the reference with BWA. Then, bcftools' mpileup and call are used to obtain variant calls, which are filtered using bcftools' view command. Finally, a consensus sequence is generated and converted to a "PSMC FASTA" for use in PSMC itself. Code for running these steps is below, assuming your reference is "reference.fa" and read files are called "sample_R1.fastq.gz" and "sample_R2.fastq.gz".

```bash
# Align with BWA
bwa index -p reference reference.fa
# assuming you have 8 threads, not that this .sam file will be huge, and we'll remove it at the end
bwa mem -t 8 reference sample_R1.fastq.gz sample_R2.fastq.gz > bwa_alignment.sam
# convert .sam to .bam
samtools view -S -b bwa_alignment.sam > unsorted_alignment.bam
# sort the .bam file
picard SortSam I=unsorted_alignment.bam O=sorted_alignment.bam SORT_ORDER=coordinate
# remove old files, comment out if you want to keep them to troubleshoot
rm bwa_alignment.sam
rm unsorted_alignment.bam

# pipeline combining bcftool's mpileup and call (consensus mode) using 8 threads, then samtools's vcfutils.pl
# the latter filters variants with a depth less than 10 or greater than 50, and those with quality score under 30
bcftools mpileup -Q 30 -q 30 -Ovu -f reference.fa sorted_alignment.bam --threads 8 | \
	bcftools call -c --threads 8 | \
	vcfutils.pl vcf2fq -d 10 -D 50 -Q 30 > variant_consensus.fq

# generate PSMC input
./psmc/utils/fq2psmcfa variant_consensus.fq > psmc_input.psmcfa
```

## Running PSMC

A basic PSMC run is outlined in the GitHub, but optimization is required to avoid overfitting problems. You can start with the default parameters and work from there. The first flag, -N, is the number of iterations to use. The second, -t, is the maximum coalescence time permitted. The third, -r, is the initial theta/rho ratio (effectively the per-base mutation rate divided by the per-base recombination rate). The final, -p, is the most complicated, but effectively delimits the number of time intervals per parameter. Using, say, '-p "25\*1"' would mean 25 parameters each spanning one time interval. The example has 28 parameters spanning 64 intervals (one spanning 4, 25 spanning 2, one spanning 4, then one spanning 6). The more recent intervals are on the left side of the expression. You can start with a set-up like below:

    ./psmc/psmc -N25 -t15 -r5 -p "4+25*2+4+6" -o sample.psmc sample.psmcfa

However, before we run bootstrapping and plot this, we want to make sure out runs are optimized.

#### Optimization

After our first run, we will view the end of the resulting .psmc file, which will look something like the table below (with many lines skipped for brevity). Each line represents a time interval, and the fifth column (i.e. 3651.038295 in the first row) the number of recombinations that occur. You want to make sure this number is at least 20 in all intervals, although the GitHub says 10 is sufficient. An example of a good result is below, keep repeating until you get a sufficient result:
	
```bash
RS	0	0.000000	1.282590	3651.038295	0.004395	0.002875
RS	1	0.005884	0.570172	8631.229543	0.010389	0.010062
RS	2	0.012114	0.786536	6563.160557	0.007900	0.006705
...
RS	75	7.181803	0.765618	56.699641	0.000068	0.000064
RS	76	7.610256	0.765618	34.277873	0.000041	0.000038
RS	77	8.063918	0.765618	43.762351	0.000053	0.000049
```

An example of what worked well for me is below, with a notably different maximum coalescent time (biggest difference for me) and free parameters spanning shorter time intervals toward the present:

```bash
./psmc/psmc -N25 -t10 -r5 -p "8*1+30*2+4+6" -o sample.psmc sample.psmcfa
```

Next, we want to bootstrap our results and plot them.

#### Bootstrapping and plotting

Bootstrapping is a step that can fortunately be run in parallel. There's no set number of bootstraps needed, but we'll do 50 here. We'll run this with GNU parallel. First, you'll need to load a parallel module. Then, you'll run what is essentially the same command as before, but with the -b flag:

```bash
module load parallel/20190222-wsvg

parallel '$SLURM_SUBMIT_DIR/psmc/psmc -N25 -t10 -r5 -b -p "8*1+30*2+4+6" \
	-o $SLURM_SUBMIT_DIR/boot/sample_r{}.psmc $SLURM_SUBMIT_DIR/sample.psmcfa' \
	::: $(seq 50)
cat sample.psmc boot/sample_r*.psmc > sample_combined.psmc
```
	
Unfortunately, perl issues make it difficult to plot the output on CARC, so we suggest you transfer the 'sample_combined.psmc' file to your personal computer. Then, install psmc on your computer like above, and run the command as follows. The -g flag specifies your organism's generation time, and -u the per-generation mutation rate (no scientific notation sadly):

```bash
/path/to/psmc/utils/psmc_plot.pl -p -g 2.2 -u 0.00000000506 bootstrapped_plot  sample_combined.psmc
```

Then you'll get an estimate of your focal population's detailed demographic history!

## Citations

Li, H., & Durbin, R. (2009). Fast and accurate short read alignment with Burrows-Wheeler transform. Bioinformatics, 25(14), 1754–1760. https://doi.org/10.1093/bioinformatics/btp324

Li, H., & Durbin, R. (2011). Inference of human population history from individual whole-genome sequences. Nature, 475(7357), 493–496. https://doi.org/10.1038/nature10231

Li, H., Handsaker, B., Wysoker, A., Fennell, T., Ruan, J., Homer, N., … Durbin, R. (2009). The Sequence Alignment/Map format and SAMtools. Bioinformatics, 25(16), 2078–2079. https://doi.org/10.1093/bioinformatics/btp352

Picard toolkit. (2019). Broad Institute, GitHub Repository. https://doi.org/http://broadinstitute.github.io/picard/

Tange, O. (2018). GNU Parallel 2018. https://doi.org/10.5281/ZENODO.1146014

<p class="carc-provenance" markdown>Migrated from [UNM-CARC QuickBytes](https://github.com/UNM-CARC/QuickBytes/blob/master/psmc_quickbyte.md){target=_blank} (last source update 2021-08-16). Spotted a problem? [Open an issue or pull request](https://github.com/UNM-CARC/QuickBytes){target=_blank}.</p>

---8<--- https://unm-carc.github.io/docs/tutorials/simcov/

---
title: "SimCov epidemiological simulation"
description: "Run the SimCov agent-based model of SARS-CoV-2 infection dynamics in lung tissue."
type: Tutorial
tags:
  - Simulation
  - Epidemiology
generated:
  by: "claude/fable-5"
  at: "2026-08-29T00:00:00Z"
sources:
  - id: quickbytes
    resource: "https://github.com/UNM-CARC/QuickBytes/blob/master/SimCov.md"
    title: "UNM-CARC QuickBytes: SimCov.md"
    author: "team:unm-carc"
    last_modified: "2022-09-21T16:03:25-06:00"
---

# SimCov epidemiological simulation

This tutorial contains instructions for compiling and running the SimCov immunology model on CARC systems.

## Download the SimCov Source Code from GitHub

Change directory to your home: 
```
cd ~
```
Clone the simcov Github repository into your home directory:
```
git clone --recurse-submodules https://github.com/AdaptiveComputationLab/simcov.git
```

## Build SimCov from Source
Load the required modules and set UPCXX variables (NOTE: modules subject to change use 'module spider' to find availability):
```
export UPCXX_THREADMODE=seq
export UPCXX_CODEMODE=opt
module load gcc/11.2.0-otgt
module load cmake/3.22.2-c2dw
module load upcxx/2021.9.0-r4of
```
Run the build script:
```
cd simcov
./build.sh Release
```
## Configure SimCov 
The config files are in ~/simcov and end with ".config". You can edit them with a text editor.

## Submit a SimCov Job 
A sample batch script is provided by the developers. We have submitted the script below to the simcov developers - so hopefully by the time you pull simcov the code below will already be in the repository's sample submission script. If not update the script to contain the following: 
 
This PBS submission script will run simcov on a compute node using covid_default.config:
```
#!/bin/bash

#PBS -q normal
#PBS -l nodes=2:ppn=8
#PBS -l walltime=01:00:00
#PBS -N simcov_test
#PBS -j oe

module load gcc/11.2.0-otgt
module load upcxx/2021.9.0-r4of
module load cmake/3.22.2-c2dw

cd $PBS_O_WORKDIR

upcxx-run -n $PBS_NP -N $PBS_NUM_NODES -- install/bin/simcov --config=covid_default.config --output=results
``` 
To run simcov on a compute node enter
```
qsub wheeler_simcov_run.pbs
```

Outputs will be in a results folder by default

## Video walkthrough

**CS491/591: Computational Immunology — SimCov Compilation and Submission** — from the [CARC video tutorials](../training/videos.md):

<iframe class="carc-video" src="https://www.youtube-nocookie.com/embed/7x3voqNX0QY" title="CS491/591: Computational Immunology \u2014 SimCov Compilation and Submission" loading="lazy" allow="accelerometer; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen></iframe>

<p class="carc-provenance" markdown>Migrated from [UNM-CARC QuickBytes](https://github.com/UNM-CARC/QuickBytes/blob/master/SimCov.md){target=_blank} (last source update 2022-09-21). Spotted a problem? [Open an issue or pull request](https://github.com/UNM-CARC/QuickBytes){target=_blank}.</p>

---8<--- https://unm-carc.github.io/docs/tutorials/stacks/

---
title: "RAD-seq analysis with Stacks"
description: "Analyze restriction-site associated DNA sequencing (RAD-seq) data with Stacks."
type: Tutorial
tags:
  - Bioinformatics
  - Genomics
generated:
  by: "claude/fable-5"
  at: "2026-08-29T00:00:00Z"
sources:
  - id: quickbytes
    resource: "https://github.com/UNM-CARC/QuickBytes/blob/master/Stacks_quickbyte.md"
    title: "UNM-CARC QuickBytes: Stacks_quickbyte.md"
    author: "team:unm-carc"
    last_modified: "2021-03-03T16:14:55-07:00"
---

# RAD-seq analysis with Stacks

Stacks is a common and [well documented](https://catchenlab.life.illinois.edu/stacks/){target=_blank} pipeline for processing RADseq data. RADseq data is a method of reduced representation genomic sequencing, where genomic DNA is cut up with restriction enzymes, which are then targeted by sequencing adapters. This allows a researcher to get thousands of loci randomly scattered across the genome, which can be sequenced at moderate depths for low prices. This is sufficient for many population genomic analyses, such as tests of population structure, phylogenetics, gene flow, and even coarse attempts to locate regions of the genome that are under selection. Stacks can be run with or without a reference genome, but using a reference genome is reccomended for improved accuracy.

Stacks can easily be run on CARC systems with installed modules, and here we outline how with some simple "quality of life" adjustments and tips. We'll be focused on the reference based method, as the non-reference-based is sufficiently run through a driver script provided by the developers of Stacks (denovo_map.pl). We'll quickly mention it at the end. This can often be run on a single node, as the only intense step tends to be alignment, which is quick due to the small size of RADseq data. For example, a dataset of ~90 bird individuals with an average of 1 million reads/sample took four hours on one node. Organisms with larger genomes will take more time and memory.

## Preliminaries ##

### Sample list and Population maps ###

Before you get started, you'll need a list of sample names to run bwa conveniently. This is a file we'll call "sample_list", and an example is below. For this and similar files, the first line describes what goes in each column and where tabs go, and the following lines give an example of what that looks like. For sample list, note that an empty newline should be at the end of the list.:

```bash
<SAMPLE NAME>
M_americana_Florida_MSBBIRD49539
M_americana_NM_MSBBIRD39487
......
```

Then, key to many pieces of population genetics software, Stacks needs a population map (popmap) for calculating metrics like F<sub>st</sub> and genetic dviersity. It will also allow the creation of certain imput files. It is simply a tab delimited file with one line per individual, with the first column representing the sample name and the second its population. Note that reduced versions of the popmaps can be made for running _gstacks_ and _populations_ for only a subset of your dataset.

```bash
<SAMPLE NAME>\t<SAMPLE POPULATION>
M_americana_Florida_MSBBIRD49539	EastBlackScoter
M_americana_NewMexico_MSBBIRD39487	WestBlackScoter
..................................	.....
```

### Demultiplexing with process_radtags ###

Demultiplexing with Stacks is comparatively easy. You just need a file of barcode information (we'll call it BARCODES.file) and your raw, multiplexed reads. The barcode information file can be paired or unapired, and should look like:

```bash
<BARCODE1>\t<BARCODE2>\t<SAMPLE NAME>
ATGCAT	GTACGT	M_americana_Florida_MSBBIRD49539
ACGTAT	CTAGAT	M_americana_NewMexico_MSBBIRD39487
......	......	......
```

Here is how to run it, assuming you are dealing with paired end reads, gzipped fastq files, and used a single enzyme (ndeI) for your restriction digest. Note the the "raw_reads" directory is the one that you'll use for the aligning step at the start of the reference-based assembly:

```bash
process_radtags -p /path/to/MULTIPLEXED_READS/ -b /path/to/BARCODES.file -o /path/to/raw_reads/ \
	-i gzfastq -e ndeI -c -q -r -E phred33 
```

The command is different for single end reads. You must specify each fastq input indiviually with the -f flag, as shown below. The rest is the same:

```bash
process_radtags -f /path/to/MULTIPLEXED_READS/RAW_READS_01.fastq.gz -b /path/to/BARCODES.file -o /path/to/raw_reads/ \
	-i gzfastq -e ndeI -c -q -r -E phred33
process_radtags -f /path/to/MULTIPLEXED_READS/RAW_READS_02.fastq.gz -b /path/to/BARCODES.file -o /path/to/raw_reads/ \
	-i gzfastq -e ndeI -c -q -r -E phred33
.......
```

You can find full details on process_radtags [here](https://catchenlab.life.illinois.edu/stacks/comp/process_radtags.php){target=_blank}.

## Reference Based Assembly ##

We'll assume you demultiplex your reads before running the pipeline described below. The contents of this and the following section should all be put in a PBS/Slurm script, just make sure your directory names are correct. This method has several intermediate files, so we'll make directories to keep them separate (do this outside of script):

```bash
mkdir raw_reads
mkdir sam_files
mkdir bam_files
mkdir stacks_out
mkdir populations_out
```

The modules you need are stacks, bwa, and samtools. All are availible on Conda, but you will almost certainly be running this on a CARC cluster, which has recent versions of all three installed:

```bash
module load stacks-2.41-gcc-7.4.0-7r6auk7
module load bwa-0.7.17-intel-18.0.2-7jvpfu2
module load samtools-1.10-gcc-9.3.0-python3-ikifznw
```

We'll also set some variables for refering to paths to stuff. We assume that the reference names (ReferenceBaseName):

```bash
src=$PBS_O_WORKDIR
bwa_ref=$src/ReferenceBaseName
threads=[number of threads]
```

Next, we need to index our reference:

```bash
bwa index -p $bwa_ref $bwa_ref.fa
```

This is the big step, which uses the Burroughs-Wheeler Aligner to align our reads to our reference. Note that this should be able to be done using pipes, but I've had issues with that, so we just remove the files at the end of each loop.

```bash
while read indiv
do
	# echo is only to help you keep track of where the pipeline is
	echo ${indiv}
	# align reads to reference genome
	bwa mem -t $threads $bwa_ref $src/raw_reads/${indiv}.fq.gz > $src/sam_files/${indiv}.sam
	# covert sam file to bam file (makes it much smaller and easier to work with)
	samtools view -bS $src/sam_files/${indiv}.sam > $src/bam_files/${indiv}_unsort.bam
	# sorts bam file, which is needed for next analyses
	samtools sort $src/bam_files/${indiv}_unsort.bam -o $src/bam_files/${indiv}.bam
	rm $src/sam_files/${indiv}.sam
	rm $src/bam_files/${indiv}_unsort.bam
done < sample_list
```

The next step is to run gstacks, which runs these "traditional" genomics files into something Stacks can work with to make output files:

```bash
gstacks -I $src/bam_files/ -M $src/popmap -O $src/stacks_out/ -t $threads
```

Finally, we run populations! This specific command will give us a 75% complete matrix of SNPs, one random SNP per locus, F<sub>st</sub> values, PLINK .bed and .map files, and use kernel smoothing for specific statistics. We also output a Variant Call Format (VCF) file, which can be used to generate inputs for most programs:
	
```bash
populations -P $src/stacks_out/ -M $src/popmap -O $src/populations_out/ \
	--vcf -R .75 --write-random-snp --fstats --plink --smooth -t $threads
```
	
A quick note, if you want input for RAxML or similar phylogenetic programs, you can get a interleaved phylip file by making a popmap file with each individual having its own population and specifying you want a phylip output. Please note that this is strict phylip format, meaning you want a maximum of 9 letters in your "population" column of the popmap (10 works, but will cause errors when input to certain programs). Also, this assumes you have a directory "populations_individual":

```bash
populations -P $src/stacks_out/ -M $src/popmap_individual -O $src/popualtions_individual/ \
	-R .75 --phylip-var-all -t $threads
```

Learn more about the outputs and options for populations [on the Stacks website](https://catchenlab.life.illinois.edu/stacks/comp/populations.php){target=_blank}. Also, as mentioned above, you can easily subset your data by changing the popmap used in gstacks and populations, as each sample has alignments performed separately.

## DeNovo Assembly ##

This is a lot simpler, but is generally considered less robust than a reference-based approach. It is described in full [here](https://catchenlab.life.illinois.edu/stacks/comp/denovo_map.php){target=_blank}. There is a lot to think about for parameters when building loci, and we use default ones here, [read up on the Stacks website about them](https://catchenlab.life.illinois.edu/stacks/param_tut.php){target=_blank}.

First, you only need the Stacks module and one new directory (assumes reads are in "raw_reads").

```bash
mkdir stacks_denovo
module load stacks-2.41-gcc-7.4.0-7r6auk7
src=$PBS_O_WORKDIR
threads=[number of threads]
```

Then you just run a single line!

```bash
denovo_map.pl -T $threads -o $src/stacks_denovo/ --popmap $src/popmap --samples $src/raw_reads/ \
	-X "<Extra parameters for populations here>"
```

As I mentioned above, I mostly included this to demystify DeNovo Stacks, please read more on it before running anything! There are many parameters to optimize.

<p class="carc-provenance" markdown>Migrated from [UNM-CARC QuickBytes](https://github.com/UNM-CARC/QuickBytes/blob/master/Stacks_quickbyte.md){target=_blank} (last source update 2021-03-03). Spotted a problem? [Open an issue or pull request](https://github.com/UNM-CARC/QuickBytes){target=_blank}.</p>

---8<--- https://unm-carc.github.io/docs/tutorials/vasp/

---
title: "VASP materials simulation"
description: "Set up and run VASP density-functional-theory calculations on CARC clusters."
type: Tutorial
tags:
  - Materials science
  - Chemistry
generated:
  by: "claude/fable-5"
  at: "2026-08-29T00:00:00Z"
sources:
  - id: quickbytes
    resource: "https://github.com/UNM-CARC/QuickBytes/blob/master/test_vasp_quickbyte.md"
    title: "UNM-CARC QuickBytes: test_vasp_quickbyte.md"
    author: "team:unm-carc"
    last_modified: "2026-07-06T14:19:00-06:00"
---

# VASP materials simulation

## Software Description

VASP performs electronic-structure calculations used in materials science, chemistry, and condensed-matter physics. A VASP run typically starts from four input files: `INCAR`, `POSCAR`, `POTCAR`, and `KPOINTS`. This QuickByte shows the Slurm pattern for launching a small parallel VASP job on Easley using a tiny NaCl example.

NOTE: You'll need to be in the `vasp6` group at UNM CARC and have a valid VASP license to run this example and other future VASP jobs. See the `Troubleshooting` section below if you're unsure whether you have access to the software and extra steps regarding how to obtain it.

This tutorial includes three small text input files in `vasp_assets`:

- [INCAR](../assets/files/vasp/INCAR): VASP calculation settings
- [POSCAR](../assets/files/vasp/POSCAR): NaCl crystal structure
- [KPOINTS](../assets/files/vasp/KPOINTS): k-point mesh

VASP `POTCAR` files are licensed pseudopotential files and are not included here. Before submitting the job, licensed VASP users should create `vasp_assets/POTCAR` for this NaCl example using the matching PAW/PBE potentials for `Na_pv` and `Cl`. See [vasp_assets/README_POTCAR.md](../assets/files/vasp/README_POTCAR.txt).

## Running the Example Slurm Script

First, log in to Easley via SSH.

`ssh user@easley.alliance.unm.edu`

Once logged into the machine, create a separate directory for this QuickByte called `example_VASP` with:

`mkdir example_VASP`

Once the directory is created, go into `example_VASP` and create a new directory called `vasp_assets`. Copy the provided `INCAR`, `POSCAR`, and `KPOINTS` files inside the `vasp_assets` directory. In addition, also add the `POTCAR` file into the `vasp_assets` directory. Use `ls` to verify that all four files are present in `vasp_assets` before continuing on with this QuickByte.

After placing the four files in the correct directory, go back to the `example_VASP` directory and use your favorite text editor to create `vasp_easley.slurm`. In this QuickByte, nano is used due to its user friendliness. Run the command below:

`nano vasp_easley.slurm`

Copy the Slurm script below into `vasp_easley.slurm` while in the text editor:

```bash
#!/bin/bash
# Run this file with: sbatch vasp_easley.slurm
# This script demonstrates a small multi-node VASP MPI run on Easley.

# Slurm resources for a short two-node VASP MPI smoke test.
#SBATCH --job-name=test-vasp
#SBATCH --output=%x-%j.out
#SBATCH --error=%x-%j.err
#SBATCH --partition=debug
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=4
#SBATCH --cpus-per-task=1
#SBATCH --mem=4G
#SBATCH --time=00:05:00

set -euo pipefail

# Start from the directory where you submitted the job. It should contain the
# vasp_assets directory.
submit_dir="${SLURM_SUBMIT_DIR:-$PWD}"
asset_dir="$submit_dir/vasp_assets"

for input_file in INCAR POSCAR POTCAR KPOINTS; do
    if [[ ! -f "$asset_dir/$input_file" ]]; then
        echo "Missing required VASP input file: $asset_dir/$input_file" >&2
        if [[ "$input_file" == "POTCAR" ]]; then
            echo "POTCAR files are licensed. Create vasp_assets/POTCAR from your licensed VASP pseudopotentials." >&2
        fi
        exit 2
    fi
done

for input_file in INCAR POSCAR POTCAR KPOINTS; do
    if [[ ! -s "$asset_dir/$input_file" ]]; then
        echo "Required VASP input file is empty: $asset_dir/$input_file" >&2
        exit 2
    fi
done

if ! awk 'NR == 6 { exit !($1 == "Na" && $2 == "Cl") }' "$asset_dir/POSCAR"; then
    echo "POSCAR should list Na and Cl as the element names on line 6." >&2
    exit 2
fi

if ! awk 'NR == 7 { exit !($1 == 1 && $2 == 1) }' "$asset_dir/POSCAR"; then
    echo "POSCAR should list one Na atom and one Cl atom on line 7." >&2
    exit 2
fi

if ! grep -q "PAW_PBE Na_pv" "$asset_dir/POTCAR" || ! grep -q "PAW_PBE Cl" "$asset_dir/POTCAR"; then
    echo "POTCAR should contain the PAW/PBE Na_pv and Cl potentials for this example." >&2
    exit 2
fi

# Create a clean per-job output directory inside the submission directory.
run_dir="$submit_dir/outputs/${SLURM_JOB_NAME}-${SLURM_JOB_ID}"
rm -rf "$run_dir"
mkdir -p "$run_dir"
cd "$run_dir"

# Copy the VASP input files into the run directory so outputs stay together.
cp "$asset_dir"/INCAR .
cp "$asset_dir"/POSCAR .
cp "$asset_dir"/POTCAR .
cp "$asset_dir"/KPOINTS .

# Load VASP.
module purge
module load vasp/6.5.1

# Launch VASP with one MPI rank per Slurm task using PMI2.
srun --mpi=pmi2 vasp_std > vasp.out
```

NOTE: In this script, we load the default version of VASP on Easley with `module load vasp/6.5.1`. For future references, if your workflow requires a different version of VASP, you can replace the version number in the `module load` command.

The important Slurm resource lines in the script are the `#SBATCH` directives near the top of the script. In this example, `--nodes=2` requests two compute nodes. `--ntasks-per-node=4` runs 4 MPI processes on each node, resulting in 8 MPI ranks being allocated. `--partition=debug` means the job is submitted specifically in the debug partition. `--time=00:05:00` limits the activity of the job to 5 minutes. The `module load` command loads the VASP software environment required to run the simulation. `srun` is used when the application should be launched through Slurm across allocated tasks.

After copying the script above, exit the file with `Ctrl + X`, then type `y` to save the modified buffer. If it asks for a filename to write to, press `Enter` to write to the newly created file. Once the file is saved, submit the job to the Slurm scheduler with:

`sbatch vasp_easley.slurm`

After submitting the job, Slurm will print a message that contains a job ID. 

`Submitted batch job <jobID>`

Take note of this job ID as it will be used when checking the results of the job.

## Example output

After the job finishes, a new directory called `outputs` should appear. In the `outputs` directory, the results of each job will be contained in a new directory called `test-vasp-<jobID>`. For this QuickByte, the `test-vasp-<jobID>` directory should contain the 4 input files mentioned earlier and VASP output files such as `OUTCAR`, `OSZICAR`, and `vasprun.xml`. The output files may change depending on your input settings.

In addition to these new directories, Slurm should report a completed job with exit code `0:0`. To check this, run `sacct -j <jobID>`. You should get a similar output below:

```text
JobID           JobName  Partition    Account  AllocCPUS      State ExitCode
------------ ---------- ---------- ---------- ---------- ---------- --------
<jobID>       test-vasp      debug   <acctID>          8  COMPLETED      0:0
<jobID>.batch     batch              <acctID>          4  COMPLETED      0:0
<jobID>.exte+    extern              <acctID>          8  COMPLETED      0:0
<jobID>.0      vasp_std              <acctID>          8  COMPLETED      0:0
```

For a successful run, the Slurm state should be `COMPLETED`, the exit code should be `0:0`, and VASP should finish without reporting fatal errors in the Slurm output or VASP output files.

## Troubleshooting

If VASP reports `POSCAR found : 0 types and 0 ions`, check that `vasp_assets/POSCAR` is the NaCl file from this tutorial and is not empty. The Slurm script checks this before starting VASP so the job fails early with a clearer message.

If VASP reports `LUSE_VDW needs to be set to .TRUE.`, check that the `POTCAR` was built from the PAW/PBE `Na_pv` and `Cl` potentials described above. That error usually means the copied `POTCAR` does not match this NaCl example.
 
A common issue that new UNM CARC users may experience when running this script is that they may not have the permissions required to run the VASP software. If your job fails, go into the `example_VASP` directory and check the Slurm error file by running:

`cat test-vasp-<jobID>.err`

If the `cat` command prints that you need to be in the "vasp6" group, please email `help@carc.unm.edu` mentioning that you need access to the VASP software. In addition, please include your research group's license number in the email.

<p class="carc-provenance" markdown>Migrated from [UNM-CARC QuickBytes](https://github.com/UNM-CARC/QuickBytes/blob/master/test_vasp_quickbyte.md){target=_blank} (last source update 2026-07-06). Spotted a problem? [Open an issue or pull request](https://github.com/UNM-CARC/QuickBytes){target=_blank}.</p>

---8<--- https://unm-carc.github.io/docs/faq/general/

---
title: "General FAQ"
description: "Quick answers about accounts, projects, cost, storage, software, and GPUs at CARC."
type: Reference
tags:
  - FAQ
  - New users
generated:
  by: "claude/fable-5"
  at: "2026-08-29T00:00:00Z"
sources:
  - id: carc-web
    resource: "https://carc.unm.edu/index.html"
    title: "CARC website (carc.unm.edu)"
    author: "team:unm-carc"
---

# General FAQ

## Accounts and projects

??? question "Who can use CARC?"

    CARC resources are free of charge for UNM faculty, staff, and student
    researchers, supported by the Office of the Vice President for Research.
    External collaborators can get accounts through a UNM project PI — see
    [Getting started](../getting-started/overview.md).

??? question "How do I get an account?"

    UNM affiliates use the self-service portal at
    [mokey.alliance.unm.edu](https://mokey.alliance.unm.edu){ target=_blank };
    collaborators without a UNM email are requested by their PI. Details in
    [Getting started](../getting-started/overview.md).

??? question "I have an account — why can't I run jobs?"

    Your account must belong to a **CARC project** that holds compute
    resources. Ask your PI to add you to their project in
    [ColdFront](https://coldfront.alliance.unm.edu){ target=_blank }. Note
    that your Slurm account is not the same as your login account — see
    [Slurm accounting and fairshare](../running-jobs/slurm-accounting.md).

??? question "How do I become a project PI?"

    PI eligibility follows the
    [UNM criteria for Principal Investigator status](https://osp.unm.edu/pi-resources/pi-eligibility.html){ target=_blank }.
    Eligible PIs create projects and request resources in ColdFront.

## Cost and allocations

??? question "Does CARC cost anything?"

    The core service is free for UNM researchers. Additional dedicated
    storage and premium services are available for purchase — see
    [premium research computing services](https://carc.unm.edu/research/premium-research-computing-services.html){ target=_blank }.

??? question "How is fair access enforced?"

    Through Slurm's fairshare system: heavy recent usage lowers your
    scheduling priority relative to lighter users, so everyone gets a fair
    opportunity. See [resource limits](../systems/resource-limits.md) and
    [Slurm accounting](../running-jobs/slurm-accounting.md).

??? question "What if I need more than CARC can provide?"

    National platforms are the next step — ACCESS-CI allocations, the
    Jetstream2 cloud, and CyVerse data services. See
    [partner cyberinfrastructure](../about/partners.md); CARC staff can help
    you apply.

## Storage and data

??? question "Where should I put my data?"

    Home for small, important files; project space for shared work; scratch
    for active job I/O. Quotas and the `quotas` command are covered in
    [resource limits](../systems/resource-limits.md), and the layout in
    [storage and backups](../systems/storage.md).

??? question "Is my data backed up?"

    Enterprise (NetApp) storage has automated snapshots (hourly to monthly,
    retained up to four months). **Scratch is not backed up.** See
    [storage and backups](../systems/storage.md).

??? question "Can I store HIPAA / regulated data on CARC?"

    No. HIPAA, PHI, PCI, FERPA, and CUI data may not be stored on or
    transferred via CARC systems — see the
    [Good Neighbor Use Policy](../getting-started/good-neighbor-policy.md).

## Software and hardware

??? question "How do I get software installed?"

    First check `module avail` ([environment modules](../running-jobs/modules.md)).
    You can install your own stacks with [conda](../software/conda-intro.md)
    or run [containers](../software/singularity.md). For system-wide
    installs, [open a ticket](../support/help.md). Export-controlled software
    requires prior written approval from UNM Export Control.

??? question "What GPUs are available?"

    Easley has NVIDIA L40S and H100 GPUs, and Hopper has A100s — see the
    [systems overview](../systems/overview.md). Request GPU partitions in
    your job script; examples are in
    [example Slurm scripts](../running-jobs/example-slurm-scripts.md).

??? question "Can I use CARC from my browser?"

    Yes — [Open OnDemand](../interactive/open-ondemand.md) for files, shells,
    and jobs, or [JupyterHub](../interactive/jupyterhub.md) for notebooks.

## Publishing

??? question "How do I acknowledge CARC in a paper?"

    Use the statement on [acknowledging CARC](../support/acknowledging-carc.md)
    and add the publication to your project's list in ColdFront.

---8<--- https://unm-carc.github.io/docs/faq/troubleshooting/

---
title: "Troubleshooting"
description: "Diagnose the most common problems: login failures, quota errors, pending or failing jobs, and module conflicts."
type: Guide
tags:
  - FAQ
  - Support
generated:
  by: "claude/fable-5"
  at: "2026-08-29T00:00:00Z"
sources:
  - id: carc-web
    resource: "https://carc.unm.edu/index.html"
    title: "CARC website (carc.unm.edu)"
    author: "team:unm-carc"
---

# Troubleshooting

Work through the section that matches your symptom. If nothing here fixes it,
[open a ticket](../support/help.md) with the details listed at the bottom of
that page — cluster, job ID, exact command, and full error message.

## I can't log in

1. **Check for maintenance** on the
   [cluster status monitor](https://stats.uptimerobot.com/kqt0LYLwFd){ target=_blank }
   and [UNM IT alerts](https://italerts.unm.edu/){ target=_blank }.
2. **Password or OTP problems** — reset via the steps in
   [password reset](../getting-started/password-reset.md).
3. **`Permission denied (publickey)`** — your SSH key setup is incomplete or
   has wrong permissions; see [SSH keys](../getting-started/ssh-keys.md)
   (`~/.ssh` must be `700`, private keys `600`).
4. **Account exists but no access** — you may not be on an active project
   yet; ask your PI to add you in ColdFront
   ([getting started](../getting-started/overview.md)).

## "Disk quota exceeded"

You've hit a storage limit ([what the limits are](../systems/resource-limits.md)):

```bash
quotas            # show your usage against each quota
du -sh ~/* | sort -rh | head   # find what's using home space
```

Clean up, move bulk data to scratch or project space
([storage layout](../systems/storage.md)), or talk to us about
[purchasing more](https://carc.unm.edu/research/premium-research-computing-services.html){ target=_blank }.
Remember conda environments and pip caches grow quietly — `conda clean --all`
and `pip cache purge` often free gigabytes.

## My job won't start

```bash
squeue -u $USER           # state and reason code
squeue --start --job <id> # predicted start time (fairshare-aware)
sinfo                     # partition and node availability
```

Common reason codes:

| Reason | Meaning | What to do |
| ------ | ------- | ---------- |
| `Priority` | Others are ahead of you (fairshare) | Wait, or request fewer/shorter resources; see [fairshare](../running-jobs/slurm-accounting.md) |
| `Resources` | Not enough free nodes for your request | Reduce cores/memory/GPUs or choose another partition |
| `QOSMax*` / limits | You've hit a partition or account limit | Check [resource limits](../systems/resource-limits.md) |
| `ReqNodeNotAvail` | Nodes down or reserved (often maintenance) | Check the [cluster status monitor](https://stats.uptimerobot.com/kqt0LYLwFd){ target=_blank } |
| `InvalidAccount` | Wrong `--account` | List yours: `sacctmgr show assoc user=$USER format=account` |

## My job failed or was killed

```bash
sacct -j <id> --format=JobID,State,ExitCode,Elapsed,MaxRSS,ReqMem
seff <id>     # efficiency summary after completion
```

* **`OUT_OF_MEMORY` / `oom-kill`** — request more memory (`--mem` or
  `--mem-per-cpu`) or use fewer tasks per node; `seff` shows what you
  actually used.
* **`TIMEOUT`** — raise `--time` within partition limits, or checkpoint and
  restart.
* **Immediate crash** — check the job's `.out`/`.err` files in the submit
  directory; a missing `module load` or unactivated conda environment is the
  usual culprit ([modules](../running-jobs/modules.md),
  [conda](../software/conda-environments.md)).

## Software and environment problems

* **`command not found`** — load the module first (`module spider <name>`
  to find it; [modules guide](../running-jobs/modules.md)).
* **Conda is slow or conflicts** — prefer clean per-project environments and
  the conda-forge channel; see [channels and pip](../software/conda-channels-pip.md).
* **GPU code can't see the GPU** — did you request one in the job
  (`--gres=gpu:1` or the cluster's GPU partition)? Verify with `nvidia-smi`
  inside the job; see [example Slurm scripts](../running-jobs/example-slurm-scripts.md).
* **My kernel is missing in JupyterHub** — register your environment as a
  kernel: [conda in JupyterHub](../software/conda-jupyterhub.md).

## Graphics won't display

X11 applications need forwarding enabled — `ssh -Y` and a local X server;
see [X11 forwarding](../getting-started/x11-forwarding.md). For heavier
visualization, use [ParaView client–server](../software/paraview.md) or an
[Open OnDemand](../interactive/open-ondemand.md) session instead.

## Transfers are slow or failing

Use `rsync` with resume (`rsync -avP`) rather than `scp` for large trees,
and transfer to the right storage tier — see
[transferring data](../getting-started/transferring-data.md).

## Still stuck?

[Open a ticket](../support/help.md) or bring it to office hours — include
your cluster, job ID, command, and the complete error text.

---8<--- https://unm-carc.github.io/docs/training/videos/

---
title: "Video tutorials"
description: "Embedded CARC recordings: the QuickBytes tutorial series, CARC Annual Meeting talks, and research presentations from the UNMCARC YouTube channel."
type: Reference
tags:
  - Training
  - Videos
generated:
  by: "claude/fable-5"
  at: "2026-08-29T00:00:00Z"
sources:
  - id: yt-quickbytes
    resource: "https://www.youtube.com/playlist?list=PLvr5gRBLi7VAzEB_t5aXOLHLfdIu2s1hZ"
    title: "QuickBytes playlist (YouTube)"
    author: "team:unm-carc"
  - id: yt-cam2026
    resource: "https://www.youtube.com/playlist?list=PLvr5gRBLi7VCveJGJHzQrtmI4Jw0AHVP7"
    title: "UNM CARC Annual Meeting (CAM) 2026 playlist (YouTube)"
    author: "team:unm-carc"
  - id: yt-channel
    resource: "https://www.youtube.com/@UNMCARC/videos"
    title: "UNMCARC YouTube channel"
    author: "team:unm-carc"
---

# Video tutorials

All CARC recordings live on the
[UNMCARC YouTube channel](https://www.youtube.com/@UNMCARC){target=_blank}.
The essentials are embedded below, and the most-watched tutorials are also
embedded directly on their matching guide pages.

## QuickBytes: Introduction to Computing at CARC

A short-episode series covering everything a new user needs, in order. Watch
it straight through here, or jump to an episode:

<iframe class="carc-video" src="https://www.youtube-nocookie.com/embed/videoseries?list=PLvr5gRBLi7VAzEB_t5aXOLHLfdIu2s1hZ" title="QuickBytes: Introduction to Computing at CARC (playlist)" loading="lazy" allow="accelerometer; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen></iframe>

| Episode | Length | Related guide |
| ------- | :----: | ------------- |
| [Intro to the UNM Center for Advanced Research Computing](https://www.youtube.com/watch?v=98lCb6A5uu4){target=_blank} | 5:53 | [About CARC](../about/index.md) |
| [Creating an Account](https://www.youtube.com/watch?v=ewyqLP-Eo_4){target=_blank} | 4:58 | [Getting started](../getting-started/overview.md) |
| [CARC Project Management with ColdFront](https://www.youtube.com/watch?v=HJcjOSNWp1U){target=_blank} | 12:50 | [Getting started](../getting-started/overview.md) |
| [Annual Project Review](https://www.youtube.com/watch?v=MGxMfmShVWk){target=_blank} | 7:45 | [Getting started](../getting-started/overview.md) |
| [Logging into CARC Systems](https://www.youtube.com/watch?v=Puhaf6tCNO0){target=_blank} | 12:48 | [Logging in](../getting-started/logging-in.md) |
| [Slurm Job Scheduler](https://www.youtube.com/watch?v=cIRyG8C3CVk){target=_blank} | 1:24:06 | [Introduction to Slurm](../running-jobs/slurm-intro.md) |
| [Storage Systems](https://www.youtube.com/watch?v=WwsbLyl7d1A){target=_blank} | 13:11 | [Storage and backups](../systems/storage.md) |
| [Transferring data](https://www.youtube.com/watch?v=2UphEzHOHGM){target=_blank} | 21:56 | [Transferring data](../getting-started/transferring-data.md) |
| [Environment Modules](https://www.youtube.com/watch?v=tz-w3vc7cGQ){target=_blank} | 6:04 | [Environment modules](../running-jobs/modules.md) |
| [Conda environments](https://www.youtube.com/watch?v=gMJzDSeGk50){target=_blank} | 12:40 | [Managing conda environments](../software/conda-environments.md) |
| [X11 Forwarding](https://www.youtube.com/watch?v=-5ic9JWHuqI){target=_blank} | 10:41 | [X11 forwarding](../getting-started/x11-forwarding.md) |
| [Parallelization 1: An Example Problem](https://www.youtube.com/watch?v=UWAReJqVIfU){target=_blank} | 16:14 | — |
| [Parallelization 3: GNU Parallel](https://www.youtube.com/watch?v=Rl06WD60afA){target=_blank} | 28:01 | [GNU Parallel](../running-jobs/gnu-parallel.md) |
| [Parallelization 4: The Message Passing Interface (MPI)](https://www.youtube.com/watch?v=JrCMi3_yCA0){target=_blank} | 24:27 | [MPI from JupyterHub](../software/jupyterhub-mpi.md) |
| [Parallel R with Future](https://www.youtube.com/watch?v=G5xGfF151Co){target=_blank} | — | [Parallel R with the future package](../software/parallel-r-future.md) |
| [Computational Chemistry: Avogadro and Gaussian](https://www.youtube.com/watch?v=Jqyf7bodT4U){target=_blank} | 28:38 | — |
| [CS491/591: SimCov Compilation and Submission](https://www.youtube.com/watch?v=7x3voqNX0QY){target=_blank} | 8:14 | [SimCov](../tutorials/simcov.md) |

## CARC Annual Meeting 2026

Recorded talks from [CAM 2026](https://www.youtube.com/playlist?list=PLvr5gRBLi7VCveJGJHzQrtmI4Jw0AHVP7){target=_blank}:

<iframe class="carc-video" src="https://www.youtube-nocookie.com/embed/videoseries?list=PLvr5gRBLi7VCveJGJHzQrtmI4Jw0AHVP7" title="UNM CARC Annual Meeting (CAM) 2026 (playlist)" loading="lazy" allow="accelerometer; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen></iframe>

* [Director Address — Patrick Bridges](https://www.youtube.com/watch?v=pOvjeu3FYRk){target=_blank}
* [Team Updates](https://www.youtube.com/watch?v=oICidaVD5ys){target=_blank}
* [CARC Panel](https://www.youtube.com/watch?v=lqW3BnMNCkg){target=_blank}
* [Security and Data Governance — Jeff Gassoway](https://www.youtube.com/watch?v=eFlZU-rgTmA){target=_blank}
* [Quantum Information Science at the University of New Mexico — Ivan Deutsch](https://www.youtube.com/watch?v=zL_7ix9vcjE){target=_blank}
* [Peptide Design Targeting PICK1 PDZ Domain Utilizing Computational Approaches — Emily Hendrix](https://www.youtube.com/watch?v=ooEeijiCxN4){target=_blank}
* [Accelerating Behavioral Neuroscience with Supercomputing and Deep Learning — Monica Gonçalves Garcia](https://www.youtube.com/watch?v=FEBGn5l2qck){target=_blank}
* [New Stream-Triggered MPI Abstractions — Patrick Bridges](https://www.youtube.com/watch?v=bseoEOt1IXY){target=_blank}

## CARC Annual Meeting 2025

Recorded talks from CAM 2025:

<iframe class="carc-video" src="https://www.youtube-nocookie.com/embed/0-buawjFOyg" title="CAM2025 - State of the Center - Patrick Bridges" loading="lazy" allow="accelerometer; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen></iframe>

* [State of the Center — Patrick Bridges](https://www.youtube.com/watch?v=0-buawjFOyg){target=_blank}
* [Center Update — Matthew Fricke, Hussein Al-Azzawi, and Anneliese Ward](https://www.youtube.com/watch?v=rNuiBw6cqf0){target=_blank}
* [The first five years of research using the New Mexico Decedent Database — Heather Edgar](https://www.youtube.com/watch?v=s6da-a0S8UM){target=_blank}
* [Chemical Reactions from Very Cold to Very Hot: A Computational Perspective — Hua Guo](https://www.youtube.com/watch?v=KB6us-Ts2Io){target=_blank}
* [Pathogen Genomics — Daryl Domman](https://www.youtube.com/watch?v=HUcWsxICfKQ){target=_blank}
* [Oh the Gall! Plant-manipulating insects — Ellen Martinson](https://www.youtube.com/watch?v=8FXI7fwp0WM){target=_blank}
* [CARC Students](https://www.youtube.com/watch?v=qfvK9ViTeKU){target=_blank}

## Research talks and center events

* [CARC AI Pipelining Support — Matthew Fricke](https://www.youtube.com/watch?v=qW245p1C5-I){target=_blank}
* [ML as an Aid to Muon and Gravimetric Volcano Feature Detection — Katherine Cosburn](https://www.youtube.com/watch?v=mEr6XnlZtgk){target=_blank}
* [Pulsars and Population Dynamics — Ethan Gyllenhaal and Seth Bruzewski](https://www.youtube.com/watch?v=N--A8GH6_SY){target=_blank}
* [Jacob McCullough CARC Presentation](https://www.youtube.com/watch?v=qLqAQDc_w_g){target=_blank}

!!! tip "Suggest a topic"

    Is there a video you wish existed? Let us know at <help@carc.unm.edu> or
    during [office hours](../support/help.md). Slides from live workshops are
    on the [workshops page](workshops.md).

---8<--- https://unm-carc.github.io/docs/training/workshops/

---
title: "Workshops and slides"
description: "Slide decks from CARC workshops and university courses, organized by series, plus how to hear about upcoming sessions."
type: Reference
tags:
  - Training
  - Workshops
generated:
  by: "claude/fable-5"
  at: "2026-08-29T00:00:00Z"
sources:
  - id: quickbytes
    resource: "https://github.com/UNM-CARC/QuickBytes/blob/master/workshop_slides.md"
    title: "UNM-CARC QuickBytes: workshop_slides.md and workshop_slides/ PDFs"
    author: "team:unm-carc"
    last_modified: "2023-02-03T11:54:26-07:00"
---

# Workshops and slides

The CARC team teaches regular workshops — from first-login basics to
domain-specific computing — and guest-lectures in UNM courses that use CARC
systems. Slides from those sessions are archived here as PDFs.

!!! tip "Upcoming workshops"

    Sessions are announced on the
    [CARC workshops & seminars page](https://carc.unm.edu/education--training/workshops-and-seminars.html){target=_blank}
    and to users by email. Prefer video? Many workshop topics are also
    covered in the [video tutorials](videos.md).

## Introduction to CARC — workshop series

The core workshop, offered every semester at two levels.

| Slides | Level |
| ------ | ----- |
| [Introduction to CARC — Beginner (2025)](../assets/files/workshops/IntroToCARCWorkshopBeginner2025-0.2.pdf) | Beginner |
| [Introduction to CARC — Beginner](../assets/files/workshops/BeginnerIntro.pdf) | Beginner |
| [Introduction to CARC — Beginner (2022)](../assets/files/workshops/beginner_intro_slides_2022.pdf) | Beginner |
| [Introduction to CARC — Intermediate (2025, v0.5, Zoom)](../assets/files/workshops/IntroToCARCWorkshopIntermediate2025-0.5_Zoom_Version.pdf) | Intermediate |
| [Introduction to CARC — Intermediate (2025, v0.4, Zoom)](../assets/files/workshops/IntroToCARCWorkshopIntermediate2025-0.4_Zoom_Version.pdf) | Intermediate |
| [Introduction to CARC — Intermediate (2025, v0.4)](../assets/files/workshops/IntroToCARCWorkshopIntermediate2025-0.4.pdf) | Intermediate |
| [Introduction to CARC — Intermediate (2025, v0.3)](../assets/files/workshops/IntroToCARCWorkshopIntermediate2025-0.3.pdf) | Intermediate |
| [Introduction to CARC — Intermediate](../assets/files/workshops/IntermediateIntro.pdf) | Intermediate |
| [Introduction to CARC — 1-hour version for programmers](../assets/files/workshops/IntroToCARCWorkshopIntermediate_1hr_version_for_programmers.pdf) | Intermediate |

## Domain-focused workshops

Introductions to CARC through the lens of a research domain or application.

| Slides | Topic |
| ------ | ----- |
| [Intro to CARC with Quantum Computing (2025)](../assets/files/workshops/IntroToCARCWithQuantumComputing2025-0.1.pdf) | Quantum computing |
| [Intro to CARC with Computational Fluid Dynamics (2025)](../assets/files/workshops/IntroToCARCWithComputationalFluidDynamics2025-0.1.pdf) | CFD |
| [Intro to CARC with Crystallography (2025)](../assets/files/workshops/IntroToCARCWithCrystallography2025-0.1.pdf) | Crystallography |
| [Intermediate workshop: NAMD and AlphaFold 3](../assets/files/workshops/IntermediateIntroToCARCWorkshopNAMDandAlphafold3.pdf) | Molecular dynamics / protein structure |
| [Intermediate workshop: NAMD](../assets/files/workshops/IntermediateIntroToCARCWorkshopNAMD.pdf) | Molecular dynamics |
| [Intermediate workshop: Machine Learning](../assets/files/workshops/IntermediateIntroToCARCWorkshopMachineLearning.pdf) | Machine learning |
| [Machine Learning at CARC](../assets/files/workshops/Machine_Learning.pdf) | Machine learning |
| [Intermediate workshop: Gaussian (1 hour)](../assets/files/workshops/IntermediateIntroToCARCWorkshopGaussian_1hr.pdf) | Computational chemistry |
| [Slurm with Gaussian and Abinit](../assets/files/workshops/IntroToCARCWorkshopSlurmGaussianAbinit.pdf) | Computational chemistry |
| [Intermediate workshop: CASA](../assets/files/workshops/IntermediateIntroToCARCWorkshopCASA.pdf) | Radio astronomy |
| [Intro to CARC with MESA star modeling](../assets/files/workshops/IntroToCARCWorkshopMesaStar.pdf) | Stellar astrophysics |
| [Intermediate workshop: FORTRAN (2024)](../assets/files/workshops/IntroToCARCWorkshopIntermediate2024-0.1-FORTRAN.pdf) | Fortran |
| [Intermediate workshop: STAR-CCM+ (2024)](../assets/files/workshops/IntroToCARCWorkshopIntermediate2024-0.2-StarCCM.pdf) | CFD (STAR-CCM+) |
| [CARC ESCAPE workshop](../assets/files/workshops/CARC_ESCAPE_WORKSHOP-0.1.pdf) | ESCAPE program |

## Course guest lectures

Slides prepared for UNM courses that compute on CARC systems.

| Slides | Course |
| ------ | ------ |
| [BIOL 419/519](../assets/files/workshops/BIOL419_519.pdf) | Biology |
| [CHEM 567](../assets/files/workshops/CHEM567.pdf) | Chemistry |
| [CS 442](../assets/files/workshops/CS442.pdf) | Computer Science |
| [CS 442 (50-minute version)](../assets/files/workshops/CS442_50min.pdf) | Computer Science |
| [CS 523](../assets/files/workshops/CS523.pdf) | Computer Science |
| [CS/Math 471](../assets/files/workshops/CS_Math_471.pdf) | Scientific Computing |
| [ENVS 352](../assets/files/workshops/ENVS352.pdf) | Environmental Science |
| [EPS 522](../assets/files/workshops/EPS522.pdf) | Earth & Planetary Science |

!!! question "Want a workshop for your group or course?"

    CARC staff regularly deliver tailored sessions for research groups and
    classes. Email <help@carc.unm.edu> or visit
    [office hours](../support/help.md) to arrange one.

<p class="carc-provenance" markdown>Slide decks migrated from [UNM-CARC QuickBytes](https://github.com/UNM-CARC/QuickBytes/tree/master/workshop_slides){target=_blank}; page curated in this repository. Spotted a problem? [Open an issue or pull request](https://github.com/UNM-CARC/QuickBytes){target=_blank}.</p>

---8<--- https://unm-carc.github.io/docs/support/acknowledging-carc/

---
title: "Acknowledging CARC"
description: "The acknowledgement statement to include in publications that used CARC resources."
type: Policy
tags:
  - Support
  - Publications
generated:
  by: "claude/fable-5"
  at: "2026-08-29T00:00:00Z"
sources:
  - id: carc-web
    resource: "https://carc.unm.edu/research/publications.html"
    title: "Publications and acknowledgement (carc.unm.edu)"
    author: "team:unm-carc"
---

# Acknowledging CARC

CARC's mission is to support UNM research, and tracking publications is an
important way we demonstrate our value. Publications, dissertations, theses,
poster presentations, and talks that used CARC resources should acknowledge
the center.

## Acknowledgement statement

Select one or more of the items within the braces `{ }`:

!!! quote "Acknowledgement"

    We would like to thank the UNM Center for Advanced Research Computing,
    supported in part by the National Science Foundation, for providing the
    {high performance computing, large-scale storage, visualization}
    resources used in this work.

## Register your publication

Please add publications that used CARC resources to your project's
publication list in [ColdFront](https://coldfront.alliance.unm.edu){ target=_blank }.

## Publications that acknowledge CARC

Browse the [Google Scholar listing of publications that mention CARC](https://scholar.google.com/scholar?q=%22Center+for+Advanced+Research+Computing%22+AND+%28UNM+OR+%22University+of+New+Mexico%22%29){ target=_blank } —
a snapshot of the broad spectrum of computational research at UNM.

---8<--- https://unm-carc.github.io/docs/support/help/

---
title: "Getting help"
description: "Open a help ticket, email CARC support, or drop into office and consultation hours."
type: Guide
tags:
  - Support
generated:
  by: "claude/fable-5"
  at: "2026-08-29T00:00:00Z"
sources:
  - id: carc-web
    resource: "https://carc.unm.edu/index.html"
    title: "CARC website (carc.unm.edu)"
    author: "team:unm-carc"
---

# Getting help

CARC provides not just computing resources but also the expertise and support
to help UNM's researchers succeed. Support is free for UNM faculty, staff, and
student researchers.

<div class="grid cards" markdown>

-   :material-ticket-confirmation:{ .lg .middle } __Help tickets__

    ---

    The fastest way to reach the CARC team. Create or manage a ticket in the
    support portal, or email us and a ticket is opened for you.

    [:octicons-arrow-right-24: support.alliance.unm.edu](https://support.alliance.unm.edu/){ target=_blank }

    [:octicons-mail-16: help@carc.unm.edu](mailto:help@carc.unm.edu)

-   :material-calendar-clock:{ .lg .middle } __Office & consultation hours__

    ---

    Talk with CARC staff in person or over video: debugging help, workflow
    design, proposal support, and general consulting.

    [:octicons-arrow-right-24: Current schedule](https://carc.unm.edu/user-support-2/office-and-consultation-hours.html){ target=_blank }

-   :material-book-open-variant:{ .lg .middle } __Documentation & training__

    ---

    Guides and tutorials on this site, plus [workshops](../training/workshops.md)
    and [video tutorials](../training/videos.md) from the CARC team.

    [:octicons-arrow-right-24: Getting started](../getting-started/index.md)

-   :material-list-status:{ .lg .middle } __System status__

    ---

    Check the live monitors before reporting an outage.

    [:octicons-arrow-right-24: Cluster login & website status](https://stats.uptimerobot.com/kqt0LYLwFd){ target=_blank }

    [:octicons-arrow-right-24: UNM IT alerts](https://italerts.unm.edu/){ target=_blank }

    [:octicons-arrow-right-24: Network performance (perfSONAR)](http://perfsonar.alliance.unm.edu){ target=_blank }

    [:octicons-arrow-right-24: Easley external DNS check](https://dnschecker.org/#A/easley.alliance.unm.edu){ target=_blank }

    [:octicons-arrow-right-24: System usage (XDMoD)](https://xdmod.alliance.unm.edu/){ target=_blank }

</div>

## When you open a ticket

Help us help you faster — include:

1. The **cluster** and **login node** you were using (e.g., Easley, Hopper).
2. The **job ID** (`squeue -u $USER`, or from `sacct`) for job problems.
3. The **exact command** you ran and the **complete error message**.
4. The path to your **batch script** and log files (we can read them with
   your permission).

---8<--- https://unm-carc.github.io/docs/about/ai-agents/

---
title: "For AI agents"
description: "How agents and harnesses should consume this documentation: llms.txt, per-page Markdown with OKF frontmatter, and trust signals."
type: Reference
tags:
  - About
  - AI agents
  - OKF
generated:
  by: "claude/fable-5"
  at: "2026-08-31T00:00:00Z"
sources:
  - id: okf-spec
    resource: "https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/SPEC.md"
    title: "Open Knowledge Format (OKF) v0.2 specification"
    author: "team:google-cloud"
  - id: llmstxt
    resource: "https://llmstxt.org"
    title: "The /llms.txt convention"
    author: "team:answer-ai"
---

# For AI agents

This site is published for people **and** for AI agents. The documentation
source is an [Open Knowledge Format (OKF) v0.2](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/SPEC.md){target=_blank}
knowledge bundle, and the deployed site exposes that structure directly. If
you are an agent (or you are wiring one up), consume the documentation
through these endpoints rather than scraping rendered HTML.

## Entry points

| Endpoint | What you get |
| -------- | ------------ |
| [`/llms.txt`](../llms.txt) | Linked outline of every page with one-line descriptions ([llms.txt convention](https://llmstxt.org){target=_blank}) |
| [`/llms-full.txt`](../llms-full.txt) | The entire corpus in one file — every page's Markdown with frontmatter, prefixed by its canonical URL |
| Any page URL + `index.md` | That page's Markdown source with full OKF frontmatter (e.g. `/running-jobs/slurm-intro/index.md`) |
| `/sitemap.xml`, `/robots.txt` | Standard crawl surface; robots.txt repeats these pointers |
| [Source repository](https://github.com/UNM-CARC/docs){target=_blank} | The bundle itself, plus `AGENTS.md` with contribution rules for coding agents |

Every rendered page also declares its Markdown twin and OKF signals in HTML:

```html
<link rel="alternate" type="text/markdown" href="index.md">
<meta name="okf:type" content="Guide">
<meta name="okf:status" content="stable">
<meta name="okf:trust-tier" content="unverified">
<meta name="okf:generated-at" content="2026-08-29T00:00:00Z">
```

## Reading the OKF frontmatter

Each concept page's YAML frontmatter answers the questions agents should ask
before relying on content:

* **What is this?** — `type` (`Guide`, `Tutorial`, `Reference`, `Policy`),
  `title`, `description`, `tags`.
* **Where did it come from?** — `generated: { by, at }` and `sources` (with
  `resource` URLs and `last_modified` from the upstream git history).
* **How much should I trust it?** — the `verified` key (OKF §5.3): absent
  means **unverified**; `by: "human:<netid>"` means **human-reviewed** by
  CARC staff. Prefer human-reviewed pages when answers conflict.
* **Is it still true?** — `status` (`stable` default; `draft` needs review
  against current systems; `deprecated` is kept for history only) and
  `stale_after` (an ISO 8601 instant; hardware pages carry one).

!!! warning "Retired systems"

    Wheeler, Taos, Gibbs, and **Xena** are retired. The active clusters are
    **Easley** and **Hopper** — see the [systems overview](../systems/overview.md).
    Pages mentioning retired systems carry a legacy notice and should not be
    used as a source for current cluster names, partitions, or GPU types.

## Related OKF bundles

The [GPT 101 generative-AI workshop](https://tyson-swetnam.github.io/intro-gpt/){target=_blank},
which CARC maintains and teaches, is published with the same agent
conventions as this documentation:

* [Workshop llms.txt](https://tyson-swetnam.github.io/intro-gpt/llms.txt){target=_blank} —
  AI platform setup, prompt engineering, education, research, and ethics pages.
* [Workshop agent guide](https://tyson-swetnam.github.io/intro-gpt/agents/){target=_blank} —
  endpoints and trust signals. Note its raw-source convention differs: replace
  a page URL's trailing `/` with `.md` (not `index.md` as on this site).

## Answering user questions

Ground answers in this documentation and cite the page URL. When the corpus
does not answer a question, direct users to the humans: open a ticket at
[support.alliance.unm.edu](https://support.alliance.unm.edu/){target=_blank}
or email <help@carc.unm.edu> — do not guess cluster-specific facts such as
partition names, quotas, or module versions.

---8<--- https://unm-carc.github.io/docs/about/contributing/

---
title: "Contributing to these docs"
description: "How to edit pages, the OKF frontmatter contract, verifying migrated content, and building the site locally."
type: Guide
tags:
  - About
  - Contributing
generated:
  by: "claude/fable-5"
  at: "2026-08-29T00:00:00Z"
sources:
  - id: okf-spec
    resource: "https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/SPEC.md"
    title: "Open Knowledge Format (OKF) v0.2 specification"
    author: "team:google-cloud"
---

# Contributing to these docs

This documentation is a git repository of Markdown files, built with
[Zensical](https://zensical.org){target=_blank} and structured as an
[Open Knowledge Format (OKF) v0.2](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/SPEC.md){target=_blank}
knowledge bundle — every page is readable by people *and* consumable by AI
agents, with provenance and trust signals in its frontmatter.

## Small fixes

Every page has an **edit button** (:material-pencil:) in the upper right that
opens the source file on GitHub. Fix the text, propose the change, and CI
validates and deploys it once merged.

## The frontmatter contract

Every content page starts with YAML frontmatter. `type` is required by OKF;
the rest make the page trustworthy and discoverable:

```yaml
---
title: "Page title"
description: "One sentence used by search, cards, indexes, and agents."
type: Guide            # Guide | Tutorial | Reference | Policy
tags:
  - Slurm
generated:
  by: "human:yournetid"          # who/what wrote the current content
  at: "2026-08-29T00:00:00Z"
sources:                          # where the content came from (optional)
  - id: quickbytes
    resource: "https://github.com/UNM-CARC/QuickBytes/blob/master/..."
    title: "Original source"
    author: "team:unm-carc"
status: stable          # draft | stable | deprecated (default: stable)
---
```

Section `index.md` files are OKF directory listings and carry **no
frontmatter**; `log.md` is the bundle's dated change log — add an entry when
you make a meaningful change.

## Verifying migrated pages

Pages migrated from QuickBytes were produced by an agent and are
intentionally **unverified**. When you review one and confirm it is correct
for current systems, record it:

```yaml
verified: { by: "human:yournetid", at: "2026-09-15T00:00:00Z" }
```

If a page is obsolete, don't delete it — set `status: deprecated`, add a
note pointing at the replacement, and log the change in `log.md`.

## Building locally

```bash
git clone <this repository> && cd carc_documentation
python3 -m venv .venv && source .venv/bin/activate
pip install zensical pyyaml
zensical serve                      # live preview at localhost:8000
python3 scripts/okf_validate.py docs   # OKF conformance check (runs in CI)
python3 scripts/gen_llms_txt.py        # regenerate llms.txt indexes
```

## The pipeline scripts

* `scripts/migrate_quickbytes.py` — the reproducible migration from
  [UNM-CARC/QuickBytes](https://github.com/UNM-CARC/QuickBytes){target=_blank} and
  [webinfo](https://github.com/UNM-CARC/webinfo){target=_blank}. It owns the page mapping
  and regenerates every section `index.md`; if you add a page, add it to the
  mapping there so the indexes stay complete.
* `scripts/okf_validate.py` — fails CI if any page breaks OKF conformance
  (missing frontmatter, missing `type`, malformed `log.md`, frontmatter on a
  section index).
* `scripts/gen_llms_txt.py` — builds `docs/llms.txt` (a linked site outline
  per [llmstxt.org](https://llmstxt.org){ target=_blank }) and
  `docs/llms-full.txt` (the full corpus with frontmatter) so AI assistants
  can consume the documentation directly.

## Style notes

* One `#` H1 per page, matching the frontmatter `title`.
* Relative links between pages (`../section/page.md`) — CI warns on broken ones.
* Admonitions (`!!! note`, `??? question`) for asides; content tabs for
  OS-specific instructions.
* Put images in `docs/assets/images/` and downloadable files in
  `docs/assets/files/`.

---8<--- https://unm-carc.github.io/docs/about/facilities/

---
title: "Facilities description"
description: "Boilerplate facilities description for grant proposals: clusters, storage, networking, and the data center."
type: Reference
tags:
  - About
  - Grants
stale_after: "2027-08-31T00:00:00Z"
generated:
  by: "claude/fable-5"
  at: "2026-08-29T00:00:00Z"
sources:
  - id: carc-facilities
    resource: "https://carc.unm.edu/about-carc/facilities-description.html"
    title: "Facilities description (carc.unm.edu)"
    author: "team:unm-carc"
---

# Facilities description

!!! tip "For grant proposals"

    This page is written to be copied into the *Facilities, Equipment, and
    Other Resources* section of proposals. See also
    [acknowledging CARC](../support/acknowledging-carc.md) for the
    publication acknowledgement statement.

The UNM Center for Advanced Research Computing (CARC) supports
high-performance and data-intensive research across the UNM community. Our
systems are designed to serve a wide range of disciplines, from traditional
scientific computing to advanced data analytics, artificial intelligence, and
machine learning.

## Compute

CARC currently operates several high-performance computing clusters:

**Easley Cluster** — 65 compute nodes with 4,160 total CPU cores and 23.3 TB
of RAM. Easley includes 36 NVIDIA L40S GPUs for AI and machine learning
workloads, along with 8 NVIDIA H100 GPUs for double-precision computing. The
system is connected through an NVIDIA NDR 800 Gbps InfiniBand core network
for high-speed communication between nodes.

**Hopper Cluster** — 61 compute nodes with 2,176 CPU cores and 37 NVIDIA A100
GPUs. Hopper is connected through an NVIDIA HDR 400 Gbps InfiniBand network
and supports both general and GPU-accelerated workloads.

Together, these systems support a broad ecosystem of research software,
including scientific computing tools, machine learning frameworks such as
TensorFlow and PyTorch, data analytics platforms, and interactive
environments like Jupyter, R, and Parallel MATLAB.

## Data storage and virtual infrastructure

CARC provides multiple tiers of high-performance storage to support both
active research workflows and long-term data management:

* 720 TB of all-flash IBM Storage Scale (GPFS) scratch storage
* 2 PB of BeeGFS working scratch storage
* 2.4 PB of NetApp enterprise storage

Enterprise storage includes automated snapshots taken hourly, daily, weekly,
and monthly, with retention for up to four months to support user-directed
data recovery.

In partnership with UNM Libraries, CARC also supports a virtual machine
infrastructure that enables custom research applications, secure data
hosting, and flexible computing environments.

## Physical infrastructure

All CARC systems are housed in a dedicated 1,200 square-foot research data
center built to support high-performance and data-intensive computing.

The facility includes 270 kVA of UPS capacity and 990 tons of dedicated
cooling across three Liebert AC systems, providing resilience during
transient power events and allowing for controlled shutdown during extended
outages.

Systems are connected to campus through multiple 10 Gbps links, including a
dedicated 10 Gbps connection to UNM's Science DMZ research network. External
connectivity includes 100 Gbps connections to ESnet and the Western Regional
Network through the Albuquerque Gigapop.

Physical access to the machine room is restricted and managed through UNM's
Electronic Network Access Control system.

---8<--- https://unm-carc.github.io/docs/about/mission/

---
title: "Mission and vision"
description: "CARC's vision and mission: leading and growing the computational research community at UNM."
type: Reference
tags:
  - About
generated:
  by: "claude/fable-5"
  at: "2026-08-29T00:00:00Z"
sources:
  - id: carc-web
    resource: "https://carc.unm.edu/about-carc/mission--vision.html"
    title: "Mission & vision (carc.unm.edu)"
    author: "team:unm-carc"
---

# Mission and vision

## Vision

The Center for Advanced Research Computing (CARC) is an interdisciplinary
community at the University of New Mexico (UNM) that uses computational
resources to create new research insights.

## Mission

To lead and grow the computational research community at UNM.

To fulfill our mission, we will:

* Provide access to high-end computing resources and associated
  infrastructure;
* Offer specialized expertise and technical support;
* Coordinate and collaborate with other UNM programs that support the
  community; and
* Grow the collaborative user community through education, workshops, and
  outreach events.

CARC is the hub of computational research at UNM and one of the largest
computing centers in the State of New Mexico. Its services are available to
faculty, staff, and student researchers free of charge through support from
the UNM Office of the Vice President for Research.

---8<--- https://unm-carc.github.io/docs/about/partners/

---
title: "Partner cyberinfrastructure"
description: "National and regional platforms CARC users can reach: ACCESS-CI, Jetstream2, CyVerse, and MESA."
type: Reference
tags:
  - About
  - Partners
generated:
  by: "claude/fable-5"
  at: "2026-08-29T00:00:00Z"
sources:
  - id: carc-web
    resource: "https://carc.unm.edu/index.html"
    title: "CARC website (carc.unm.edu)"
    author: "team:unm-carc"
---

# Partner cyberinfrastructure

CARC connects UNM researchers to national cyberinfrastructure. When your
research outgrows campus resources — or needs capabilities we don't host —
these platforms are the next step, and CARC staff can help you get started on
any of them.

<div class="grid cards" markdown>

-   :material-transit-connection-variant:{ .lg .middle } __ACCESS-CI__

    ---

    The NSF's Advanced Cyberinfrastructure Coordination Ecosystem: one
    allocation system for national supercomputers, clouds, and storage.
    Start with an easy Explore allocation and scale up.

    [:octicons-arrow-right-24: access-ci.org](https://access-ci.org){ target=_blank }

-   :material-cloud:{ .lg .middle } __Jetstream2__

    ---

    NSF-funded, always-on cloud computing for research and education, led by
    Indiana University. Virtual machines, GPUs, Kubernetes, and science
    gateways through ACCESS allocations.

    [:octicons-arrow-right-24: docs.jetstream-cloud.org](https://docs.jetstream-cloud.org){ target=_blank }

-   :material-dna:{ .lg .middle } __CyVerse__

    ---

    Open science workspace for data-driven discovery: the Data Store (iRODS),
    Discovery Environment apps, and cloud-native analyses, operated by the
    University of Arizona and TACC.

    [:octicons-arrow-right-24: cyverse.org](https://cyverse.org){ target=_blank }

-   :material-robot:{ .lg .middle } __MESA__

    ---

    An NSF NAIRR research-infrastructure project building an open platform
    where scientific data are automatically described, organized, and
    connected by metadata-aware AI agents.

    [:octicons-arrow-right-24: idss-mesa.github.io](https://idss-mesa.github.io){ target=_blank }

</div>

## How CARC fits in

CARC operates UNM's on-premises HPC clusters and storage
([systems overview](../systems/overview.md)) and is expanding into
on-premises cloud (OpenStack), container orchestration (Kubernetes), and
data management (iRODS) services that interoperate with these national
platforms. If you are unsure which platform fits your project, come to
[office hours](../support/help.md) and we'll map it out together.

---8<--- https://unm-carc.github.io/docs/log/

# Documentation update log

## 2026-08-31

* **Update**: The [agent guide](about/ai-agents.md) now cross-links the [GPT 101 generative-AI workshop](https://tyson-swetnam.github.io/intro-gpt/){target=_blank} — an OKF v0.2 bundle with the same llms.txt conventions, maintained and taught by CARC.

## 2026-08-30

* **Update**: System-status links now point straight at the live monitors — the [cluster login & website status board](https://stats.uptimerobot.com/kqt0LYLwFd), [UNM IT alerts](https://italerts.unm.edu/), [perfSONAR network performance](http://perfsonar.alliance.unm.edu), the [Easley DNS check](https://dnschecker.org/#A/easley.alliance.unm.edu), and [XDMoD usage metrics](https://xdmod.alliance.unm.edu/) — instead of the intermediary carc.unm.edu downtime page. The landing button and troubleshooting steps use the cluster status board; the support card and systems overview list all five.

* **Update**: The header logo is now a Googie starburst — the same 12-ray construction as the homepage hero's atomic bursts (alternating ray lengths, tip dots, cycling colors), in the cream/turquoise/white subset that reads on the cherry header.

* **Update**: Completed the retirement of Wheeler, Taos, Gibbs, and Xena across the corpus. The "Legacy content" admonitions are gone, and every active page now reads against the current clusters: hostnames and prompts point at Hopper, retired-only sections were removed (the Wheeler/PBS Orca variant, the Xena AlphaFold script, Xena-specific partition flags), and the historical Xena and Wheeler queue tables moved from [resource limits](systems/resource-limits.md) into the [legacy cluster reference](systems/cluster-specifications.md). The migration pipeline now enforces this: it fails if a retired system name appears outside the sanctioned legacy pages (provenance frontmatter and the changelog stay truthful).

* **Update**: Converted the remaining PBS-era material on active pages to Slurm ([storage](systems/storage.md) example script and wording, [R package installs](software/r-packages.md) interactive-session request) and fenced all file paths on the storage page. Fixed the [SSH config example](getting-started/ssh-keys.md) (now a single well-formed block covering Hopper and Easley).

* **Update**: Footer social links: removed the X/Twitter icon (account no longer exists) and pointed YouTube at the [main UNM CARC channel](https://www.youtube.com/@UNMCARC).

## 2026-08-29

* **Update**: Made the deployed site directly consumable by AI agents: every page's Markdown source (OKF frontmatter intact) is now served at its URL plus `index.md`; rendered pages advertise it via `link rel=alternate` and `okf:*` meta tags (type, status, trust tier, generated-at); `robots.txt` points crawlers at `llms.txt`, the full corpus, and the mirror convention (`scripts/postbuild_agent_surface.py`, wired into CI). Added the [For AI agents](about/ai-agents.md) guide and a repository `AGENTS.md`/`CLAUDE.md` for coding harnesses.

* **Update**: Embedded CARC YouTube recordings across the site: the [Video tutorials](training/videos.md) page now carries the full QuickBytes playlist, CARC Annual Meeting 2025 talks, and research presentations from the UNMCARC channel; ten guide pages (logging in, Slurm intro, storage, transfers, modules, conda, X11, GNU Parallel, SimCov, parallel R) embed their matching walkthrough via the migration pipeline.
* **Creation**: Rebuilt the [Workshops and slides](training/workshops.md) page (previously a stub) as a curated catalog of 33 slide decks: the Introduction to CARC series, domain-focused workshops, course guest lectures, and legacy material.

* **Update**: The Xena cluster has been retired. Removed Xena from the [Systems overview](systems/overview.md), [Facilities description](about/facilities.md), FAQ, and landing page; added it to the retired-systems list in [Cluster specifications](systems/cluster-specifications.md); marked Xena-specific GPU guides (PyTorch, MATLAB GPU/deep learning, deep-learning packages) as `status: draft` with legacy notices pending review against Hopper and Easley GPUs.
* **Update**: Annotated all code across the corpus: tab-indented QuickBytes code now renders as language-fenced blocks with syntax highlighting; restructured [Installing deep learning packages](software/deep-learning-packages.md) (now curated in-repo); annotated inline code references in [Parallel R with the future package](software/parallel-r-future.md).

* **Creation**: Added the Interactive computing section ([Open OnDemand](interactive/open-ondemand.md), [JupyterHub](interactive/jupyterhub.md)) and the FAQ section ([General FAQ](faq/general.md), [Troubleshooting](faq/troubleshooting.md)).
* **Creation**: Added [Contributing to these docs](about/contributing.md) — the OKF frontmatter contract, verification workflow, and local build instructions for CARC staff.
* **Update**: Added machine-readable `llms.txt` and `llms-full.txt` indexes generated from OKF frontmatter (`scripts/gen_llms_txt.py`, enforced in CI); moved the page table of contents into the left sidebar.

* **Initialization**: Created this documentation bundle with [Zensical](https://zensical.org){target=_blank}, structured as an Open Knowledge Format (OKF v0.2) knowledge bundle.
* **Migration**: Migrated 56 tutorials and guides from [UNM-CARC/QuickBytes](https://github.com/UNM-CARC/QuickBytes){target=_blank} and [UNM-CARC/webinfo](https://github.com/UNM-CARC/webinfo){target=_blank} with provenance frontmatter (`generated`, `sources`, per-file `last_modified` from git history). All migrated pages are unverified pending CARC staff review.
* **Creation**: Wrote the [Getting started overview](getting-started/overview.md), [Good Neighbor Use Policy](getting-started/good-neighbor-policy.md), [Systems overview](systems/overview.md), [Video tutorials](training/videos.md), [Getting help](support/help.md), [Acknowledging CARC](support/acknowledging-carc.md), [Mission and vision](about/mission.md), [Facilities description](about/facilities.md), and [Partner cyberinfrastructure](about/partners.md) pages from carc.unm.edu content.
* **Deprecation**: Marked [Cluster specifications](systems/cluster-specifications.md) (retired Wheeler, Taos, and Gibbs systems) and [R batch jobs with PBS](software/r-pbs-jobs.md) as deprecated; both are kept for history and links.