--- title: "Data transfer using GoCommands" description: "Download and upload files and folders between a local system and the Data Store with GoCommands, including thread-count tuning." type: Guide tags: - GoCommands - Data Transfer generated: by: "claude/opus-5" at: "2026-09-11T00:00:00Z" sources: - id: cyverse-learning-materials resource: "https://github.com/CyVerse-learning-materials/learning-materials-home/blob/b7392d21be4fd6a67d051847f98c81a18e058a12/docs/ds/gocommands/data_transfer.md" title: "CyVerse Learning Materials: docs/ds/gocommands/data_transfer.md" author: "team:cyverse" last_modified: "2026-03-19T14:18:43-07:00" --- # Data transfer using GoCommands GoCommands provides a range of commands designed to efficiently and conveniently transfer large datasets between your local machine and the Data Store. --- ## :material-package-variant-closed: Download Data from the Data Store 1. **Download a collection to a specific local path:** ```sh gocmd get --progress -f -k /iplant/home/myUser/mydir /local/dir/ ``` This command downloads the collection `/iplant/home/myUser/mydir` and its contents to `/local/dir/`. A new directory named `mydir` will be created under `/local/dir/`, resulting in `/local/dir/mydir` containing all the downloaded files and subdirectories. - The `--progress` flag shows progress bars, providing visual feedback during file transfers. - The `-f` flag forces overwriting of existing local files without confirmation. - The `-k` flag ensures file integrity by verifying checksums after the download. 2. **Sync a collection in the Data Store with a local directory:** ```sh gocmd get --progress -f -k --diff --delete /iplant/home/myUser/mydir /local/dir/ ``` This command downloads the collection `/iplant/home/myUser/mydir` and its contents to `/local/dir/`, syncing any new updates made to the original data. - The `--diff` flag transfers only new or modified files, comparing sizes and checksums. - The `--delete` flag removes files from the destination that don't exist in the source. This command is equivalent to: ```sh gocmd sync --progress -k --delete i:/iplant/home/myUser/mydir /local/dir/ ``` 3. **Download a collection using multiple parallel transfers:** ```sh gocmd get --progress -f -k --thread_num 10 /iplant/home/myUser/mydir /local/dir/ ``` This command downloads the collection `/iplant/home/myUser/mydir` and its contents to `/local/dir/` using 10 parallel transfer threads. Utilizing more transfer threads can maximize I/O and network bandwidth, often speeding up the transfer process significantly. - The `--thread_num` flag sets the maximum number of threads to use for the transfer. !!! warning "Thread Count Considerations" 1. Higher thread counts require more CPU and memory. Excessive threads may overload your system, causing performance issues. For example, Discovery Environment (DE) apps limit transfer threads to 5 due to RAM constraints. 2. The Data Store limits concurrent connections, potentially restricting high thread counts. !!! tip "Using GoCommands on the University of Arizona Campus Network?" When using GoCommands on the UA Campus network, include the `--webdav` flag for stable large file transfers. --- ## :material-package-variant-closed: Upload Data to the Data Store 1. **Upload a local directory to a specific path in the Data Store:** ```sh gocmd put --progress -f -k /local/dir/ /iplant/home/myUser/mydir/ ``` This command uploads the local directory `/local/dir/` and its contents to the Data Store at `/iplant/home/myUser/mydir/`. A new directory named `dir` will be created under `/iplant/home/myUser/mydir/`, resulting in `/iplant/home/myUser/mydir/dir` containing all the uploaded files and subdirectories. - The `--progress` flag shows progress bars during file transfers. - The `-f` flag forces overwriting of existing files in the Data Store without confirmation. - The `-k` flag ensures file integrity by verifying checksums after upload. 2. **Sync a local directory with a collection in the Data Store:** ```sh gocmd put --progress -f -k --diff --delete /local/dir/ /iplant/home/myUser/mydir/ ``` This command uploads the local directory `/local/dir/` to `/iplant/home/myUser/mydir/` in the Data Store, syncing any new or updated files. It also removes extra files in the destination collection that do not exist locally. - The `--diff` flag transfers only new or modified files, comparing sizes and checksums. - The `--delete` flag removes files from the destination that don't exist in the source. This command is equivalent to: ```sh gocmd sync --progress -k --delete /local/dir/ i:/iplant/home/myUser/mydir/ ``` 3. **Upload a local directory using multiple parallel transfers:** ```sh gocmd put --progress -f -k --thread_num 10 /local/dir/ /iplant/home/myUser/mydir/ ``` This command uploads the local directory `/local/dir/` and its contents to `/iplant/home/myUser/mydir/` in the Data Store using 10 parallel transfer threads. This can significantly speed up the upload process by maximizing I/O and network bandwidth. - The `--thread_num` flag sets the maximum number of threads to use for the transfer. !!! warning "Thread Count Considerations" 1. Higher thread counts require more CPU and memory. Excessive threads may overload your system, causing performance issues. For example, Discovery Environment (DE) apps limit transfer threads to 5 due to RAM constraints. 2. The Data Store limits concurrent connections, potentially restricting high thread counts. !!! tip "Using GoCommands on the University of Arizona Campus Network?" When using GoCommands on the UA Campus network, include the `--webdav` flag for stable large file transfers. 4. **Upload a local directory containing many small files:** ```sh gocmd bput --progress -k --thread_num 10 /local/dir/ /iplant/home/myUser/mydir/ ``` This command uploads the local directory `/local/dir/` containing numerous small files to `/iplant/home/myUser/mydir/` in the Data Store. Using bundle transfer and parallel transfer can significantly speed up the transfer of many small files. This command is equivalent to: ```sh gocmd sync --bulk_upload --progress -k /local/dir/ i:/iplant/home/myUser/mydir/ ``` !!! tip "UNM CARC users" GoCommands is a single binary that installs without administrator rights, so you can install it in your CARC home directory and move data directly between CARC storage and the Data Store. CyVerse is one of [CARC's partner platforms](https://carc.unm.edu/docs/about/partners/){target=_blank}; for moving files onto and off CARC systems in general, see [Transferring data](https://carc.unm.edu/docs/getting-started/transferring-data/){target=_blank} in the CARC documentation.

Adapted from [CyVerse Learning Materials](https://github.com/CyVerse-learning-materials/learning-materials-home/blob/b7392d21be4fd6a67d051847f98c81a18e058a12/docs/ds/gocommands/data_transfer.md){target=_blank} (last source update 2026-03-19), CC BY 4.0. Spotted a problem? [Open an issue](https://github.com/UNM-CARC/cyverse/issues){target=_blank}.