Adapted from [CyVerse FOSS](https://github.com/CyVerse-learning-materials/foss/blob/86d4cc384976a59bd19cce2a6fdcf994b19051bd/docs/code_of_conduct.md){target=_blank} (last source update 2022-07-13), CC BY 4.0. Spotted a problem? [Open an issue](https://github.com/UNM-CARC/foss/issues){target=_blank}.
---8<--- https://unm-carc.github.io/foss/course/glossary/ --- title: "Glossary & Acronyms" description: "Definitions of the computing, open-science, and cyberinfrastructure terms and acronyms used throughout FOSS." type: Reference tags: - Glossary - Reference generated: by: "claude/opus-5" at: "2026-09-11T00:00:00Z" sources: - id: cyverse-foss resource: "https://github.com/CyVerse-learning-materials/foss/blob/86d4cc384976a59bd19cce2a6fdcf994b19051bd/docs/glossary.md" title: "CyVerse FOSS: docs/glossary.md" author: "team:cyverse" last_modified: "2022-07-15T16:10:54+07:00" --- # Glossary & Acronyms **A** - **action:** automate a workflow in the context of CI/CD, see [GitHub Actions](https://github.com/features/actions){target=_blank} - **agile:** [development methodology](https://en.wikipedia.org/wiki/Agile_software_development){target=_blank} for organizing a team to complete tasks organized over short periods called 'sprints' - **allocation:** portion of a resource assigned to a particular recipient, typical unit is a core or node hour - **Anaconda:** open source data science platform. [Anaconda.com](https://www.anaconda.com/){target=_blank} - **application:** also called an 'app', a software designed to help the user to perform specific task - **awesome:** a curated set of lists that provide insight into [awesome software projects on GitHub](https://github.com/topics/awesome-list){target=_blank} - **AVU:** Attribute-Value-Unit a components for [iRODS metadata](https://docs.irods.org/4.1.9/icommands/metadata/). **B** - **beta:** a software version which is not yet ready for publication but is being tested - **bash:** Bash is the GNU Project's shell, the [Bourne-Again Shell](https://www.gnu.org/software/bash/) - **biocontainer:** a community-driven project that provides the infrastructure and basic guidelines to create, manage and distribute bioinformatics packages (e.g conda) and containers (e.g docker, singularity) - **bioconda:** a channel for the conda package manager specializing in bioinformatics software **C** - **CLI:** the UNIX shell [command line interface](https://en.wikipedia.org/wiki/Command-line_interface){target=_blank}, most typically BASH - **command:** a set of instructions sent to the computer, typically in a typed interface - **conda:** an installation type of the Anaconda data science platform. Command line application for managing packages and environments - **container:** virtualization of an operating system run within an isolated user space - **Continuous Integration:** (CI) is testing automation to check that the application is not broken whenever new commits are integrated into the main branch - **Continuous Delivery:** (CD) is an extension of 'continuous integration' to make sure that you can release new changes in a sustainable way - **Continuous Deployment:** a step further than 'continuous delivery', every change that passes all stages of your production pipeline is released - **Continuous Development:** a process for iterative software development and is an umbrella over several other processes including 'continuous integration', 'continuous testing', 'continuous delivery' and 'continuous deployment' - **Continuous Testing:** a process of testing and automating software development. - **CRAN:** The [Comprehensive R Archive Network](https://cran.r-project.org/) - **CyVerse tool:** Software program that is integrated into the back end of the DE for use in DE apps - **CyVerse app:** graphic interface of a tool made available for use in the DE **D** - **Debian:** [a free OS](https://www.debian.org/){target=_blank}, base of other Linux distributions such as Ubuntu - **Development:** the environment on your computer where you write code - **DevOps** Software *Dev*elopment and information techology *Op*erations techniques for shortening the time to change software in relation to CI/CD - **Discovery Environment (DE):** a data science workbench for running executable, interactive, and high throughput applications in [CyVerse DE](https://de.cyverse.org){target=_blank} - **distribution:** abbreviated as 'distro', an operating system made from a software collection based upon the Linux kernel - **Docker:** [Docker](https://www.docker.com/){target=_blank} is an open source software platform to create, deploy and manage virtualized application containers on a common operating system (OS), with an ecosystem of allied tools. A program that runs and handles life-cycle of containers and images - **DockerHub:** an official registry of docker containers, operated by Docker. [DockerHub](https://hub.docker.com/){target=_blank} - **DOI:** a digital object identifier. A persistant identifier number, managed by the [doi.org](https://www.doi.org/){target=_blank} - **Dockerfile:** a text document that contains all the commands you would normally execute manually in order to build a Docker image. Docker can build images automatically by reading the instructions from a Dockerfile **E** - **environment:** software that includes operating system, database system, specific tools for analysis - **entrypoint:** In a Dockerfile, an ENTRYPOINT is an optional definition for the first part of the command to be run **F** - **FOSS:** (1) [Free and Open Source Software](https://en.wikipedia.org/wiki/Free_and_open-source_software){target=_blank}, (2) [Foundational Open Science Skills](https://cyverse.org/foss){target=_blank} - this class! - **function:** a named section of a program that performs a specific task **G** - **git:** a version control system software - **gitter:** a Github based messaging service that uses markdown [gitter.im](https://gitter.im){target=_blank} - **GitHub:** a website for hosting `git` repositories - owned by Microsoft [GitHub](https://github.com){target=_blank} - **GitLab:** a website for hosting `git` repositories [GitLab](https://gitlab.com){target=_blank} - **GitOps:** using `git` framework as a means of deploying infrastructure on cloud using Kubernetes - **GPU:** graphic processing unit - **GUI:** graphical user interface **H** - **hack:** a quick job that produces what is needed, but not well - **HPC:** High Performance Computer, for large syncronous computation - **HTC:** High Throughput Computer, for many parallel tasks **I** - **IaaS:** [Infrastructure as a Service](https://en.wikipedia.org/wiki/Infrastructure_as_a_service){target=_blank}. online services that provide APIs - **iCommands:** [command line application](https://docs.irods.org/master/icommands/user/){target=_blank} for accessing iRODS Data Store - **IDE:** integrated development environment, typically a graphical interface for working with code language or packages - **instance:** a single virtul machine - **image:** self-contained, read-only 'snapshot' of your applications and packages, with all their dependencies - **iRODS:** an open source integrated Rule-Oriented Data Management System, [iRODS.org](https://irods.org/){target=_blank} **J** - **Java:** programming language, class-based, object-oriented - **JavaScript:** programming language - **JSON:** Java Script Object Notation, data interchange format that uses human-readable text - **Jupyter(Hub,Lab,Notebooks):** an IDE, originally the iPythonNotebook, operates in the browser [Project Jupyter](https://jupyter.org/) **K** - **kernel:** central component of most operating systems (OS) - **Kubernetes:** an open source container orchestration platform created by Google [Kubernetes](https://kubernetes.io/){target=_blank} is often referred to as `K8s` **L** - **lib:** a UNIX library - **linux:** open source Unix-like operating system **M** - **makefile:** a file containing a set of directives used by a [make build automation tool](https://www.gnu.org/software/make/) - **markdown:** a lightweight markup language with plain text formatting syntax - **metadata::** data about data, useful for searching and querying - **multi-thread:** a process which runs on more than one CPU or GPU core at the same time - **master node:** responsible for deciding what runs on all of the cluster's nodes. Can include scheduling workloads, like containerized applications, and managing the workloads' lifecycle, scaling, and upgrades. The master also manages network and storage resources for those workloads - **Mac OS X:** Apple's popular desktop OS **N** - **node:** a computer, typically 1 or 2 core (with many threads) server in a cloud or HPC center **O** - **ontology:** formal naming and structural hierarchy used to describe data, also called a [knowledge graph](https://en.wikipedia.org/wiki/Ontology_(information_science)) - **organization:** a group, in the context of GitHub a place where developers contribute code to repositories - **Operating System (OS):** software that manages computer hardware, software resources, and provides common services for computer programs - **Open Science Grid (OSG):** national, distributed computing partnership for data-intensive research [opensciencegrid.org](https://opensciencegrid.org/){target=_blank} - **ORCID:** Open Researcher and Contributor ID ([ORCiD](https://orcid.org/){target=_blank}), a persistent digital identifier that distinguishes you from every other researcher **P** - **PaaS:** [Platform as Service](https://en.wikipedia.org/wiki/Platform_as_a_service){target=_blank} run and manage applications in cloud without complexity of developing it yourself - **package:** an app designed for a particular langauge - **package manager:** a collection of software tools that automates the process of installing, upgrading, configuring, and removing computer programs for a computer's operating system in a consistent manner - **Production:** environment where users access the final code after all of the updates and testing - **Python:** interpreted, high-level, general-purpose programming language [Python.org](https://www.python.org/){target=_blank} **Q** - **QUAY.io:** private Docker registry [QUAY.io](https://quay.io){target=_blank} **R** - **R:** data science programming language [R Project](https://cran.r-project.org/){target=_blank} - **recipe file:** a file with installation scripts used for building software such as containers, e.g. Dockerfile - **registry:** a storage and content delivery system, such as that used by Docker - **remote desktop:** a VM with a graphic user interface accessed via a browser - **repo(sitory):** a directory structure for hosting code and data - **RST:** ReStructuredText, a markdown type file - **ReadTheDocs:** a web service for rendering documentation [readthedocs.org](https://readthedocs.org){target=_blank} and [readthedocs.com](https://readthedocs.com/){target=_blank} - **root:** the administrative user on a linux kernel - use your powers wisely **S** - **SaaS:** [Software as a Service](https://en.wikipedia.org/wiki/Software_as_a_service){target=_blank} web based platform for using software - **schema:** a metadata standard for labeling, tagging or coding for recording & cataloging information or structuring descriptive records. see [schema.org](https://schema.org/){target=_blank} - **scrum:** daily set of tasks and evalautions as part of a sprint. - **shell:** is a command line interface program that runs other programs (may be complex, technical programs or very simple programs such as making a directory). These simple, stand-alone programs are called commands - **Singularity:** a container software, used widely on HPC, created by [SyLabs](https://sylabs.io/){target=_blank}; the open-source project continues as [Apptainer](https://apptainer.org/){target=_blank}, which keeps `singularity` as a command alias - **SLACK:** Searchable Log of All Conversation and Knowledge, a team communication tool [slack.com](https://slack.com/){target=_blank} - **sprint:** set period of time during which specific work has to be completed and made ready for review - **Singularity def file:** (definition file) recipe for building a Singualrity container - **Stage:** environment that is as similar to the production environment as can be for final testing **T** - **tar:** software utility for collecting many files into one archive file, often referred to as a tarball - **tensor:** algebraic object that describes a linear mapping from one set of algebraic objects to another - **terminal:** a windowed emulator for directly enterinc commands to a computer - **thread:** a CPU process or a series of linked messages in a discussion board - **tool:** In the context of CyVerse Discovery Environment, a Docker Container - **TPU:** tensor processing unit - **Travis:** [Travis-CI](https://travis-ci.org/){target=_blank}, a continuous integration software **U** - **Ubuntu:** most popular [Linux OS distribution](https://ubuntu.com/), based on Debian - **UNIX:** operating system - **user:** the profile under which applications are started and run, `root` is the most powerful system administrator **V** - **VICE:** [Visual Interactive Computing Environment](https://unm-carc.github.io/cyverse/discovery-environment/vice/overview/) - Cyverse Data Science Workbench - **virtual machine:** is a software computer that, like a physical computer, runs an operating system and applications **W** - **waterfall:** software development broken into linear sequential phases, similar to a Gantt chart - **webGL:** JavaScript API for rendering interactive 2D and 3D graphics within any compatible web browser without the use of plug-ins - **Windows:** Microsoft's most popular desktop OS - **workspace:** (vs. repo) - **worker node:** A cluster typically has one or more nodes, which are the worker machines that run your containerized applications and other workloads. Each node is managed from the master, which receives updates on each node's self-reported status. **X** - **XML:** Extensible Markup Language, data interchange format that uses human-readable text **Y** - **YAML:** YAML Ain't Markup Language, data interchange format that uses human-readable text **Z** - **ZenHub:** team collaboration solution built directly into GitHub that uses kanban style boards - **Zenodo:** general-purpose open-access repository developed under the European OpenAIRE program and operated by CERN - **zip:** a compressed file format - **zsh:** [Z-Shell](https://www.zsh.org/){target=_blank}, now the default shell on new Mac OS XAdapted from [CyVerse FOSS](https://github.com/CyVerse-learning-materials/foss/blob/86d4cc384976a59bd19cce2a6fdcf994b19051bd/docs/glossary.md){target=_blank} (last source update 2022-07-15), CC BY 4.0. Spotted a problem? [Open an issue](https://github.com/UNM-CARC/foss/issues){target=_blank}.
---8<--- https://unm-carc.github.io/foss/lessons/01-open-science/ --- title: "Introduction to Open Science" description: "What open science is, its six pillars (open access, data, educational resources, methodology, peer review, and source software), and why researchers practice it." type: Lesson tags: - Open Science - FAIR - Open Access - Peer Review generated: by: "claude/opus-5" at: "2026-09-11T00:00:00Z" sources: - id: cyverse-foss resource: "https://github.com/CyVerse-learning-materials/foss/blob/86d4cc384976a59bd19cce2a6fdcf994b19051bd/docs/01_intro_open_sci.md" title: "CyVerse FOSS: docs/01_intro_open_sci.md" author: "team:cyverse" last_modified: "2025-09-01T12:42:44-07:00" --- # Introduction to Open Science !!! success "Learning Objectives" After this lesson, you should be able to: - Explain what Open Science is - Explain the components of Open Science - Describe the behaviors of Open Science - Explain why Open Science matters in education, research, and society - Understand the advantages and the challenges to Open Science## What is Open Science? _"Open Science is transparent and accessible knowledge that is shared and developed through collaborative networks"_ [-Vincente-Saez & Martinez-Fuentes 2018](https://doi.org/10.1016/j.jbusres.2017.12.043){target=_blank}
_"Open Science is a collaborative and transparent approach to scientific research that emphasizes the accessibility, sharing, and reproducibility of data, methodologies, and findings to foster innovation and inclusivity"_ -ChatGPT
_"A series of reforms that interrogate every step in the research life cycle to make it more efficient, powerful and accountable in our emerging digital society"._ -Jeffrey Gillan
!!! quote "Other Definitions" "Open Science is defined as an inclusive construct that combines various movements and practices aiming to make multilingual scientific knowledge openly available, accessible and reusable for everyone, to increase scientific collaborations and sharing of information for the benefits of science and society, and to open the processes of scientific knowledge creation, evaluation and communication to societal actors beyond the traditional scientific community." - [UNESCO Definition](https://www.unesco.org/en/natural-sciences/open-science){target=_blank} - [UNESCO's Recommendation on Open Science](https://unesdoc.unesco.org/ark:/48223/pf0000379949.locale=en){target=_blank} "Open Science is the movement to make scientific research (including publications, data, physical samples, and software) and its dissemination accessible to all levels of society, amateur or professional..." [ :material-wikipedia: Wikipedia definition](https://en.wikipedia.org/wiki/Open_science){target=_blank}
## Foundational Open Science Skills
_2. Mastery of digital tools to create reproducible science that others can build upon_
_3. Understanding the push towards increased transparency and accountability for those practicing science (ie., compliance)_
!!! info "Governamental footprints: 2023 vs 2025" 2023 was deemed the Year of Open Science, but recent changes at the administration level are seeing the approach to science being reshaped. ??? tip "2023: the Year of Open Science" The White House, joined by 10 federal agencies, and a coalition of more than 85 universities, declared 2023 the Year of Open Science as a way to bring awareness to the benefits of Open Science and to steer the scientitic community towards its adoption. In 2024, [the first year of Open Science was celebrated in an effort to continue the promotion of the Open Science movement](https://bidenwhitehouse.archives.gov/ostp/news-updates/2024/01/31/fact-sheet-biden-harris-administration-marks-the-anniversary-of-ostps-year-of-open-science/){target=_blank}. The goals and activities around the White House's promotion of Open Science were centered around five key themes: - Strengthening Open Science Policies - Investing in Open Science Infrastructures - Supporting the Research Community in Building Open Science Skills - Engaging Communities to Broaden Participation in Open Science - Promoting Incentives for Open Research Practices Grant funding organizations such as the NIH, NASA and the US Department of Agriculture were prompt at sustaining the White House' vision by releasing Open Science policy guidelines across the board.
Open Access Publications
:material-pillar:
Open Data
:material-pillar:
Open Educational Resources
:material-pillar:
Open Methodology
:material-pillar:
Open Peer Review
:material-pillar:
Open Source Software
:material-pillar:
:material-pillar: Open Access Publications
**:material-pillar: Open Data
** !!! quote "Definitions" “Open data and content can be freely used, modified, and shared by anyone for any purpose” - [The Open Definition](https://opendefinition.org/){target=_blank} "Open data is data that can be freely used, re-used and redistributed by anyone - subject only, at most, to the requirement to attribute and sharealike." - [Open Data Handbook](https://opendatahandbook.org/guide/en/what-is-open-data/){target=_blank} [:material-wikipedia: Wikipedia definition](https://en.wikipedia.org/wiki/Open_data){target=_blank} Data are the foundation for any scientific endeavor. A lot of thought needs to go into how to best collect, store, analyze, curate, share, and archive data.:material-pillar: Open Educational Resources
**:material-pillar: Open Methodology
** !!! quote "Definitions" "An open methodology is simply one which has been described in sufficient detail to allow other researchers to repeat the work and apply it elsewhere." - [Watson (2015)](https://doi.org/10.1186/s13059-015-0669-2){target=_blank} "Open Methodology refers to opening up methods that are used by researchers to achieve scientific results and making them publicly available." - [Open Science Network Austria](https://www.oana.at/en/about-open-science){target=_blank} #### Sharing Research Computer Code Scientists around the globe are creating computer code for scientific analysis. These are valuable contributions that need to be shared! Platforms like [GitHub](https://github.com/search?q=open+science){target=_blank} and [GitLab](https://gitlab.com/explore/projects/topics/Open%20Science){target=_blank} are ideal for collaboratively developing code and sharing with the open internet. In FOSS, we will show you how to use Github for sharing code, [documentation](https://unm-carc.github.io/foss/lessons/02-data-management/#project-documentation), hosting websites, and [software version control](https://unm-carc.github.io/foss/lessons/05-version-control/).
:material-pillar: Open Peer Review
** !!! quote "Definitions" Open peer review is an umbrella term for a number of overlapping ways that peer review models can be adapted in line with the aims of Open Science, including making reviewer and author identities open, publishing review reports and enabling greater participation in the peer review process. - [-Ross-Hellauer et al. (2017)](https://doi.org/10.12688%2Ff1000research.11369.2){target=_blank} - [:material-wikipedia: Wikipedia's definition](https://en.wikipedia.org/wiki/Open_peer_review){target=_blank} #### Traditional Closed Peer-Review System:material-pillar: Open Source Software
** [{width=240 target=_blank }](https://opensource.org/){target=_blank} !!! quote "Definitions" "Open source software is code that is designed to be publicly accessible—anyone can see, modify, and distribute the code as they see fit. Open source software is developed in a decentralized and collaborative way, relying on peer review and community production." - [:material-redhat: Red Hat](https://www.redhat.com/en/topics/open-source/what-is-open-source){target=_blank} [:material-wikipedia: Wikipedia definition](https://en.wikipedia.org/wiki/Open-source_software){target=_blank} Research science (and also many companies) rely on open source software to operate !!! tip "Open Source Software" - Linux operating system and shell - Python - R - git - Conda - Docker - Cyverse - Pytorch - [Tyson's Awesome List](https://tyson-swetnam.github.io/awesome-open-science/software/){target=_blank} When you create a new software, library, or package, you become its parent and guardian.
Adapted from [CyVerse FOSS](https://github.com/CyVerse-learning-materials/foss/blob/86d4cc384976a59bd19cce2a6fdcf994b19051bd/docs/01_intro_open_sci.md){target=_blank} (last source update 2025-09-01), CC BY 4.0. Spotted a problem? [Open an issue](https://github.com/UNM-CARC/foss/issues){target=_blank}.
---8<--- https://unm-carc.github.io/foss/lessons/02-data-management/ --- title: "Data Management and Documentation" description: "The data life cycle, FAIR and CARE principles, data management plans, licensing, and tools for documenting research projects." type: Lesson tags: - Data Management - FAIR - CARE - Documentation - Licensing generated: by: "claude/opus-5" at: "2026-09-11T00:00:00Z" sources: - id: cyverse-foss resource: "https://github.com/CyVerse-learning-materials/foss/blob/86d4cc384976a59bd19cce2a6fdcf994b19051bd/docs/02_managing_data.md" title: "CyVerse FOSS: docs/02_managing_data.md" author: "team:cyverse" last_modified: "2025-09-08T16:33:24-07:00" --- # Data Management and Documentation ??? info "Relating to Open Science" As you may recall, Open Science is an ideology built around the goal of making the science you work on clear and accessible to everyone. Data management and documentation fit well within this framework, as they emphasize that the first person who needs to understand the science is the one doing it — **you**. !!! success "Learning Objectives" After this lesson, you should be able to: - Recognize data as the foundation of open science and be able to describe the "life cycle of data" - Use self-assessments to evaluate your current data management practices - Cite tools and resources to improve your data management practices - Know the biggest challenge to effective data management ??? question "How would you answer?" - If you give your data to a colleague who has not been involved with your project, would they be able to make sense of it? Would they be able to use it properly? - If you come back to your own data in five years, will you be able to make sense of it? Will you be able to use it properly? - When you are ready to publish a paper, is it easy to find all the correct versions of all the data you used and present them in a comprehensible manner? ## Why should you care about data management? !!! danger "The biggest challenge to data management is making it an afterthought. " Poor data management doesn't have an upfront cost. **You can do substantial work before realizing you are in trouble.** **The solution?** Make data management the first thing you consider when starting a research project. Well-managed Data Sets: - Can make life much easier for you and your collaborators - Benefit the scientific research community by allowing others to reuse your data - Are becoming required by most funders and many journals, which are requesting a submission of a Data Management Plan (DMP) with the initial submission of your proposal. !!! info "The NSF is stepping in, getting stricter about data" - Recent [Dear Colleague letter](https://www.nsf.gov/pubs/2019/nsf19069/nsf19069.jsp){target=_blank} from NSF's points out that: - Open science promotes broader access to research data, enhancing public benefits and replicability. - NSF requires (DMPs) in proposals, encouraging use of persistent IDs and machine-readable DMPs. - NSF [proposal preparation guidelines](https://new.nsf.gov/policies/pappg/23-1/ch-2-proposal-preparation#2D2i-ii){target=_blank} now require at least the following: - Proposals must include a 2-page DMP outlining data types, formats, sharing, and archiving. - The DMP must address privacy, intellectual property, and reuse policies, and collaborative projects should submit one unified DMP. - A DMP stating no detailed plan is needed is allowed with justification, and the DMP will be reviewed as part of the proposal's merit. --- ## What Classifies as Data? Different types of data require different management practices. Here are some examples of what we can call Data (Adapted from DMPTool [Data management general guidance](https://dmptool.org/general_guidance#types-of-data){target=_blank}). **Data Types**: - **Text**: field or laboratory notes, survey responses - **Numeric**: tables, counts, measurements - **Audiovisual**: images, sound recordings, video - **Models, computer code** - **Discipline-specific**: FASTA in biology, FITS in astronomy, CIF in chemistry - **Instrument-specific**: equipment outputs **Data Sources**: *Observational* - Captured in real-time, typically outside the lab - Usually irreplaceable and therefore the most important to safeguard - Examples: Sensor readings, telemetry, survey results, images *Experimental* - Typically generated in the lab or under controlled conditions - Often reproducible, but can be expensive or time-consuming - Examples: gene sequences, chromatograms, magnetic field readings *Simulation* - Machine generated from test models - Likely to be reproducible if the model and inputs are preserved - Examples: climate models, economic models *Derived / Compiled* - Generated from existing datasets - Reproducible, but can be very expensive and time-consuming - Examples: text and data mining, compiled database, 3D models --- ## The Data Life Cycle **Data management** is the set of practices that allow researchers to effectively and efficiently handle data throughout the data life cycle. Although typically shown as a circle (below) the actually life cycle of any data item may follow a different path, with branches and internal loops. Being aware of your data's future helps you plan how to best manage them.Read it.
!!! tip "Why Principles?" FAIR is a collection of principles. Ultimately, different communities within different scientific disciplines must work to interpret and implement these principles. Because technologies change quickly, focusing on the desired end result allows FAIR to be applied to a variety of situations now and in the foreseeable future. **Findable** - F1. (meta)data are assigned a globally unique and persistent identifier - F2. data are described with rich metadata (defined by R1 below) - F3. metadata clearly and explicitly include the identifier of the data it describes - F4. (meta)data are registered or indexed in a searchable resource **Accessible** - A1. (meta)data are retrievable by their identifier using a standardized communications protocol - A1.1 the protocol is open, free, and universally implementable - A1.2 the protocol allows for an authentication and authorization procedure, where necessary - A2. metadata are accessible, even when the data are no longer available **Interoperable** - I1. (meta)data use a formal, accessible, shared, and broadly applicable language for knowledge representation. - I2. (meta)data use vocabularies that follow FAIR principles - I3. (meta)data include qualified references to other (meta)data **Reusable** - R1. meta(data) are richly described with a plurality of accurate and relevant attributes - R1.1. (meta)data are released with a clear and accessible data usage license - R1.2. (meta)data are associated with detailed provenance - R1.3. (meta)data meet domain-relevant community standard !!! tip "Open vs. Public vs. FAIR" Open: “Open data and content can be freely used, modified, and shared by anyone for any purpose”. FAIR does **NOT** demand that data be open: See one definition of open: http://opendefinition.org/. ### CARE PrinciplesNihil de nobis, sine nobis.
!!! question "Who owns the açaí?" The [CARE Principles](https://www.gida-global.org/care){target=_blank} for Indigenous Data Governance were drafted at the International Data Week and Research Data Alliance Plenary co-hosted event "Indigenous Data Sovereignty Principles for the Governance of Indigenous Data Workshop," 8 November 2018, Gaborone, Botswana. **Collective Benefit** - C1. For inclusive development and innovation - C2. For improved governance and citizen engagement - C3. For equitable outcomes **Authority to Control** - A1. Recognizing rights and interests - A2. Data for governance - A3. Governance of data **Responsibility** - R1. For positive relationships - R2. For expanding capability and capacity - R3. For Indigenous languages and worldviews **Ethics** - E1. For minimizing harm and maximizing benefit - E2. For justice - E3. For future use !!! tip "Connecting FOSS and CARE: [Lydia Jennings](https://nativesoilnerd.com/){target=_blank}" Dr. Lydia Jennings was a Data Science Fellow at the University of Arizona, who attended FOSS in Fall of 2022. Lydia graduated from the University of Arizona's Department of Evironemtal Sciences, and has published a paper on the application of the CARE principles to ecology and biodiversity research. Go Lydia! [Appying the 'CARE Principles for Indigenous Data Governance' to ecology and biodiversity](https://www.nature.com/articles/s41559-023-02161-2){target=_blank}, *Nature Ecology & Evolution*, 2023. ### How to get to FAIR? This is a question that only you can answer, that is because it depends on (among other things) 1. Your scientific discipline: Your datatypes and existing standards for what constitutes acceptable data management will vary. 2. The extent to which your scientific community has implemented FAIR: Some disciplines have significant guidelines on FAIR, while others have not addressed the subject in any concerted way. 3. Your level of technical skills: Some approaches to implementing FAIR may require technical skills you may not yet feel comfortable with. While a lot is up to you, the first step is to evaluate how FAIR you think your data are: ??? question "Assessing the FAIRness of you data" Thinking about a dataset you work with, complete the ARDC [FAIR assessment](https://ardc.edu.au/resource/fair-data-self-assessment-tool/){target=_blank} in your own time. ??? note "Resources" - [The FAIR Guiding Principles for scientific data management and stewardship](https://www.nature.com/articles/sdata201618){target=_blank} - [Wilkinson et al. (2016)](https://doi.org/10.1038/sdata.2016.18){target=_blank} established the guidelines to improve the Findability, Accessibility, Interoperability, and Reuse (FAIR) of digital assets for research. - [Go-FAIR website](https://www.go-fair.org/fair-principles/){target=_blank} - [Carroll *et al.* (2020)](http://doi.org/10.5334/dsj-2020-043){target=_blank} established the CARE Principles for Indigenous Data Governance. [full document :fontawesome-solid-file-pdf:](https://static1.squarespace.com/static/5d3799de845604000199cd24/t/5da9f4479ecab221ce848fb2/1571419335217/CARE+Principles_One+Pagers+FINAL_Oct_17_2019.pdf){target=_blank} - [Indigenous Data Sovereignty Networks](https://indigenousdatalab.org/networks/){target=_blank} --- ## Data Self-assessment !!! question "Activity" In small groups, discuss the following questions. 1. What are the two or three data types that you most frequently work with? - Think about the sources (observational, experimental, simulated, compiled/derived) - Also consider the formats (tabular, sequence, database, image, etc.) 2. What is the scale of your data? ??? tip "Tip: think of the [Three V's](https://www.bigdataframework.org/four-vs-of-big-data/){target=_blank}" - Volume: Size of the data (MBs, GBs, TBs); can also include how many files (e.g dozens of big files, or millions of small ones) - Velocity: How quickly are these data produced and analyzed? A lot coming in a single batch infrequently, or, a constant small amount of data that must be rapidly analyzed? - Variety: How many different data types (raw files? databases?) A fourth V (Veracity) captures the need to make decisions about data processing (i.e., separating low- and high-quality data) 3. What is your strategy for storing and backing up your data? 4. What is your strategy for verifying the integrity of your data? (i.e. verifying that your data has not be altered) 5. What is your strategy for searching your data? 6. What is your strategy for sharing (and getting credit for) your data? (i.e. How will do you share with your community/clients? How is that sharing documented? How do you evaluate the impact of data shared? ) --- ## Data Management Plans
"Those who fail to plan, plan to fail."
!!! success "Learning Objectives" - Describe the purpose of a data management plan - Describe the important elements of a data management plan !!! quote "What is a DMP?" "A data management plan or DMP is a formal document that outlines how data are to be handled both during a research project, and after the project is completed. The goal of a data management plan is to consider the many aspects of data management, metadata generation, data preservation, and analysis before the project begins; this may lead to data being well-managed in the present, and prepared for preservation in the future." Source: [Wikipedia](https://en.wikipedia.org/wiki/Data_management_plan){target=_blank}. Here are some [Example DMPs](https://dmptool.org/public_plans){target=_blank} made public from the DMPtool website. You can use these as example for creating your own DMP. **Why bother with a DMP?** ??? question "How would you answer?" Do you have a data management plan? If so, how do you use it? Returning to the assertion that data (and its value) is at the foundation of your science, working without a data management plan should be considered scientific misconduct. Those are strong words. And while we might have an intuition of the boundaries of research ethics - data mismanagement seems more like an annoyance than misconduct. However, if your mismanagement leads to error in your research data, or the inability to make publicly-funded research open to the public, these are serious consequences. Increasingly, funders realize this.**Additional Info** - General guidance on how to choose a license https://choosealicense.com/ - More good guidance on how to choose a license https://opensource.guide/legal/ - Licensing options for your [Github Repository](https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/customizing-your-repository/licensing-a-repository#choosing-the-right-license){target=_blank} ??? note "References and Resources" - [NSF Guidelines on DMPs](https://www.nsf.gov/bio/biodmp.jsp){target=_blank} - https://dmptool.org/general_guidance - https://dmptool.org/public_templates - Professional and scholarly societies, e.g., theEcological Society of America http://www.esa.org/esa/science/data-sharing/resources-and-tools/ - DataOne - https://dataoneorg.github.io/Education/bestpractices/ - Data Carpentry - http://datacarpentry.org/ - The US Geological Survey https://www.usgs.gov/data-management - Repository registry (and search) service: http://www.re3data.org/ - Your university library --- ## :material-file-document-multiple: Project Documentation _A great Open Scientist is someone who documents their work and shares it with the world. This means going well beyond peer-reviewed publications._ Here are some discussing points on *why* you should carry out documentation: - Describe how to use or build your computer code or tools - Share best practices for a method or protocol - Create and share educational material so others in your field can learn from you - Share a first-person account of your journey through a project
!!! quote "Things to remember about Documentation" - Documentation should be written in such a way that people who did not write the documentation can read and then use or read and then teach others in the applications of the material. - Documentation is best treated as a living document, but version control is necessary to maintain it - Technology changes over time, expect to refresh documentation every 3-5 years as your projects age and progress.
--- ## Self Assessment ??? question "What is a Data Management Plan?" **Important**: A data management plan (DMP) is now required aspect of publicly funded research. DMPs are short, formal, documents outlining what types of data will be used, and what will be done with the data both during and after a research project concludes. ??? question "True or False: When science project funding ends, the data should end with it" **False** Data live on after a project ends. Ensuring that data have a full lifecycle where they can be (re)hosted and made available after a project ends is critical to open science and reproducible research ??? danger "... or maybe?" Sometimes destroying data is part of the life cycle of data - this may be required if data are sensitive and could be used unethically in the future, beyond the control of the original investigator team. ??? question "True or False: FAIR and CARE data principles are the same" **False** The CARE principles were created in order to help guide and answer when and how applying FAIR data principles to soverign indigenous-controlled data should be done and when it should not. ??? question "(1/2) Your project has been greenlit and you require to create a GitHub repository for your work; What you are working on is likely going to be used by others in your field, thus you want to use a licence that allow for others to access your work and make changes where needed. Which licence do you choose?" **MIT License** A short and simple permissive license with conditions only requiring preservation of copyright and license notices. Licensed works, modifications, and larger works may be distributed under different terms and without source code. ??? question "(2/2) You approach your PI with the idea, and they're not happy with the choice. They are pushing to keep the attribution to your lab; Which license type would be useful in this situation?" **CC By** Rights are not waived, attribution is required.
Adapted from [CyVerse FOSS](https://github.com/CyVerse-learning-materials/foss/blob/86d4cc384976a59bd19cce2a6fdcf994b19051bd/docs/02_managing_data.md){target=_blank} (last source update 2025-09-08), CC BY 4.0. Spotted a problem? [Open an issue](https://github.com/UNM-CARC/foss/issues){target=_blank}.
---8<--- https://unm-carc.github.io/foss/lessons/03-talk-to-computers/ --- title: "How to Talk to Computers" description: "An introduction to the Unix shell: navigating directories, working with files, and automating tasks with shell scripts." type: Lesson tags: - Command Line - Shell - Bash - Linux generated: by: "claude/opus-5" at: "2026-09-11T00:00:00Z" sources: - id: cyverse-foss resource: "https://github.com/CyVerse-learning-materials/foss/blob/86d4cc384976a59bd19cce2a6fdcf994b19051bd/docs/03_talk_to_computer.md" title: "CyVerse FOSS: docs/03_talk_to_computer.md" author: "team:cyverse" last_modified: "2025-09-16T10:46:46-07:00" --- # How to Talk to Computers ## The Command Line Interface When using a computer, it is typical to use a keyboard and mouse to navigate a cursor across the screen or simply tap on the screens of our smart phones or tablets. Both of these methods make use of the Graphical User Interface (GUI) and have become central to the way we interact with computers. GUIs make computers so easy to use! However, for a more direct and powerful way to instruct your computer, you should learn to use the **Command Line Interface (CLI)**. CLIs are found throughout all operating systems (Windows, MacOS, Linux) though they might have different commands and syntax. For this FOSS lesson on CLI, we will focus on the Unix CLI which is present in MacOS and all Linux operating systems.!!! warning "**Attention** :material-microsoft-windows: Windows users" Much of what we are going to be teaching is based on open-source software which operates on cloud and is incompatible with Windows OS. Unix-based systems such as Linux [:material-ubuntu: Ubuntu](https://ubuntu.com/){target=_blank} and [:material-apple: MacOS X](https://www.apple.com/macos/){target=_blank}, as many scientific tools require a Unix Operating System (OS). There are a number of software that allow :material-microsoft-windows: Windows users to execute Unix commands, however we recommend the use of [:simple-linux: Windows Subsystem for Linux (WSL) 2.0](https://docs.microsoft.com/en-us/windows/wsl/install){target=_blank}. ??? tip "Quickstart installation of Window's WSL" !!! warning "A system reboot is necessary" 1. Open :material-powershell: PowerShell in Administrator mode (open :octicons-search-16: Search and look for PowerShell, right click and select "Run as Administrator") 2. type `wsl --install` 3. Restart your machine 4. Open :octicons-search-16: Search and open :simple-linux: WSL; create a username and password, wait for it to finish setting up (should take a few minutes) 5. You're now ready to use :simple-linux: Linux on your Windows Machine! ??? question "Where is the WSL Home folder?" The Home folders for Linux and Windows are different. The Windows path to the :simple-linux: WSL home folder is `\\wsl$\Ubuntu\home\
---
## The Unix Shell The CLI sees the computer stripped down to only a [Terminal](https://en.wikipedia.org/wiki/Terminal_emulator){target=_blank} from where one can run powerful commands executed through the [Shell](https://en.wikipedia.org/wiki/Shell_(computing){target=_blank}). Though there are technical differences between them, the terms **Command Line Interface**, **Terminal**, **Shell**, and **BASH** will be used more or less interchangeably throughout the lesson.
??? info "Tips for Directory Navigation" `.` refers to *current* directory `..` refers to *above* directory `/` is the directory separator `~` indicates the home directory For example: ``` $ ls . # lists files and folders in the current directory $ ls .. # lists files and folders in the above directory $ ls ~ # lists files and folders in the home directory $ ls ~/Documents # lists files and folders in Documents (a folder present in the home directory) ``` !!! tip "Use the Tab key to autocomplete" You do not need to type the entire name of a folder or file. By using the tab key, the Shell will autocomplete the name of the files or folders. For example, typing the following ``` $ ls exer ``` and pressing the tab key, will result in autocompletion. ``` $ ls exercise-data/ ``` You can then press tab twice, to print a list of the contents of the folder. ``` $ ls exercise-data/ animal-counts/ creatures/ numbers.txt proteins/ writing/ ``` ### Working with Files and Directories | Command | Explanation | |---|---| |`mkdir`| make a directory | |`touch`| creat empty file | |`nano` or `vim`| text editors | |`mv`| move command | |`cp`| copy command | |`rm`| remove command | ??? info "Help with Commands" For every command, typing `man` (manual) before the command, will open the manual for said command. ``` $ man ls ``` - The above command will result in opening the *manual* for the `ls` command. You can exit the man page by pressing `q`. Return to `shell-lesson-data`, and create a directory with `mkdir
??? question "What will happen if you create a directory with spaces?" You will obtain as many folders as typed words! ``` $ mkdir my folder $ ls -F exercise-data/ folder/ my/ north-pacific-gyre/ ``` Notice the two folders `my` and `folder`. Create an empty file with `touch
!!! example "Script exercises" ??? example "word counting" ```bash #!/bin/bash # Find haiku.txt starting from current directory file_path=$(find . -name "haiku.txt" | head -n 1) # Navigate to its directory cd "$(dirname "$file_path")" # Print absolute path echo "haiku.txt found at: $(pwd)/haiku.txt" # Define keyword to search keyword="not" # Count how many times the keyword appears keyword_count=$(grep -o "$keyword" haiku.txt | wc -l) # Extract and append keyword context echo "" >> haiku.txt echo "---- Keyword Summary ----" >> haiku.txt echo "Keyword '$keyword' appears $keyword_count times" >> haiku.txt echo "Summary generated on: $(date)" >> haiku.txt echo "--------------------------" >> haiku.txt # Show updated file cat haiku.txt ``` ??? example "Create a compressed backup with a timestamp" ```bash #use Bash shell to run the following commands #!/bin/bash ## Variables #the directory you want to back up (e.g., shell-lesson-data) SOURCE_DIR=$(find $PWD -type d -name "shell-lesson-data" 2>/dev/null) #location where the backup will be stored BACKUP_DIR="$PWD" #used to create a unique name for each backup based on the current date and time TIMESTAMP=$(date +"%Y-%m-%d_%H-%M-%S") # name of the compressed backup file ARCHIVE_NAME="backup_$TIMESTAMP.tar.gz" # Create backup directory if it doesn't exist mkdir -p "$BACKUP_DIR" # Create a compressed archive of the source directory tar -czf "$BACKUP_DIR/$ARCHIVE_NAME" -C "$SOURCE_DIR" . # Output the result echo "Backup of $SOURCE_DIR completed!" echo "Archive created at $BACKUP_DIR/$ARCHIVE_NAME" ``` Exit nano with `ctrl + x` Modify permission to make the shell script executable ``` $ chmod +x script.sh ``` Run the shell script ``` $ ./script.sh ``` ### More Carpentries Lessons on Linux Command line - [Pipes and Filters](https://swcarpentry.github.io/shell-novice/04-pipefilter.html){target=_blank} - [Loops](https://swcarpentry.github.io/shell-novice/05-loop.html){target=_blank} - [Scripts](https://swcarpentry.github.io/shell-novice/06-script.html){target=_blank} - [Finding Things](https://swcarpentry.github.io/shell-novice/07-find.html){target=_blank} ---
Adapted from [CyVerse FOSS](https://github.com/CyVerse-learning-materials/foss/blob/86d4cc384976a59bd19cce2a6fdcf994b19051bd/docs/03_talk_to_computer.md){target=_blank} (last source update 2025-09-16), CC BY 4.0. Spotted a problem? [Open an issue](https://github.com/UNM-CARC/foss/issues){target=_blank}.
---8<--- https://unm-carc.github.io/foss/lessons/04-talk-to-llms/ --- title: "How to Talk to LLMs" description: "How large language models work, prompt-engineering techniques such as priming, chaining, and the CRAFT framework, and AI coding tools for research." type: Lesson tags: - LLMs - Prompt Engineering - AI generated: by: "claude/opus-5" at: "2026-09-11T00:00:00Z" sources: - id: cyverse-foss resource: "https://github.com/CyVerse-learning-materials/foss/blob/86d4cc384976a59bd19cce2a6fdcf994b19051bd/docs/04_talk_to_llms.md" title: "CyVerse FOSS: docs/04_talk_to_llms.md" author: "team:cyverse" last_modified: "2025-09-22T15:52:56-07:00" --- # How to Talk to LLMs --- [CyVerse Full Prompt Engineering Workshop](https://tyson-swetnam.github.io/intro-gpt){target=_blank} --- !!! success "Learning Objectives" - **Fundamentals**: How AI models process and respond to prompts - **Modern Features**: Leveraging document uploads, web search, and multi-modal inputs - **Best Practices**: Structured approaches to writing effective prompts - **Advanced Techniques**: Context management, chaining, and custom instructions --- ## How is the AI revolution impacting Open Science? !!! quote "Center for Open Science: [Evaluating AI’s Impact on Open Research Infrastructure](https://www.cos.io/blog/evaluating-ai-impact-on-open-research-infrastructure){target=_blank} " !!! success ""*When used responsibly, it can support open models and data, accelerate discovery, and aid in the evaluation of research...**... but it can also undermine credibility when used to plagiarize, fabricate findings, or mislead readers.* "" !!! failure ""*With the emergence of generative AI, it has become very easy to create content that looks like a real research paper, but is not genuine research.*"" !!! quote "MDPI Blog: [How Artificial Intelligence is Accelerating Open Access Science](https://blog.mdpi.com/2025/09/09/open-science-artificial-intelligence/){target=_blank}" !!! success ""*Artificial intelligence, like any new technology, presents both a threat and an opportunity. It requires reflection, adjustment, and adaptation.
If implemented carefully and thoughtfully, AI could help us respond to some of the issues that the Open Access scientific publishing industry currently faces. These include the increasing amounts of data being produced and also language barriers and imbalances in outputs between countries.
Further, AI could help to promote openness in datasets and content aggregators.*"" !!! failure ""*GPT is not fully reliable.
GPT training involves analysing a huge body of text and noticing patterns so it can predict the next word in a passage.
This results in human-like text, meaning it sounds like it’s written by a human but may not necessarily be by one. Similarly, it may sound like it is conveying meaning, but the argument or claim being made may be without evidence or structure.*"" !!! success ""*Artificial intelligence is changing Open Access; it’s changing everything. Ultimately, though, it’s a tool, so how it’s used determines its value.
If used carefully, AI could help advance Open Access by automating repetitive data-related tasks, making the translation process more interactive, and promoting openness in datasets and content aggregators.*
***However, attention must be paid to its flaws and potential for misuse.***""
Current LLMs generally provide recommendation for how _you_ could do things. ie, they provide you code and text recommendations but don't actually execute anything. But these technologies are advancing quickly and new capabilities are developed and released constantly. Soon, [AI Agents](https://github.com/Significant-Gravitas/AutoGPT){target=_blank} could be everywhere executing on instructions in autonomous and semi-autonomous ways.
### Commercial Chatbots
⚠️⚠️ VERIFY EVERTHING CHATBOTS TELL YOU! ⚠️⚠️
--- ### :material-robot-outline: Prompt Writing LLM Chatbots are meant to be conversational. In general, you are asking the Chatbot questions (known as **Prompts**) and the Chatbot will respond with answers. It is a bit of an artform to get the Chatbot to provide answers with the specificity and format that you want. An entire field of study has sprung up, called **Prompt Engineering**, is a technique of crafting effective instructions using AI large language models. With modern AI-powered tools like Claude Desktop, ChatGPT, Gemini, and NotebookLM offering capabilities to upload documents, search the web, and process multiple file types, mastering prompt engineering has become essential for productive AI interactions.
#### **Prompt Priming** Provide lots of organized details to help the Chatbot understand the question and what it's task is. This could include adding a backstory or context for why you are asking the question. Be very specific in terms of what you want from the Chatbot and how you want it. Zero-shot unconditioned prompts are likely to return the least specific responses. Responses are more likely to be useful when multiple specific output types are defined. | Types of Priming | Example | |------------------|---------| | Zero (Shot) | "Write five examples of assessments for watershed health." | | Single | "Write five examples of assessments for watershed health. Here is one example: Geomorphology" | | Multiple | "Write five examples of assessments for watershed health related to geomorphology, water quality, and species diversity." |
#### **Linked Prompts** Responses to prompts may not return the exact details or information that you are after the first time. Follow-up by rephrasing your prompts more carefully and continuing with iterative prompting can build upon your priors. "Chain prompting" or "Linked Prompting" brings multiple prompts together. | Linked Prompting | Examples | |------------------|----------| | Step 1: Priming | "I want you to act as an eminent hydrologist from CUASHI. Provide me with a list of the ten most important topics in hydrology over the last decade focused around research in the global south, working with indigenous communities, and traditional ecological knowledge systems." | | Step 2: Summarizing | "Based on the list you just created, summarize the most pressing financial challenges faced by indigenous communities in the Global South, versus indigenous communities in North America, in less than 50 words." | | Step 3: Try again with a web search (Control) | "Based on the results of web access, can you confirm the validity of the ten important topics and provide at least one reference to each." | !!! tip "Encouraging the Chatbot to do Better" Chatbot responses can be missing information or just plain wrong. When this occurs, you can point out the mistake and ask the Chatbot to provide a more complete or better answer. Don't settle for poor responses! ``` mermaid graph LR A[Priming] --> B{Result?}; B -->|Yes| C[Summarize]; C --> D[Quality Control]; D --> B; B ----> E[Yay!]; ```
#### Role Playing Some people find that asking the Chatbot to adopt a persona will lead to better responses. "I want you to act as ..." will establish what type of conversation you are planning to have. | Types of Roles | |---| | Project Manager | | Copywriter / Editor | | Paper Reviewer | | Teacher / Mentor / Advisor | | Student / Learner / Participant | | Software Engineer | | DevOps Engineer | | Linux Terminal | | Python Interpreter | | Web Browser | --- ## Understanding Modern AI Capabilities ### How AI Models Process Your Input !!! info "The Processing Pipeline" 1. **Tokenization**: Your prompt is broken into smaller units (tokens) 2. **Context Assembly**: Uploaded documents and conversation history are included 3. **Attention Mechanism**: The model identifies relevant information 4. **Generation**: Response is produced token by token 5. **Formatting**: Output is structured according to your specifications ### The Foundation: Clear Instructions Start with simple, direct prompts before advancing to complex techniques: ```markdown # Basic Prompt "Summarize this research paper in 3 bullet points" ``` ```markdown # Better Prompt "As a research scientist, summarize the key findings from this paper in 3 bullet points, focusing on methodology and results" ``` ```markdown # Best Prompt "You are a research scientist reviewing papers for a journal. Summarize the attached PDF in 3 bullet points that cover: 1. Research question and hypothesis 2. Methodology and sample size 3. Key findings and limitations Format as a bullet list with sub-points for clarity." ``` ### Core Features of Today's AI Tools Modern AI assistants have evolved beyond simple text chat: | Feature | :simple-claude: Claude | :material-robot-outline: ChatGPT | :simple-googlegemini: Gemini | :simple-notebooklm: NotebookLM | :material-microsoft: CoPilot | |---------|--------|---------|--------|------------|---------| | **Document Upload** | PDFs, text, code | PDFs, images, data | PDFs, images, GDrive | PDFs, Google Docs | PDFs, OneDrive | | **Web Search** | Via MCP | Yes | Yes | Yes | Yes | | **Context Window (tokens)** | 200K | 128K| 2M | Document-based | 128K | | **File Analysis** | Yes | Yes | Yes | Deep analysis | Yes | | **Code Execution** | Yes (MCP) | Yes | Yes | No | Yes | ### Working with Documents Modern AI tools excel at document analysis. Here's how to maximize their potential: !!! success "Document Upload Best Practices" - **Specify the document**: "In the attached PDF..." or "Based on the uploaded spreadsheet..." - **Direct attention**: "Focus on Section 3.2 of the document" - **Request specific outputs**: "Create a table comparing the methods described in chapters 2 and 5" - **Combine multiple sources**: "Compare the findings in these three papers" #### Example: Multi-Document Analysis ```markdown I've uploaded three research papers on climate change. Please: 1. Create a comparison table with columns for: - Paper title and authors - Methodology - Key findings - Limitations 2. Identify common themes across all papers 3. Highlight any contradictory findings Format the response with clear headers and use markdown tables. ``` --- ## The CRAFT Framework For consistent, high-quality results, use the [CRAFT framework](https://www.geeky-gadgets.com/craft-prompt-framework/){target=_blank}: | Action | Description | |---|---| | **Context** | Provide background information and set the scene | | **Role** | Define who the AI should act as | | **Action** | Specify exactly what you want done | | **Format** | Describe how the output should be structured | | **Tone** | Indicate the style and voice to use | #### CRAFT Example ```markdown Context: I'm preparing a grant proposal for NSF funding on AI in education Role: Act as an experienced grant writer and education researcher Action: Review my draft introduction and suggest improvements Format: Provide feedback as tracked changes with explanations Tone: Professional, constructive, and encouraging ``` --- ## Advanced Techniques ### 1. Custom Instructions and System Prompts Modern AI platforms allow you to set persistent instructions: !!! example "'Custom Instructions' or 'System Instructions'" Platforms like Gemini and Claude allow you to add "Custom Instructions" or "System Instructions" as prior prompts, which act as a global rule to subsequent prompt chaining. For example: ```markdown # Project Context I'm a data scientist working on machine learning projects. Always provide Python code examples using scikit-learn and pandas. Include docstrings and type hints in all code. # Response Preferences - Be concise but thorough - Explain complex concepts with analogies - Always cite sources when making factual claims ``` ### 2. Leveraging Web Search Most featured GPTs now feature a web browse or search engine capability. Enabling search allows the GPT to use document retrieval on websites and PDFs when reasoning out its response. ```markdown Search for the latest research on the public health benefits of vaccination published in 2024. Focus on: - Papers from top conferences (AHA, ASPPH, NRHA, ICFMDP) - mRNA - Bird Flu and COVID Summarize the top 5 papers with links to the originals. ``` ### 3. Multi-Modal Prompting Combine different input types for richer interactions: ```markdown I've uploaded: 1. A screenshot of my dashboard 2. The underlying data in CSV format 3. Our brand guidelines PDF Create a redesigned dashboard that: - Improves data visualization based on best practices - Adheres to our brand colors and fonts - Highlights the KPIs mentioned in the data dictionary ``` ### 4. Prompt Chaining Build complex outputs through sequential prompts: !!! tip "Effective Chaining Strategy" 1. **Start broad**: "Outline a research paper on sustainable AI" 2. **Zoom in**: "Expand section 3 on energy-efficient training methods" 3. **Refine**: "Add citations and make the tone more academic" 4. **Polish**: "Format according to IEEE standards" ### 5. Using Examples (Few-Shot Learning) Provide examples to guide the AI's output: ```markdown I need to classify customer feedback. Here are examples: "The product arrived damaged" → Category: Shipping Issue "Can't log into my account" → Category: Technical Support "Love the new features!" → Category: Positive Feedback Now classify these: 1. "The app keeps crashing on startup" 2. "Best purchase I've made this year" 3. "Package was left in the rain" ``` --- ## Practical Applications ### Research and Analysis ```markdown Analyze the attached dataset (CSV) and: 1. Identify statistical patterns and outliers 2. Create visualizations for the top 3 insights 3. Write a methods section describing the analysis 4. Suggest additional analyses based on the data Use pandas profiling techniques and create matplotlib visualizations. Include code that I can run locally. ``` ### Writing and Editing ```markdown I've uploaded my draft manuscript. Please: 1. Check for consistency in terminology throughout 2. Ensure all figures are referenced in the text 3. Verify the citation format matches APA 7th edition 4. Highlight any unclear passages 5. Suggest improvements for flow between sections Provide a tracked-changes version and a summary of major edits. ``` ### Code Development ```markdown Based on the uploaded requirements document: 1. Create a Python class structure for the described system 2. Include comprehensive docstrings and type hints 3. Add unit tests for each method 4. Create a README with installation and usage instructions 5. Follow PEP 8 style guidelines Use modern Python features (3.10+) and include error handling. ``` --- ## Common Pitfalls and Solutions ### Pitfall 1: Vague Instructions !!! failure "*Poor*: "Make this better"" !!! success "*Better*: "Improve this abstract by making it more concise (under 250 words), adding keywords, and ensuring it follows the journal's structure: background, methods, results, conclusions"" ### Pitfall 2: Information Overload !!! failure "*Poor*: "Uploading 50 documents without guidance"" !!! success "*Better*: "Focus on documents 1-3 which contain the methodology. Ignore the appendices."" ### Pitfall 3: Assuming Knowledge !!! failure "*Poor*: "Fix the usual issues"" !!! success "*Better*: "Check for: passive voice, sentences over 25 words, undefined acronyms, and missing Oxford commas"" ### Pitfall 4: No Output Format !!! failure "*Poor*: "Summarize this"" !!! success "*Better*:
Create an executive summary with:
- 3-sentence overview
- 5 key points as bullets
- 1 paragraph on implications
- Formatted with markdown headers" --- ## Prompting Chatbots for FOSS
#### Provide a general outline for a data management plan ``` I am writing a grant proposal to the National Science Foundation. Could you please provide me a basic template for a data management plan (DMP) and please provide url links to resources that can help me with NSF DMP requirements. ```
#### Provide a step-by-step recipe to create and serve an mkdocs website in Github ``` I would like to create a personal website using the MKdocs style and host it on Github pages. Could you please write me a step-by-step guide starting with importing an existing github repository that has the mkdocs material. ```
#### Write shell commands and shell scripts ``` I would like to create a linux shell script to automate the backup of my working directory. Could you please suggest a shell script that will copy my working directory in a different directory and compress the file into an archive. Please name the file based on the current time and date. ```
#### Write git commands ``` Could you please provide me a step-by-step workflow for using git with github? I found a repository that I want to build on in Github. I would like to work on the material on my local machine and then save it back up to github. I would like to workflow to be for the linux command line. ```
#### Write download and conda commands ``` I am writing a lot of scripts using python. I have heard that environment managers such as conda may be useful to me. I don't know anything about conda, so can you explain some things? 1. Give me a high level overview of what environment managers are and what conda is specifically. 2. Please create a step-by-step guide for downloading conda on my machine, and how to use conda to create custom environments. 3. Please explain and give my steps to share my environment with colleagues. ```
#### Write docker run commands ``` I would like to run a docker container that consists of a jupyter notebook. Can you please suggest a docker run command that launches the jupyter notebook and mounts a volume of data in it. ```
#### Write docker files ``` I would like to create a docker image that consists of R studio and some customized Rcode. Can you tell me the steps to 1. make a dockerfile and build the docker image; and 2. Upload the docker image to docker hub. ```
??? tip "ChatGPT :simple-awesomelists: Awesome Lists" There is an ever changing meta-list of :simple-awesomelists: Awesome lists curated around ChatGPT plugins and extensions. [:simple-github: search: `chatgpt+awesome`](https://github.com/search?q=awesome-chatgpt+&type=repositories&s=stars&o=desc) Check out lists around: [:simple-awesomelists: ChatGPT Prompts](https://github.com/f/awesome-chatgpt-prompts){target=_blank} [:simple-awesomelists: ChatGPT Data Science Prompts](https://github.com/travistangvh/ChatGPT-Data-Science-Prompts){target=_blank} [:simple-awesomelists: API plugins, extensions, & applications](https://github.com/humanloop/awesome-chatgpt){target=_blank} --- ## Local LLMs vs APIs ### Managing API keys #### Extension Installation 1. **Open VS Code**. 2. Navigate to the **Extensions view** by clicking the :material-puzzle-outline: icon in the Activity Bar on the side of the window or by pressing `Ctrl+Shift+X` (Windows/Linux) or `Cmd+Shift+X` (macOS). 3. In the search bar, type "**Cline**" 4. Find the official extension from the search results and click **Install**. 5. Once installed, you might need to **reload VS Code** if prompted. #### Selecting an API After installation, you'll typically need to configure an LLM API endpoint and key. Look for settings related to Roo Code or Cline in VS Code's settings (`Ctrl+,` or `Cmd+,`). ##### Google Gemini 1. Obtain your **Google Gemini API key** from [Google AI Studio](https://aistudio.google.com/){target=_blank} or Google Cloud Console. 2. In VS Code settings, search for "Roo Code Gemini" or a similar setting. 3. Enter your API key in the designated field (e.g., `Roo Code: Gemini API Key`). 4. You might also need to specify the model (e.g., `gemini-pro-2.5`). ##### Ollama (for Local Models) [Ollama](https://ollama.com){target=_blank} allows you to run open-source LLMs locally. 1. Ensure **Ollama is installed and running** on your machine with the desired models downloaded (e.g., `ollama pull gemma3:1b`). 2. In VS Code settings for Roo Code/Cline, look for an option to specify the **Ollama API endpoint**. This is usually `http://localhost:11434` by default. 3. Select or specify the Ollama model you wish to use (e.g., `gemma`, `qwen`). No API key is typically needed for local Ollama usage directly, but the extension must be configured to point to the local server. ##### OpenAI Compatible This is for services that adhere to the OpenAI API specification, which can include OpenAI itself or other providers like Azure OpenAI or local LLM servers. 1. Obtain your **API key** and **API base URL** (endpoint) from your provider. * For OpenAI: Key from [platform.openai.com](https://platform.openai.com/api-keys){target=_blank}. Endpoint is typically `https://api.openai.com/v1`. * For Azure OpenAI: Key and endpoint from your Azure deployment. * For others: Refer to your provider's documentation. 2. In VS Code settings for Roo Code/Cline: * Enter the API key (e.g., `Roo Code: OpenAI API Key`). * Enter the API base URL if it's different from the default (e.g., `Roo Code: OpenAI API Base URL`). * Select the desired model (e.g., `gpt-4o`). ##### Claude (via API) If Roo Code/Cline supports direct Claude API integration (distinct from the Claude Desktop app): 1. Obtain your **Anthropic API key** from the [Anthropic Console](https://console.anthropic.com/){target=_blank}. 2. In VS Code settings for Roo Code/Cline, search for "Roo Code Claude" or a similar setting. 3. Enter your API key (e.g., `Roo Code: Claude API Key`). 4. Specify the Claude model you wish to use (e.g., `claude-4-sonnet`). !!! tip "Restart for Changes" After changing API settings, it's often a good idea to restart VS Code or the extension itself if it provides such an option, to ensure the new settings take effect. --- ## Setting up GitHub Copilot on VS Code Locally GitHub Copilot is deeply integrated into the GitHub ecosystem and VS Code (local). ### In GitHub CodeSpaces 1. **Enable Copilot for your account**: Ensure you have an active GitHub Copilot subscription associated with your GitHub account. 2. **Launch a CodeSpace**: When you create or open a repository in GitHub CodeSpaces, Copilot is often enabled by default if your account has access. 3. **Check Status**: Look for the Copilot icon :octicons-copilot-16: in the status bar at the bottom of the VS Code interface within CodeSpaces. If it's not active, click it to see options or troubleshoot. You might need to authorize it for the specific CodeSpace. ### Extension Installation in VS Code (Desktop) 1. **Open VS Code**. 2. Navigate to the **Extensions view** (:material-puzzle-outline: or `Ctrl+Shift+X` / `Cmd+Shift+X`). 3. Search for "**GitHub Copilot**". 4. Find the official extension by GitHub and click **Install**. 5. **Sign In**: After installation, VS Code will prompt you to sign in with your GitHub account. Follow the prompts to authorize VS Code to use GitHub Copilot. * If you're not prompted, you can often click the user icon in the bottom left of VS Code and sign in there, or find a "Sign In to GitHub Copilot" command in the Command Palette (`Ctrl+Shift+P` or `Cmd+Shift+P`). 6. Once signed in and with an active subscription, Copilot will be ready to assist you. You'll see its icon :octicons-copilot-16: in the status bar. --- ## Model Context Protocol (MCP) Model context protocol (MCP) is an open-source standard created by Anthropic, designed to communicate with LLMs and other AI systems. MCP establishes a common protocol (language) for an AI assistant ("client") to request information or to execute actions from an external service ("server"). MCP actions include reading the contents of files, querying other APIs or databases, writing new files or copying data, or executing commands. the MCP protocol defines the structure of the messages. MCPs offer a powerful framework for enhancing scientific reproducibility. - Standardized Data Access - Containerized Computational environments - Sharable Workflows ### MCPs for Coding and commands A foundational MCP tool to use is the `filesystem` which gives the LLM the ability to `read`, `write`, and `execute` code on your computer or a remote server. - Ask the LLM to create a new file - Request the AI to refactor code or edit text in a file - Search through your codebase for relevant functions - Execute terminal commands --- ## Vibe Coding Vibe coding refers to using an LLM to generate and edit code directly within your IDE (e.g., VS Code). This approach allows for a more fluid and interactive coding experience, where the LLM acts as a collaborative partner. !!! warning "Allowing an LLM to execute code on your computer may be a violation of institutional security and privacy policy" Coding tools like Cline and Windsurf give you the option to allow 'execution' of code on your machine. You must understand the implications of giving these LLMs the authority to execute code on your computer and the network it is running upon. !!! danger "Malicious code lives on the internet, and your Vibing LLM might install it while you're not paying attention" Read more: [📰 Vibe Check: False Packages A New LLM Security Risk](https://hackaday.com/2025/04/12/vibe-check-false-packages-a-new-llm-security-risk/){target=_blank} ## Vibe Coding Platforms | Emoji | Meaning | |-------|---------| | :material-microsoft-visual-studio-code: | VS Code | | :octicons-codespaces-16: | GitHub CodeSpace | | :material-apple: | Apple OS | | :material-microsoft-windows: | Windows | | :simple-gnubash: | Command Line Interface | | :material-open-source-initiative: | Open Source | | :material-license: | Licensed | | :material-api: | API based | * [:simple-anthropic: Claude Desktop](https://claude.ai/download){target=_blank} :material-apple: :material-microsoft-windows: :material-api: An easy-to-install desktop platform that connects to Anthropic's powerful LLM API, and allows you to connect to MCP servers. * [:material-cursor-default-click: Cursor](https://www.cursor.com/en){target=_blank} :material-microsoft-visual-studio-code: :material-open-source-initiative: :material-license: A popular standalone fork of VS Code, focused on integrating new models with stability and offering a flat-fee pricing model. * [:octicons-copilot-16: GitHub Copilot](https://github.com/features/copilot){target=_blank} :material-microsoft-visual-studio-code: :octicons-codespaces-16: :material-license: :material-api: Integrated with VS Code and GitHub CodeSpaces, provides agentic coding with periodic performance fluctuations and tiered pricing. * [:material-robot: Cline](https://github.com/cline/cline){target=_blank} :material-microsoft-visual-studio-code: :material-open-source-initiative: :material-api: Open-source and model-agnostic, pioneering features like “bring your own model” (BYOM) and operating on a per-request billing structure. * [:material-surfing: Windsurf](https://windsurf.com/editor){target=_blank} :material-microsoft-visual-studio-code: :material-license: :material-api: Offers similar agentic and inline features with tiered pricing and a “just works” usability orientation. --- ## Quick Reference Card !!! success "Prompt Engineering Checklist" - [ ] **Clear objective**: What do you want to achieve? - [ ] **Context provided**: Background information included? - [ ] **Role defined**: Who should the AI act as? - [ ] **Specific action**: Exact task described? - [ ] **Output format**: Structure specified? - [ ] **Examples given**: For complex tasks? - [ ] **Constraints noted**: Length, style, or content limits? - [ ] **Documents referenced**: If using uploads? - [ ] **Follow-up planned**: For iterative improvement? ## Assessment Questions ??? question "How do modern AI tools handle uploaded documents?" !!! success "Answer" Modern AI tools process uploaded documents by: - Converting them to text (OCR for images/PDFs) - Adding them to the context window - Allowing specific references ("In section 2.3...") - Enabling cross-document analysis - Maintaining document structure awareness ??? question "What's the most important element of an effective prompt?" !!! success "Answer" **Clarity of instruction** is paramount. The AI needs to understand: - What you want done (action) - How you want it done (format) - Why you want it done (context) Without clear instructions, even the most advanced AI will produce suboptimal results. ??? question "How can you ensure consistent outputs across multiple sessions?" !!! success "Answer" 1. **Use custom instructions** (ChatGPT, Claude) or system prompts 2. **Create templates** for common tasks 3. **Save successful prompts** for reuse 4. **Use platform features** like GPTs or Projects 5. **Include examples** in your prompts 6. **Specify exact formats** with templates ??? question "True or False: Longer prompts always produce better results" !!! failure "False" Prompt quality matters more than length. A well-structured, concise prompt often outperforms a lengthy, unfocused one. However, providing sufficient context and clear instructions is important. Aim for: - **Completeness** over brevity - **Clarity** over complexity - **Structure** over stream-of-consciousness ## Further Resources - [:simple-claude: Anthropic's Prompt Engineering Guide](https://docs.anthropic.com/claude/docs/prompt-engineering){target=_blank} - [:material-robot-outline: OpenAI's Best Practices](https://platform.openai.com/docs/guides/prompt-engineering){target=_blank} - [:simple-googlegemini: Google's Gemini Prompting Strategies](https://ai.google.dev/gemini-api/docs/prompting-strategies){target=_blank} - [:simple-github: Awesome ChatGPT Prompts](https://github.com/f/awesome-chatgpt-prompts){target=_blank} - [Learn Prompting Online Courses](https://learnprompting.org/){target=_blank} ---
Adapted from [CyVerse FOSS](https://github.com/CyVerse-learning-materials/foss/blob/86d4cc384976a59bd19cce2a6fdcf994b19051bd/docs/04_talk_to_llms.md){target=_blank} (last source update 2025-09-22), CC BY 4.0. Spotted a problem? [Open an issue](https://github.com/UNM-CARC/foss/issues){target=_blank}.
---8<--- https://unm-carc.github.io/foss/lessons/05-version-control/ --- title: "Reproducibility I: Version Control" description: "Version control with Git and GitHub: repositories, the commit-push-pull cycle, branching, pull requests, and handling large files." type: Lesson tags: - Git - GitHub - Version Control - Reproducibility generated: by: "claude/opus-5" at: "2026-09-11T00:00:00Z" sources: - id: cyverse-foss resource: "https://github.com/CyVerse-learning-materials/foss/blob/86d4cc384976a59bd19cce2a6fdcf994b19051bd/docs/05_version_control.md" title: "CyVerse FOSS: docs/05_version_control.md" author: "team:cyverse" last_modified: "2025-09-29T14:46:38-07:00" --- # Reproducibility I: Version Control !!! success "Learning Objectives" After this lesson, you should be able to: * Understand the basics of `git` as a resource for reproducible programming * Describe tools and approaches to creating your `git` Repositories * Describe best practices for maintaining GitHub Organizations and Repositories * Maintain own GitHub user profile and repositories Version control refers to keeping track of the version of a file, set of files, or a whole project. Some version control tools: - :material-microsoft-office: Microsoft Office's [*Track Changes*](https://support.microsoft.com/en-au/office/track-changes-in-word-197ba630-0f5f-4a8e-9a77-3712475e806a){target=_blank} functionality - :simple-apple: Apple's [*Time Machine*](https://support.apple.com/en-us/HT201250){target=_blank} - :simple-googledocs: Google Docs' [*Version History*](https://support.google.com/docs/answer/190843?hl=en&co=GENIE.Platform%3DDesktop){target=_blank} - :simple-git: [Git](https://git-scm.com/){target=_blank} Version control is as much a philosophy as a set of tools; you don't need to master Git to utilize version control (though it is certainly a worthwhile tool for many researchers). !!! info "Why Version Control Matters to Open Science" GitHub is widely used to share code, build websites and documentation, manage workflows or simply allow for your work to be accessible everywhere. Importantly, **git** is a tool that allows for your code to be accessed by machines which only allow for terminal use, making your code accessibility largely platform agnostic. Let's recall a few of Open Science's goals: accessibility, reproducibility, collaboration and transparency. Working with GitHub allows for your work to attain all of these points: - Accessibility: with your work being saved in the GitHub platform, either publically or privately, you're ensuring collaborators can access your work. - Reproducibility: git saved content allows for accessing your work at different stages thanks to Version Control. - Collaboration: GitHub is accessible everywhere (with an internet connection). - Transparency: everything is tracked unless specified in the `.gitignore`, a special file that when populated correctly allows for certain files to be skipped when saving your work.(graphic's correction: ~~marged~~ merged)
Adapted from [CyVerse FOSS](https://github.com/CyVerse-learning-materials/foss/blob/86d4cc384976a59bd19cce2a6fdcf994b19051bd/docs/05_version_control.md){target=_blank} (last source update 2025-09-29), CC BY 4.0. Spotted a problem? [Open an issue](https://github.com/UNM-CARC/foss/issues){target=_blank}.
---8<--- https://unm-carc.github.io/foss/lessons/06-software-environments/ --- title: "Reproducibility II: Software Environments" description: "Computing environments, the system PATH, and using Conda, pip, and renv to create, share, and reproduce software environments." type: Lesson tags: - Conda - Environments - Package Managers - Reproducibility generated: by: "claude/opus-5" at: "2026-09-11T00:00:00Z" sources: - id: cyverse-foss resource: "https://github.com/CyVerse-learning-materials/foss/blob/86d4cc384976a59bd19cce2a6fdcf994b19051bd/docs/06_soft_env.md" title: "CyVerse FOSS: docs/06_soft_env.md" author: "team:cyverse" last_modified: "2025-08-26T15:58:14-07:00" --- # Reproducibility II: Software Environments !!! success "Learning Objectives" After this lesson, you should be able to: * Understand the value of reproducible computing * Know the challenges of reproducible computing * Define a computing environment * Share a software environment with a colleague * Set up a software project with an environment## Reproducible Scientific Computing
!!! quote "Defining Reproducibility" "Reproducing the result of a computation means running the same software on the same input data and obtaining the same results." Rougier et al. 2016 "Getting someone else's code to run on my computer" - Anonymous
### Interactive (ie, point-and-click) Computing
### Scripted Computing
#### Advantages - Much faster to run through commands - The script runs identically every time, reducing the human element - Easy for someone else to quickly reproduce the exact analysis and result - Enables analysis tasks to scale up
#### Challenges - Requires deeper computer knowledge - More upfront effort to produce the script
!!! question "Discussion Question" **What are some tasks you have automated or want to automate?** - Have you ever successfully automated a task? - Found a way to make something scale or take less time? - What was the task, and how did you do it? - Are there any things you wish you could automate? - What are some barriers to automating them?
## Scripting Languages The most common open-source scripting languages (for science) are Python, R, and shell (Bash).
## Computing Environment A computing environment is the combination of hardware, software, and network resources that provide the infrastructure for computing operations and user interactions. - **Hardware**: CPUs, GPUs, RAM - **Operating system & version**: many flavors of Linux, MacOS, Windows - **Software versions:** R, Python, etc. - **Package versions:** specific R or Python packages, which often depend on other packages
!!! warning "**!!Very Important!!**" #### The scripts you create: * Were designed to work in _your_ specific computing environment * May not work on someone else's computer because their computing environment is different * May not work on your computer in the future, because your computing enviroment will probably change (eg., updated software versions)
### Software Dependency Hell Sometimes, it can be _nearly impossible_ to get your computing environment correct enough to run someone else's code. This can be caused by incorrect software versions of the packages you are using or their dependencies. _Don't Dispair! There are solutions to avoid software dependency hell and ensure reproducibility from one computer to another_
## Software Installation When you download and install software onto your computer, it will typically install it in a set of specific directories that we call the **System Path**. ### System Path In the context of computing, the **system path**, often referred to simply as **PATH**, is the set of directories in which the operating system looks for executable files when a command is issued. When you go to launch an application by clicking on a desktop icon or with a CLI command, the computer will search for the application within the PATH directories. If it finds the executable, it will launch. !!! tip "Find the PATH on your computer" In Linux and Mac Terminal `echo $PATH`
In Windows Terminal `$env:PATH`
!!! warning "The PATH prefers one version of any given software."
## Environment Managers One solution to software dependency hell is to use an Environment Manager An environment manager allows you to create software installation directories (similar to PATH) that are **isolated** from your computer's PATH. You can create unique environments and install specific software version to run specific scripts.
### :simple-anaconda: Conda - Open Source Environment Manager [Conda](https://docs.conda.io/en/latest/){target=_blank} is a popular and open source environment manager tool that can be installed on any operating system (Windows, MacOS, Linux). * Users can create environments that have their own set of packages, dependencies, and even their own version of Python. * Projects can have their own specific requirements without interfering with each other * It allows for consistent and reproducible results across different systems and setups
## Sharing your Environment with Colleagues Whether you are using Conda, Pip, or Renv, you should be able to share the specifications of your software environment so colleagues can reproduce the environment. The general sharing workflow: 1. Output an _environment file_ that lists the software and versions of the environment 2. Share the file with colleagues through a platform like Github 3. Colleagues create an empty environment on their computer and populate it with the contents of the _environment file_ ??? tip ":simple-anaconda: Conda to Share Environment" ### :simple-anaconda: [Conda](https://docs.conda.io/en/latest/){target=_blank} 1. Export your Conda Environment ``` conda env export > my_conda_env.yml ``` 2. Share the .yml file through Github 3. Reproduce the Environment on a Different Computer ``` conda env create --file environment.yml ``` !!! success "Conda exports your Pip environment as well" Exporting your environment using Conda (`conda env export > my_conda_env.yml`) will **ALSO** export your pip environment! ??? tip ":simple-python: Pip to Share Environment" ### :simple-python: Python 1. Export python libraries present in your environment ``` pip3 freeze > requirements.txt ``` 2. Share the `requirements.txt` on Github 3. Reproduce the Environment on a Different Computer ``` pip install -r requirements.txt ``` ??? tip ":simple-r: Renv to Share Environment" ### :simple-r: [Renv](https://rstudio.github.io/renv/articles/renv.html){target=_blank} 1. Create an isolated environment ``` renv::init() ``` 2. Export R packages to the renv.lock file ``` renv:snapshot() ``` 3. Share the `renv.lock`, `.Rprofile`, `renv/settings.json` and `renv/activate.R` files to Github 4. Reproduce the Environment on a Different Computer ``` renv::restore() ```
## Package Managers A software tool to find, download, and install software packages to PATH or virtual environment ??? tip ":simple-anaconda: Conda" ### :simple-anaconda: [Conda](https://docs.conda.io/en/latest/){target=_blank} **Software:** Python, R, Django, Celery, PostgreSQL, nginx, Node.js, Java programs, C and C++, Perl, and command line tools **Repository:** [Conda-Forge](https://conda-forge.org/){target=_blank}. ??? tip ":simple-python: Pip" ### :simple-python: [Pip](https://pypi.org/project/pip/){target=_blank} **Software:** python **Repository:** [PyPi](https://pypi.org/){target=_blank} **Note:** Pip can be used together with Conda environment manager. ??? tip ":simple-r: R" ### :simple-r: R With the R language, a package manager is built directly into the R Base Package. ``` install.packages('ggplot2') ``` **Repository:** [R Comprehensive R Archive Network (CRAN)](https://cran.r-project.org/){target=_blank} ---
## Reproducibility Tutorial Using Conda ### Set Up !!! note "OS of choice" To get everyone on the same page, we will do this exercise together using the Linux terminal in Github Codespaces. However, if you'd like to use your own computer feel free to! If you're on Mac or Linux, open your terminal; If you're on Windows, please use the Windows Subsystem for Linux (WSL) so you can follow along. ??? tip "How to Scroll in Cyverse (Tmux) Cloud Shell" If you're using the Cyverse Cloud Shell, you can scroll up and down by pressing `Ctrl + b` and then `[` to enter scroll mode. You can then use the arrow keys to scroll up and down. Press `q` to exit scroll mode. !!! note "The CLI in CyVerse is controlled with [Tmux](https://en.wikipedia.org/wiki/Tmux){target=_blank}, a software that allows to "window" the CLI; Here is a [cheat sheet](https://tmuxcheatsheet.com/){target=_blank} that will teach you more Tmux tricks!"
### Launch Github Codespaces **1** Go to this [Github repository](https://github.com/cyverse-learning-materials/foss_conda_lesson){target=_blank} and Fork it (i.e., make a copy of it in your Github account).
### Installing Conda **If you are using Codespaces, Conda is already installed.** When you download and install Conda it comes in two different flavors: [Miniconda](https://docs.anaconda.com/free/miniconda/){target=_blank} - lightweight (500 mb) program that includes *Conda*, the environment and package manager, as well as a recent version of the Standard Python Library. [Anaconda](https://anaconda.org/){target=_blank} - a larger (2.5GB) program that includes *Conda* and many more python libraries pre-installed (in Conda base environment), as well as graphical user interface, acccess to jupyter notebooks, and support for easily integrating the R language.
[Taken from *Getting Started with Conda, Medium*](https://medium.com/hydroinformatics/getting-started-with-conda-environment-332182d1e937){target=_blank}.
### Environment Management with Conda When you start a Codespaces terminal, the prompt will look something like this: ``` @jeffgillan ➜ /workspaces/foss_conda_lesson (main) $ ```
Type the following command to see the current conda environment. ``` conda info ```
Initialize conda by running the following commands. ``` conda init exec $SHELL ```
View the list of conda environments. There should only be one environment called `base`. ``` conda env list ```
View the software installed in the base directory. Notice the version of Python. ``` conda list ```
Create our own custom environment (type `y` when prompted). ``` conda create --name myenv ```
Activate your new environment with ``` conda activate myenv ```
You will notice that the prompt changed to `(myenv)`
View the software that is installed in your new custom environment. It should be empty! ``` conda list ```
### Package management with Conda Within your new custom environment (ie, myenv) download and install a specific version of python. This may take a few minutes to complete. ``` conda install python=3.9 ```
View the new software that has been installed. Notice the version of Python is now 3.9. while the base is 3.12 ``` conda list ```
Install Salmon (genomics software) using Conda ``` conda install -c bioconda salmon ```
!!! info "Conda channels" Conda operates through **channels**, specififc repositories where packages are stored. Specific packages sometimes may appear in multiple channels, however it is always helpful to specify a channel with the `-c` flag.
### Share and Reproduce a Conda Environment Export all of the software in your custom environment to a file ``` conda env export --no-builds > myenv.yml ```
Let's view the contents of the .yml file. It should contain all the software you installed in the environment. This `myenv.yml` file can be shared with a colleague so they can reproduce the same environment on their computer. ``` nano myenv.yml ```
Reproduce someone else's environment with `mandelbrot.yml` environment file located in the repository. ``` conda env create --file mandelbrot.yml ```
Activate the environment to use the software installed in the environment. ``` conda activate mandelbrot ```
Look at the software installed in the environment. ``` conda list ```
Run a python script that generates a Mandelbrot set ``` python3 mandelbrot.py ```
Adapted from [CyVerse FOSS](https://github.com/CyVerse-learning-materials/foss/blob/86d4cc384976a59bd19cce2a6fdcf994b19051bd/docs/06_soft_env.md){target=_blank} (last source update 2025-08-26), CC BY 4.0. Spotted a problem? [Open an issue](https://github.com/UNM-CARC/foss/issues){target=_blank}.
---8<--- https://unm-carc.github.io/foss/lessons/07-containers/ --- title: "Reproducibility III: Containers" description: "Why containers matter for reproducible research, core Docker commands, running an example container, and writing, building, and pushing a Dockerfile." type: Lesson tags: - Docker - Containers - Reproducibility generated: by: "claude/opus-5" at: "2026-09-11T00:00:00Z" sources: - id: cyverse-foss resource: "https://github.com/CyVerse-learning-materials/foss/blob/86d4cc384976a59bd19cce2a6fdcf994b19051bd/docs/07_containers.md" title: "CyVerse FOSS: docs/07_containers.md" author: "team:cyverse" last_modified: "2025-10-14T10:43:02-07:00" --- # Reproducibility III: Containers !!! success "Learning Objectives" After this lesson, you should be able to: - Explain what containers are used for in reproducible research contexts - Search for and run a Docker container locally or on a remote system - Understand how version control and data can be used inside a container - Understand the Dockerfile structure and fields - Build, execute and push your own Docker image ## Reproducible Computer Code Sharing your scientific analysis code with your colleagues is an essential pillar of Open Science that will help push your field forward. There are, however, technical challenges that may prevent your colleagues from effectively running your code on their computers. In the previous lesson [Reproducibility II](https://unm-carc.github.io/foss/lessons/06-software-environments/) we described how [computing environments](https://unm-carc.github.io/foss/lessons/06-software-environments/#computing-environment) can lead the way of reproducibility by creating a "recipe" for replicating a computing environment. There is another method to overcoming the replicability obstacle. Instead of recreating a computing environment, **why not package up the code _and_ all of the software and send it to your colleague?** ## What *are* containers? A [container](https://www.docker.com/resources/what-container/){target=_blank} is a standard unit of software that packages up code and all its dependencies so the application runs quickly and reliably from one computing environment to another - Container images are a lightweight, standalone, executable package of software that includes everything needed to run an application: code, runtime, system tools, system libraries and settings - Each of these elements are specifically versioned and do not change - The recipient does not need to *install* the software in the traditional senseA useful analogy is to think of software containers as shipping containers. It allows us move cargo (software) around the world in standard way. The shipping container can be offloading and executed anywhere, as long the destination has a shipping port (i.e., Docker)
Containers are similar to virtual machines (VMs), but are smaller and easier to share. A big distinction between Containers and VMs is what is within each environment: VMs require the OS to be present within the image, whilst containers rely on the host OS and the container engine (e.g., Docker Engine).
## Containers for Reproducible Science Software containers, such as those managed by Docker or Apptainer (formerly Singularity), are incredibly useful for reproducible science for several reasons: #### Environment Consistency: Containers encapsulate the software environment, ensuring that the same versions of software, libraries, and dependencies are used every time, reducing the "it works on my machine" problem. - **Ease of Sharing**: - Containers can be easily shared with other researchers, allowing them to replicate the exact software environment used in a study. - **Platform Independence**: - Containers can run on different operating systems and cloud platforms, allowing for consistency across different hardware and infrastructure. - **Version Control**: - Containers can be versioned, making it easy to keep track of changes in the software environment over time. - **Scalability**: - Containers can be easily scaled and deployed on cloud infrastructure, allowing for reproducible science at scale. - **Isolation**: - Containers isolate the software environment from the host system, reducing the risk of conflicts with other software and ensuring a clean and controlled environment.
The most common container software is [:material-docker: Docker](https://www.docker.com/){target=_blank}, which is a platform for developers and sysadmins to develop, deploy, and run applications with containers. [Apptainer](https://apptainer.org/docs/user/main/){target=_blank} (formerly, Singularity), is another popular container engine, which allows you to deploy containers on HPC clusters. !!! tip "UNM CARC users" CARC clusters run containers with Singularity/Apptainer rather than Docker. Build and push your image with Docker (on your own computer or in Codespaces), then pull and run it on CARC. See [Singularity on CARC](https://carc.unm.edu/docs/software/singularity/){target=_blank}, which also shows how to find the available versions with `module avail singularity`. [DockerHub](https://hub.docker.com/){target=_blank} is the world's largest respository of container images. Think of it as the 'Github' of container images. It facilitates collaboration amongst developers and allows you to share your container images with the world. Dockerhub allows users to maintain different versions of container images. !!! warning "While Docker allows you to quickly run software from other people, it may not work across every platform.
There are different CPU architectures (`arm`, `amd64`, `x64`, `x86`) deployed across cloud, computer workstations, laptops, and cellular phones. Docker containers and their software can be cross-compiled across architectures, but this must be done by the creators."
## Introduction to :material-docker: Docker
containing code, libraries, dependencides
and files needed for the application to run | A runtime instance of the Docker image | | **When** | Created through a Dockerfile | Runtime executed after Image is created | | **Why** | Reproducibility and consistency! | Reproducibility and consistency! | | **Where** | Stored in an online registry (e.g., [Docker Hub](https://hub.docker.com/){target=_blank}) | Executed on your machine |
--- ## Fundamental Docker Commands :octicons-terminal-16: Docker commands in the terminal use the prefix `docker`. !!! note "For every command listed, the correct execution of the commands through the command line is by using `docker` in front of the command: for example `docker help` or `docker search`. Thus, every :material-docker: = `docker`." !!! warning "To follow along, please start a Codespace in the following repository: **https://github.com/jeffgillan/r-script-containerized**. We are going to be using this repository throughout today, to learn docker commands, run docker containers and building our own container." ### :material-docker: help Like many other command line applications the most helpful flag is the `help` command which can be used with the Management Commands: ``` $ docker $ docker --help ``` ### :material-docker: search We talk about the concept of [Docker Registries](https://unm-carc.github.io/container-camp/docker/registry/){target=_blank} in the next section, but you can search the public list of registeries by using the `docker search` command to find public containers on the Official [Docker Hub Registry](https://hub.docker.com){target=_blank}: ``` $ docker search ``` ### :material-docker: pull Go to the [Docker Hub](https://hub.docker.com){target=_blank} and type `hello-world` in the search bar at the top of the page. Click on the 'tag' tab to see all the available 'hello-world' images. Click the 'copy' icon at the right to copy the `docker pull` command, or type it into your terminal: ``` $ docker pull hello-world ``` !!! note If you leave off the `:` and the tag name, it will by default pull the `latest` image ``` $ docker pull hello-world Using default tag: latest latest: Pulling from library/hello-world 2db29710123e: Pull complete Digest: sha256:bfea6278a0a267fad2634554f4f0c6f31981eea41c553fdf5a83e95a41d40c38 Status: Downloaded newer image for hello-world:latest docker.io/library/hello-world:latest ``` Now try to list the files in your current working directory: ``` $ ls -l ``` ??? question "Where is the image you just pulled?" Docker saves container images to the Docker directory (where Docker is installed). You won't ever see them in your working directory. Use 'docker images' to see all the images on your computer: ``` $ docker images ``` ### :material-docker: run The single most common command that you'll use with Docker is `docker run` ([see official help manual](https://docs.docker.com/engine/reference/commandline/run/){target=_blank} for more details). ``` $ docker run hello-world:latest ``` In the demo above, you used the `docker pull` command to download the `hello-world:latest` image. What about if you run a container that you haven't downloaded? ``` $ docker run alpine:latest ``` When you executed the command `docker run alpine:latest`, Docker first looked for the cached image locally, but did not find it, it then ran a `docker pull` behind the scenes to download the `alpine:latest` image and then execute your command. ### :material-docker: images You can now use the `docker images` command to see a list of all the cached images on your system: ``` $ docker images REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE alpine latest c51f86c28340 4 weeks ago 1.109 MB hello-world latest 690ed74de00f 5 months ago 960 B ``` ??? info "Inspecting your containers" To find out more about a Docker images, run `docker inspect hello-world:latest` ### :material-docker: ps Now it's time to see the `docker ps` command which shows you all containers that are currently running on your machine. ``` docker ps ``` Since no containers are running, you see a blank line. ``` $ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES ``` Let's try a more useful variant: `docker ps --all` ``` $ docker ps --all CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES a5eab9243a15 hello-world "/hello" 5 seconds ago Exited (0) 3 seconds ago loving_mcnulty 3bb4e26d2e0c alpine:latest "/bin/sh" 17 seconds ago Exited (0) 16 seconds ago objective_meninsky 192ffdf0cbae opensearchproject/opensearch-dashboards:latest "./opensearch-dashbo…" 3 days ago Exited (0) 3 days ago opensearch-dashboards a10d47d3b6de opensearchproject/opensearch:latest "./opensearch-docker…" 3 days ago Exited (0) 3 days ago opensearch-node1 ``` What you see above is a list of all containers that you have run. Notice that the `STATUS` column shows the current condition of the container: running, or as shown in the example, when the container was exited. ### :material-docker: stop The `stop` command is used for containers that are actively running, either as a foreground process or as a detached background one. You can find a running container using the `docker ps` command. ### :material-docker: rm You can remove individual stopped containers by using the `rm` command. Use the `ps` command to see all your stopped contiainers: ``` @user ➜ /workspaces $ docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 03542eaac9dc hello-world "/hello" About a minute ago Exited (0) About a minute ago unruffled_nobel ``` Use the first few unique alphanumerics in the CONTAINER ID to remove the stopped container: ``` @user ➜ /workspaces (mkdocs ✗) $ docker rm 0354 0354 ``` Check to see that the container is gone using `ps -a` a second time (`-a` is shorthand for `--all`; the full command is `docker ps -a` or `docker ps --all`). ### :material-docker: rmi The `rmi` command is similar to `rm` but it will remove the cached images. Used in combination with `docker images` or `docker system df` you can clean up a full cache ``` docker rmi ``` ``` @user ➜ /workspaces/ (mkdocs ✗) $ docker images REPOSITORY TAG IMAGE ID CREATED SIZE opendronemap/webodm_webapp latest e075d13aaf35 21 hours ago 1.62GB redis latest a10f849e1540 5 days ago 117MB opendronemap/nodeodm latest b4c50165f838 6 days ago 1.77GB hello-world latest feb5d9fea6a5 7 months ago 13.3kB opendronemap/webodm_db latest e40c0f274bba 8 months ago 695MB @user ➜ /workspaces (mkdocs ✗) $ docker rmi hello-world Untagged: hello-world:latest Untagged: hello-world@sha256:10d7d58d5ebd2a652f4d93fdd86da8f265f5318c6a73cc5b6a9798ff6d2b2e67 Deleted: sha256:feb5d9fea6a5e9606aa995e879d862b825965ba48de054caab5ef356dc6b3412 Deleted: sha256:e07ee1baac5fae6a26f30cabfe54a36d3402f96afda318fe0a96cec4ca393359 @user ➜ /workspaces (mkdocs ✗) $ ``` ### :material-docker: system The `system` command can be used to view information about containers on your cache, you can view your total disk usage, view events or info. You can also use it to `prune` unused data and image layers. To remove all cached layers, images, and data you can use the `-af` flag for `all` and `force` ``` docker system prune -af ``` ### :material-docker: tag By default an image will recieve the tag `latest` when it is not specified during the `docker build` Image names and tags can be created or changed using the `docker tag` command. ``` docker tag imagename:oldtag imagename:newtag ``` You can also change the registry name used in the tag: ``` docker tag docker.io/username/imagename:oldtag harbor.cyverse.org/project/imagename:newtag ``` The cached image laters will not change their `sha256` and both image tags will still be present after the new tag name is generated. --- ## Run an Example Container The example container is located in the following Github repository **https://github.com/jeffgillan/r-script-containerized**. The docker container launches Rstudio where a user can run an R script to convert a point cloud to a canopy height model (CHM).
#### 1. Launch Codespaces from the `r-script-containerized` repository
#### 2. On the codespaces terminal, run the following command to run the docker container: ``` docker run --rm -ti -e DISABLE_AUTH=true -v $(pwd):/home/rstudio/data -p 8787:8787 jeffgillan/pointcloud_to_chm:amd64 ``` ??? tip "Dissecting the Docker Run Command" Within the command we are doing the following: | Flag/Option | Explanation | | --- | --- | |`--rm` | Automatically remove the container when it exits | |`-ti` | Allocate a pseudo-TTY (terminal) connected to the container’s stdin. It gives you an interactive terminal session in the container, allowing you to run commands in the container just as you would in a regular terminal window. | | `-e DISABLE_AUTH=true` | Disable authentication for the Rstudio server | | `-v $(pwd):/home/rstudio/data` | Mount the current working directory to the `/home/rstudio/data directory` in the container. This allows us to access the data on our local machine (in this case Codespaces) from within the container. The directory is where you should have pointcloud `.laz` files. | |`-p 8787:8787` | Expose port 8787 on the container to port 8787 on the host machine. This allows us to access the Rstudio server from our web browser. | | `jeffgillan/pointcloud_to_chm:amd64` | The name and version number of the docker image we want to run. In this case, we are running the `pointcloud_to_chm` image from the `jeffgillan` repository in Docker Hub. |
#### 3. View Rstudio in a Browser Open a new browser tab and navigate to `localhost:8787` to access the Rstudio server running in the container. In Codespaces you can click on _Ports_ and then _Open Browser_ to access the Rstudio server.
#### 4. Run the Script Open the script `pointcloud_to_DTM.R` Highlight all of the code and click *Run*
Upon completing the code, you should have a 2D plot of the tree
---
## Building Docker Images
!!! tip "What is a Dockerfile?" A Dockerfile is a text file that contains a list of commands, known as **instructions** used by Docker to build an image. **If a *Docker Image* is a cake, the *Dockerfile* is the recipe!** ``` # Tells the Dockerfile which image to start from FROM rocker/geospatial:4.2.3 # Sets the initial working directory inside the container WORKDIR /home/rstudio # Copies the R script file from your localdirectory to the container COPY pointcloud_to_DTM.R . # Installs an R package RUN R -e "install.packages('RCSF', dependencies=TRUE, repos='http://cran.rstudio.com/')" #Declare the port Rstudio will run on EXPOSE 8787 # Starts the Rstudio server CMD ["/init"] ```
#### Dockerfile Statements ??? abstract "ARG" The only command that can come before a `FROM` statement is `ARG` `ARG` can be used to set arguments for later in the build, e.g., ``` ARG VERSION=latest FROM ubuntu:$VERSION ``` ??? abstract "FROM" A valid `Dockerfile` must start with a `FROM` statement which initializes a new build stage and sets the **base image** for subsequent layers. We’ll start by specifying our base image, using the FROM statement ``` FROM ubuntu:latest ``` If you are building on an `arm64` or Windows system you can also give the optional `--platform` flag, e.g., ``` FROM --platform=linux/amd64 ubuntu:latest ``` ??? question "When to use a multi-stage build pattern?" Docker has the ability to build container images from one image, and run that "builder" image from a second "base" image, in what is called a "builder pattern". Build patterns are useful if you're compiling code from (proprietary) source code and only want to feature the binary code as an executed function in the container at run time. Build patterns can greatly reduce the size of your container. You can use multiple `FROM` commands as build stages. The `AS` statement follows the `image:tag` as a psuedo argument. ``` # build stage FROM golang:latest AS build-env WORKDIR /go/src/app ADD . /go/src/app RUN go mod init RUN cd /go/src/app && go build -o hello # final stage FROM alpine:latest WORKDIR /app COPY --from=build-env /go/src/app /app/ ENTRYPOINT ./hello ``` ??? abstract "LABEL" You can create labels which are then tagged as JSON metadata to the image ``` LABEL author="your-name" LABEL email="your@email-address" LABEL version="v1.0" LABEL description="This is your first Dockerfile" LABEL date_created="2022-05-13" ``` You can also add labels to a container when it is run: ``` $ docker run --label description="this label came later" ubuntu:latest $ docker ps -a $ docker inspect ### ``` ??? abstract "RUN" Different than the `docker run` command is the `RUN` build function. `RUN` is used to create new layers atop the "base image" Here, we are going to install some games and programs into our base image: ``` RUN apt-get update && apt-get install -y fortune cowsay lolcat ``` Here we've installed `fortune` `cowsay` and `lolcat` as new programs into our base image. !!! warning "Best practices for building new layers" Ever time you use the `RUN` command it is a good idea to use the `apt-get update` or `apt update` command to make sure your layer is up-to-date. This can become a problem though if you have a very large container with a large number of `RUN` layers. ??? abstract "ENV" In our new container, we need to change and update some of the environment flags. We can do this using the `ENV` command ``` ENV PATH=/usr/games:${PATH} ENV LC_ALL=C ``` Here we are adding the `/usr/games` directory to the `PATH` so that when we run the new container it will find our newly installed game commands We are also updating the "[locales](https://www.tecmint.com/set-system-locales-in-linux/){target=_blank}" to set the language of the container. ??? abstract "COPY" The `COPY` command will copy files from the directory where `Dockerfile` is kept into the new image. You must specify where to copy the files or directories ``` COPY . /app ``` ??? question "When to use `COPY` vs `ADD`" `COPY` is more basic and is good for files `ADD` has some extra features like `.tar` extraction and URL support ??? abstract "CMD" The `CMD` command is used to run software in your image. In general use the ["command"] syntax: ``` CMD ["executable", "parameter1", "parameter2"] ``` ??? abstract "ENTRYPOINT" ENTRYPOINT works similarly to `CMD` but is designed to allow you to run your container as an executable. ``` ENTRYPOINT fortune | cowsay | lolcat ``` The default `ENTRYPOINT` of most images is `/bin/sh -c` which executes a `shell` command. `ENTRYPOINT` supports both the `ENTRYPOINT ["command"]` syntax and the `ENTRYPOINT command` syntax ??? tip "What is an *entrypoint*?" An entrypoint is the initial command(s) executed upon starting the Docker container. It is listed in the `Dockerfile` as `ENTRYPOINT` and can take 2 forms: as commands followed by parameters (`ENTRYPOINT command param1 param2`) or as an executable (`ENTRYPOINT [“executable”, “param1”, “param2”]`) ??? question "What is the difference in the `ENTRYPOINT` and `CMD`" The CMD instruction is used to define what is execute when the container is run. The ENTRYPOINT instruction cannot be overridden, instead it is appended to when a new command is given to the `docker run container:tag new-cmd` statement the executable is defined with ENTRYPOINT, while CMD specifies the default parameter ??? abstract "USER" Most containers are run as `root` meaning that they have super-user privileges within themselves Typically, a new user is necessary in a container that is used interactively or may be run on a remote system. During the build of the container, you can create a new user with the `adduser` command and set up a `/home/` directory for them. This new user would have something like 1000:1000 `uid:gid` permissions without `sudo` privileges. As a last step, the container is run as the new `USER`, e.g., ``` ARG VERSION=18.04 FROM ubuntu:$VERSION RUN useradd ubuntu && \ chown -R ubuntu:ubuntu /home/ubuntu USER ubuntu ``` ??? abstract "EXPOSE" You can open [ports](https://unm-carc.github.io/container-camp/docker/intro/#understanding-ports){target=_blank} using the `EXPOSE` command. ``` EXPOSE 8888 ``` The above command will expose port 8888. !!! note Running multiple containers using the same port is not trivial and would require the usage of a web server such as [NGINX](https://www.nginx.com/){target=_blank}. However, you can have multiple containers interact with each other using [Docker Compose](https://docs.docker.com/compose/){target=_blank}.
### Summary of Instructions | Instruction Command | Description | | :---: | --- | | `ARG` | Sets environmental variables during image building | | `FROM` | Instructs to use a specific Docker image | | `LABEL` | Adds metadata to the image | | `RUN` | Executes a specific command | | `ENV` | Sets environmental variables | | `COPY` | Copies a file from a specified location to the image | | `CMD` | Sets a command to be executed when running a container | | `ENTRYPOINT` | Configures and run a container as an executable | | `USER` | Used to set User specific information | | `EXPOSE` | exposes a specific port |
---
### Building the Image Once the Dockerfile has been created, we can build the image using the `docker build` command. At the codespaces terminal, run the following command: ``` docker build -t
--- ### Executable Images ``` _____________________________ < Avoid reality at all costs. > ----------------------------- \ ^__^ \ (oo)\_______ (__)\ )\/\ ||----w | || || ``` Thus far we have worked using interactive containers such as RStudio. Not all containers are interactive, many (if not most of) are executables: containerized software that can take the raw input and output data without requiring further work from the user. Executable containers are usually called with ``` docker run
--- --- ### Exercises and Additional Dockerfiles !!! success "" We have a number of pre-written Dockerfiles that you can access in our [Intro 2 Docker Repository](https://github.com/cyverse-education/intro2docker){target=_blank}. It contains: - **Alpine**: a minimal Linux distribution - **Cowsay**: will output wise messages when executed - **Jupyter/minimal-notebook**: Jupyter Notebook accessible from your browser - **rstudio/verse**: RStudio containing the Tidyverse data science packages - **Ubuntu** Clone our example repository, or work directly on the repository using Codespaces. Clone command: ``` $ cd /workspaces $ git clone https://github.com/cyverse-education/intro2docker $ cd intro2docker/ ```
Adapted from [CyVerse FOSS](https://github.com/CyVerse-learning-materials/foss/blob/86d4cc384976a59bd19cce2a6fdcf994b19051bd/docs/07_containers.md){target=_blank} (last source update 2025-10-14), CC BY 4.0. Spotted a problem? [Open an issue](https://github.com/UNM-CARC/foss/issues){target=_blank}.
---8<--- https://unm-carc.github.io/foss/lessons/08-hpc/ --- title: "Remote Computing: HPC" description: "Logging in to a high-performance computing cluster, submitting Slurm batch and interactive jobs, loading modules, and running Apptainer containers." type: Lesson tags: - HPC - Slurm - Apptainer - Remote Computing generated: by: "claude/opus-5" at: "2026-09-11T00:00:00Z" sources: - id: cyverse-foss resource: "https://github.com/CyVerse-learning-materials/foss/blob/86d4cc384976a59bd19cce2a6fdcf994b19051bd/docs/09_HPC.md" title: "CyVerse FOSS: docs/09_HPC.md" author: "team:cyverse" last_modified: "2025-10-21T10:26:16-07:00" --- # Remote Computing: HPC
and
`#SBATCH --ntasks=N` | ntasks specifies the number of tasks (or processes) the job will run. By default, you will be allocated one CPU/task. This can be increased by including the additional directive --cpus-per-task. | | `#SBATCH --mem=Ngb` | Select N gb of memory per node. If "gb" is not included, this value defaults to MB. | After setting your directives, you can instruct the HPC to do what you require similar to a bash script. Here's an example of a batch job: ``` #!/bin/bash #SBATCH --job-name=blast_job # Job name #SBATCH --partition=standard # Sets the job priority to standard #SBATCH --nodes=1 # Number of nodes #SBATCH --ntasks=1 # Number of tasks (processes) per node #SBATCH --cpus-per-task=4 # Number of CPU cores per task #SBATCH --mem=8G # Memory per node (in this case, 8GB) #SBATCH --time=02:00:00 # Time limit (HH:MM:SS) # Load necessary modules module load blast/2.12.0 # Load BLAST module (adjust version as needed) # Change to the directory where the job will run cd $SLURM_SUBMIT_DIR # Define input and output files query_file="query.fasta" # Input query file (FASTA format) database_file="database.fasta" # BLAST database file (FASTA format) output_file="blast_results.out" # Output file for BLAST results # Run BLAST command blastp -query $query_file -db $database_file -out $output_file -evalue 0.001 -num_threads $SLURM_CPUS_PER_TASK ``` !!! tip "Submitting a Batch Script" - To submit jobs you need to use `sbatch`, such as `sbatch script.slurm` - To cancel your job you do `scancel`, such as `scancel $JOBID` or `scancel -u $NETID` This will submit your job to the queue. Execution will depend on your submission type (partition). #### Launching an Interactive Node An **interactive node**, unlike batch jobs which are run asynchronously, allows immediate access to compute. Similar to batch jobs, interactive nodes are submitted to the queue, but once available, you will receive a prompt for a node with the selected resources. Read more on how to launch interactive jobs in [the official documentation](https://uarizona.atlassian.net/wiki/spaces/UAHPC/pages/75989825/Interactive+Jobs){target=_blank}. !!! tip "The Quick and Dirty" Don't need a lot of resources and just want access to the compute? Just type `interactive`. Disclaimer: you may require to wait longer as your job is going to fall in the `windfall` queue. Following are a list of useful flags (options) for setting up the interactive node. |Flag|Default value| Description| Example| |-|-|-|-| | `-n` | 1 | Number of CPUs requested per node | interactive -n 8 | | `-m` | 4GB | Memory per CPU | interactive -m 5GB | | `-a` | none | Account (group) to charge | interactive -a datalab | | `--partition=` | windfall | Partition to determine CPU time charges and is set to windfall when no account is specified, and is **set to standard when an account is provided.** | interactive --partition=windfall | | `-t` | 01:00:00 | Time allocated to session. | interactive -t 08:00:00 | | `-N` | 1 | Number of nodes. | There is no reason to exceed 1 node unless the number of CPUs requested is greater than the number of CPUs per node on a given cluster. | (in el gato, where number of CPU per node is 16 max ) interactive -n 32 -N 2 | An example for an interactive node is: ``` interactive -n 8 -m 16GB -a datalab -t 02:00:00 ``` The above example will request an interactive node with 8 cores, 16GB RAM, "charging" the datalab, running for 2 hours. Try it! !!! note "Modules" There are 100s of tools installed on the HPC, few of which are available on the login screen. These tools are available only during a batch job submission or within interactive jobs. To see what tools are already running, or which are available, you will need to use the `module` command. !!! tip "Helpful `module` commands" |Command|Purpose| |-|-| |`module list`| Lists loaded modules| |`module avail`| Lists available modules| |`module spider`| Lists ALL modules| |`module load`| Loads a module| |`module help`| Help command!| --- ## Singularity/Apptainer In 2021, the Sylabs, the developers behind the original Singularity, made a fork of the original project and renamed it [SingularityCE (Community Edition)](https://github.com/sylabs/singularity){target=_blank}. This would allow for the SingularityCE to be compliat with FOSS and allowing for the community to contribute to the builds. The Singularity team then joined the [Linux Foundation](https://en.wikipedia.org/wiki/Linux_Foundation){target=_blank} and decided to rename their effor to [**Apptainer**](https://github.com/apptainer/apptainer){target=_blank}. The technology behind Singularity/Apptainer is similar to the one of Docker, but, as mentioned before, it was created with the HPC in mind, and therefore bypasses the requirement of sudo. !!! note Until now we have used Singularity/Apptainer to refer to the same software. Onwards, you can decide whether to use Singularity *OR* Apptainer; in order to keep up with the latest release, we are going to be executing apptainer commands. Apptainer also installs a `singularity` command as an alias, so older instructions that use `singularity` still work. ??? info "Docker vs SingularityCE & Apptainer in the blink of an eye" **:material-open-source-initiative: Apptainer and SingularityCE are 100% compatible with Docker but they do have some distinct differences** **:material-docker: Docker** :octicons-container-24: Docker containers run as `root` - This privilege is almost never supported by administrators of High Performance Computing (HPC) centers. Meaning Docker is not, and will likely never be, installed natively on your HPC cluster. :octicons-container-24: uses compressed layers to create one image **:material-open-source-initiative: SingularityCE & Apptainer**: :octicons-container-24: Same user and group identity inside as outside the container :octicons-container-24: User only has `root` privileges if elevated with `sudo` when the container is run :octicons-container-24: Can run and modify any existing Docker image - These key differences allow Singularity to be installed on most HPC centers. Because you can run virtually all Docker containers in Singularity, you can effectively run Docker on an HPC. ### General Executable Commands Resources: - https://unm-carc.github.io/container-camp/singularity/intro/ - https://unm-carc.github.io/container-camp/singularity/hpc/ - https://unm-carc.github.io/container-camp/singularity/advanced/ Apptainer’s [command line interface](https://apptainer.org/docs/user/main/cli.html){target=_blank} allows you to build and interact with containers transparently. You can run programs inside a container as if they were running on your host system. You can easily redirect IO, use pipes, pass arguments, and access files, sockets, and ports on the host system from within a container. #### :octicons-container-24: help The `help` command gives an overview of Apptainer options and subcommands as follows: ``` $ apptainer help pull Pull an image from a URI Usage: apptainer pull [pull options...] [output file]
Adapted from [CyVerse FOSS](https://github.com/CyVerse-learning-materials/foss/blob/86d4cc384976a59bd19cce2a6fdcf994b19051bd/docs/09_HPC.md){target=_blank} (last source update 2025-10-21), CC BY 4.0. Spotted a problem? [Open an issue](https://github.com/UNM-CARC/foss/issues){target=_blank}.
---8<--- https://unm-carc.github.io/foss/lessons/09-cyverse/ --- title: "Remote Computing: CyVerse" description: "An overview of the CyVerse cloud platform: the Discovery Environment, Data Store, Data Commons, analysis apps, and the analysis dashboard." type: Lesson tags: - CyVerse - Cloud Computing - Data Store - Remote Computing generated: by: "claude/opus-5" at: "2026-09-11T00:00:00Z" sources: - id: cyverse-foss resource: "https://github.com/CyVerse-learning-materials/foss/blob/86d4cc384976a59bd19cce2a6fdcf994b19051bd/docs/08_remote_computing_cyverse.md" title: "CyVerse FOSS: docs/08_remote_computing_cyverse.md" author: "team:cyverse" last_modified: "2025-10-27T14:59:39-07:00" --- # Remote Computing: CyVerse !!! warning "This page is a summary of what today's lesson is going to cover. For the information covered, we are going to rely on the CyVerse Learning documentation: https://unm-carc.github.io/cyverse/."!!! success "Advantages of Cloud Computing" * With cloud computing, you can avoid the upfront cost and complexity of owning and maintaining your own IT infrastructure * Cloud computing allows groups or individuals to scale up (or down) their operations quickly as their computing needs change * Cloud computing allows users to access their data and applications from anywhere, on any device, at any time * Cloud empowers colleagues to work directly together on the same data, models, and applications * Easily share your work with the world
## CyVerse Discovery Environment
The [CyVerse Discovery Environment](https://de.cyverse.org){target=_blank} is where users can store and share data as well as run analysis applications with the click of a button. It is backed by robust computing resources that can help scale your analysis beyond what is possible on your laptop.
## Data Storage and Sharing
Cyverse Data Store is the ideal cloud storage to host your large (or small) datasets, share data with colleagues, and meet publication/grant archival requirements.
* Cyverse Data Store is object cloud storage similar to Azure Blob, or Amazon S3
* Pro account has a 5TB limit
* [Moving data in or out of the Data Store](https://unm-carc.github.io/cyverse/data-store/){target=_blank} can be done through website or multiple command line tools
* Share your data with your colleagues and world with a URL
* Data can be public/private, shared with anyone, set permission levels
* Never lose your data due to hardware failure
[CyVerse Data Commons](https://datacommons.cyverse.org/){target=_blank} is our public facing data storage interface. It enables you to share data with people outside of CyVerse.
_**Community Released:**_ A folder of data you want to share with colleagues, stakeholders, or the entire world. You control read/write/own permissions and make it public or private.
_**Curated:**_ Data that is tied to a peer-reviewed publication and needs permanent archival. You can apply many types of metadata templates your data as well as receive a permanent DOI.
## Apps
The CyVerse Discovery Environment has a large number of analysis applications that can be run on your data. These applications are pre-installed and ready to use with a few clicks. You get to choose the amount of computing resources you need to run your analysis.
Apps come in two basics flavors:
* Executable Apps: Run a script or a series of scripts on your data
* Interactive Apps: Launch software such as Jupyter notebooks, RStudio, QGIS, VScode, and more
!!! tip "Important Notes"
* When you launch an app in Cyverse DE, you are actually launching containers such as Docker
* When you close the app, the container is destroyed but your data can be saved in the Data Store
## Analysis Dashboard
The Analysis Dashboard is where you can monitor the status of Apps you have launched
* Stop and Relaunch Apps
* Extend the time of an App (beyond 72 hours)
* View you history of Apps launched
!!! warning "Attention"
If you are not using an app, please shut it down!
Adapted from [CyVerse FOSS](https://github.com/CyVerse-learning-materials/foss/blob/86d4cc384976a59bd19cce2a6fdcf994b19051bd/docs/08_remote_computing_cyverse.md){target=_blank} (last source update 2025-10-27), CC BY 4.0. Spotted a problem? [Open an issue](https://github.com/UNM-CARC/foss/issues){target=_blank}.
---8<--- https://unm-carc.github.io/foss/lessons/extra-nextflow/ --- title: "Extra: Workflows with Nextflow" description: "A comparison of the Nextflow and Snakemake workflow managers and a hands-on RNA-seq tutorial that runs a Nextflow pipeline in a Conda environment." type: Lesson tags: - Nextflow - Snakemake - Workflows - Conda generated: by: "claude/opus-5" at: "2026-09-11T00:00:00Z" sources: - id: cyverse-foss resource: "https://github.com/CyVerse-learning-materials/foss/blob/86d4cc384976a59bd19cce2a6fdcf994b19051bd/docs/nextflow.md" title: "CyVerse FOSS: docs/nextflow.md" author: "team:cyverse" last_modified: "2024-07-02T16:37:25-07:00" --- # Extra: Workflows with Nextflow ## Automation: Workflow Management with Nextflow and Snakemake Workflow management is key to automation: it helps with bridging the gap between software, creating useful pipelines that help extracting information from data. Two examples of workflow management system popular (but not restricted to!) within the field of Bioinformatics are [Nextflow](https://www.nextflow.io/docs/latest/index.html){target=_blank} and [Snakemake](https://snakemake.readthedocs.io/en/stable/){target=_blank}. Both workflow managers are powerful tools that tackle reproducibility and scalability, however the two have some key differences: | Feature | Snakemake | Nextflow | |---|---|---| | Language style | Python-based DSL* |Groovy-based DSL* | | Execution Model | Rule-based | Dataflow-driven | | Parallelization and Resource Management |Implicit parallelization, limited resource management | Explicit parallelization, more flexible resource management | | Integration | seamless with Python | seamless with Java** | \* = Domain Specific Langauge** = No, it does not mean you need to learn Java Although there are differences between the two workflow managers, there are a number of features which both share: - Available on all OS - Well integrated with Conda - Support the use of Containers - Deployable on HPC systems - Integrated error reporting - Well established communities*** *** = It should be noted that Nextflow has a [community driven repository, nf-core](https://nf-co.re/){target=_blank}, where people can upload their Nextflow pipelines for others to use. Power to the people! Both are incredibly powerful when it comes to creating pipelines, ultimately the choice on which to use depends on **you** (and the goal of your project). Here are a couple of script examples for each workflow manager. Although different, the examples try to 1. Download a genome 2. Align reads using BWA ([Burrows-Wheeler Aligner](https://bio-bwa.sourceforge.net/){target=_blank} tool) 3. Gather aignment statistics using [Samtools](https://www.htslib.org/){target=_blank} (Sequence Alignment/Map) **Snakemake:** ``` # Snakefile # Define rule for downloading reference genome rule download_reference_genome: output: "reference_genome.fa" shell: "wget http://example.com/reference_genome.fa" # Define rule for aligning reads to the reference genome using BWA rule align_reads: input: "reference_genome.fa", fastq="data/{sample}.fastq" output: "aligned/{sample}.bam" shell: "bwa mem {input.reference_genome} {input.fastq} | samtools view -b - > {output}" # Define rule to aggregate alignment statistics rule aggregate_stats: input: expand("aligned/{sample}.bam", sample=config['samples']) output: "alignment_stats.txt" shell: "samtools flagstat {input} > {output}" # Define samples configfile: "config.yaml" ``` Notice how in the Snakefile above, there are a set number of rules that define input, output and commands. **Nextflow:** ``` // main.nf params.samples = ['sample1', 'sample2', 'sample3'] // Define process to download reference genome process download_reference_genome { output: file('reference_genome.fa') script: """ wget http://example.com/reference_genome.fa -O reference_genome.fa """ } // Define process to align reads using BWA process align_reads { input: file(reference_genome), file(fastq) from fastqs output: file("aligned/${sample}.bam") script: """ bwa mem ${reference_genome} ${fastq} | samtools view -b - > aligned/${sample}.bam """ } // Define process to aggregate alignment statistics process aggregate_stats { input: set file(aligned_bam) from aligned_bams output: file("alignment_stats.txt") script: """ samtools flagstat ${aligned_bam} > alignment_stats.txt """ } // Define workflow execution workflow { // Define input data Channel.fromPath("data/*.fastq").set { fastqs } // Execute processes in parallel download_reference_genome() align_reads(fastqs) aggregate_stats() } ``` The Nextflow file "forces" you to know what you are expecting prior to execution by first defining what you want processes you need before defining the workflow script. Although not shown here, Nextflow uses "Channels" to help with workflow execution. A channel can have one or more defined processes, but it can "wait": these processes wait until the input is ready before executing, thus creating the possibilty of an asynchronous processing pipeline (and parallel processing!). This is also what makes Nextflow able to manage resources better than Snakemake (the processes in "inactive" channels do not use resources until they are needed). ## Reproducibility Tutorial This section is going to cover a short tutorial on reproducibility using software, tools and practices discussed today and throughout FOSS. !!! note "OS of choice" This tutorial will be performed using the [CyVerse CLI (Command Line Interface)](https://de.cyverse.org/apps/de/5f2f1824-57b3-11ec-8180-008cfa5ae621){target=_blank}. However, if you'd like to use your own computer feel free to! If you're on Mac or Linux, open your terminal; If you're on Windows, use the Windows Subsystem for Linux (WSL) ??? tip "How to Scroll in Cyverse(Tmux) Cloud Shell" If you're using the Cyverse Cloud Shell, you can scroll up and down by pressing `Ctrl + b` and then `[` to enter scroll mode. You can then use the arrow keys to scroll up and down. Press `q` to exit scroll mode. !!! success "Tutorial Goals" - Run a small workflow using NextFlow - Understand best practices for reproducing a workflow - Apply FOSS procedures in order to enable easiness of reproducibility ### Prerequisites What you'll be using: - [GitHub](https://github.com/){target=_blank} (already installed) - [Conda](https://docs.conda.io/en/latest/){target=_blank} - [Mamba](https://mamba.readthedocs.io/en/latest/installation.html){target=_blank} (optional, recommended) Installable through Conda/Mamba: - [Nextflow](https://www.nextflow.io/docs/latest/index.html){target=_blank} - [Salmon](https://salmon.readthedocs.io/en/latest/){target=_blank} - [FastQC](https://www.bioinformatics.babraham.ac.uk/projects/fastqc/){target=_blank} Installabe through Pip: - [MultiQC](https://multiqc.info/docs/){target=_blank} ??? question "What's a Conda and how do I install it?" [Conda](https://docs.conda.io/en/latest/){target=_blank} is a popular tool for installing software. Typically software you want to use requires other software (dependancies) to be installed. Conda can manage all of this for you. Each available Conda package is part of a “recipe” that includes everything you need to run your software. There are different versions of Conda, including some specific for bioinformatics like [Bioconda](https://bioconda.github.io/){target=_blank}. The CyVerse CLI already comes with Conda installed; Please follow these steps in order to install [MiniConda](https://docs.conda.io/en/latest/miniconda.html){target=_blank} (the lightweight version of Conda) on your system. For the appropriate installation package, visit https://docs.conda.io/en/latest/miniconda.html. ⚠️ Note: **If you are using the WSL, install the Linux version!!** ``` # Download conda and add right permissions wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh # Modify this to match the OS you're using. chmod +x Miniconda3-latest-Linux-x86_64.sh # install conda silenty (-b) and update (-u) and initial conda run ./Miniconda3-latest-Linux-x86_64.sh -b -u ~/miniconda3/bin/conda init # Restart bash so that conda is activated source ~/.bashrc ``` You'll be able to tell when conda is active when next `(base)` is present next to the to the shell prompt such as ``` (base) user@machine ``` Conda should now be installed and can be used to install other necessary packages! When you start a Cyverse Cloud shell, it will start you in the directory: ``` /home/user/work ``` Let's change to the Data-Store directory, where we will be working for the rest of the tutorial. This is the Cyverse cloud-storage directory where you can put all your data and files. ``` cd home cd
Adapted from [CyVerse FOSS](https://github.com/CyVerse-learning-materials/foss/blob/86d4cc384976a59bd19cce2a6fdcf994b19051bd/docs/nextflow.md){target=_blank} (last source update 2024-07-02), CC BY 4.0. Spotted a problem? [Open an issue](https://github.com/UNM-CARC/foss/issues){target=_blank}.
---8<--- https://unm-carc.github.io/foss/capstone/overview/ --- title: "Capstone Project Overview" description: "Objectives, example projects, and presentation prompts for the FOSS capstone, in which learners apply a course skill to their own work." type: Guide tags: - Capstone - Projects generated: by: "claude/opus-5" at: "2026-09-11T00:00:00Z" sources: - id: cyverse-foss resource: "https://github.com/CyVerse-learning-materials/foss/blob/86d4cc384976a59bd19cce2a6fdcf994b19051bd/docs/final_project/overview.md" title: "CyVerse FOSS: docs/final_project/overview.md" author: "team:cyverse" last_modified: "2024-10-10T09:02:48-07:00" --- # Capstone Project Overview The capstone project for FOSS is designed to give you the opportunity to apply some of the skills and ideas from the rest of the course to one of your own projects, get support and feedback from your peers, and share your experiences with the rest of the attendees. --- ## Objectives - "Level up" the openness of an existing or planned project - Apply a skill or concept from FOSS - Discuss your applied skills/concepts with groupmates - Determine possible next steps to "level up" your project - Share your experiences with the rest of the attendees --- ## Description Choose one (or more) skill(s) or concept(s) from FOSS and apply it to one of your own projects. Here are some example projects you might consider: !!! tip "Example Projects" - Make a fully developed Data Management Plan (DMP) for a current or future project - Deposit your data in an open repository that is appropriate for your field (cannot be a previous effort) - Create a Git repository that shares code or data from a current project. Must have a detailed README that explains exactly how colleagues would use the code or data. Repo should provide an 'environment' file that allows others to reproduce your environment on their computer OR the scripts should be containerized. - Create a fully developed personal website hosted in Github - Create documentation of a project, tool, or workflow using Mkdocs or similar template - Develop an educational tutorial or how-to guide to teach colleagues some aspect of your research. Use mkdocs or similar template on the web. - Pre-register a protocol on a platform like Open Science Framework - Publish a protocol on protocols.io or other 'protocol' website - Conduct an open peer-review of new research in a preprint server - Demonstrate how you are using a remote computing resource to run your analyses (e.g., Cyverse, HPC) You should have your skill/topic chosen by **Week 8**, and you will then enter your name/topic into a spreadsheet (shared during the week 8 session). We'll use this to help learners find others working on similar projects. - You can do a project completely solo - You can work solo, but join a 'support group' to: 1. discuss your experiences applying your skill or concept 2. give each other help with sticking point you may encounter - You can work directly with a FOSS colleague on a joint project ### Final Presentations Each student or group will deliver a short (3-7 minute) presentation to the rest of the class on the final day of the course (see the [schedule](https://unm-carc.github.io/foss/course/schedule/)) Your short presentation should focus on the challenges and tips you may have for other FOSS attendees who want to utilize your skill or concept in the future. Here are some prompts that you should address during your presentation: !!! tip "Presentation Thoughts" - What was the general topic or skill that members of your group worked on? - What are some challenges you encountered while working on your projects? - Were you able to overcome these challenges? If so, how? - Where did you look for help? - Do any roadblocks remain? How might you try to overcome them? - Are there any new things you learned while working on your project? - Did you end up using any new or different tools? - What are some tips you might have for other FOSS attendees who want to work on the same topic/skill in the future? - Are there any pitfalls to avoid? - What things do you want to do next? - How might you "level up" in the current topic/skill compared to your project's current state? - Are there any other FOSS skills that you want to tackle next? If so, how might they integrate with the topic/skill you focused on for your project?Adapted from [CyVerse FOSS](https://github.com/CyVerse-learning-materials/foss/blob/86d4cc384976a59bd19cce2a6fdcf994b19051bd/docs/final_project/overview.md){target=_blank} (last source update 2024-10-10), CC BY 4.0. Spotted a problem? [Open an issue](https://github.com/UNM-CARC/foss/issues){target=_blank}.
---8<--- https://unm-carc.github.io/foss/about/credits/ --- title: "Credits and attribution" description: "The CyVerse origin of FOSS, its authors and instructors, funding, how to cite the course, and the CC BY 4.0 attribution for this adaptation." type: Reference tags: - About - Credits - Citation generated: by: "claude/opus-5" at: "2026-09-11T00:00:00Z" sources: - id: cyverse-foss resource: "https://github.com/CyVerse-learning-materials/foss/blob/86d4cc384976a59bd19cce2a6fdcf994b19051bd/README.md" title: "CyVerse FOSS: README.md (citation)" author: "team:cyverse" - id: cyverse-foss-license resource: "https://github.com/CyVerse-learning-materials/foss/blob/86d4cc384976a59bd19cce2a6fdcf994b19051bd/License.md" title: "CyVerse FOSS: License.md" author: "team:cyverse" - id: cyverse-foss-home resource: "https://github.com/CyVerse-learning-materials/foss/blob/86d4cc384976a59bd19cce2a6fdcf994b19051bd/docs/index.md" title: "CyVerse FOSS: docs/index.md (funding and citations)" author: "team:cyverse" --- # Credits and attribution ## Origin Foundational Open Science Skills (FOSS) was created by [CyVerse](https://cyverse.org){target=_blank}, the National Science Foundation–funded cyberinfrastructure project housed at the University of Arizona, and was presented with the University of Arizona [Data Science Institute](https://datascience.arizona.edu){target=_blank}. CyVerse taught FOSS as a weekly online workshop for several years; the most recent offering recorded here ran in Fall 2025. * Original site: [foss.cyverse.org](https://foss.cyverse.org){target=_blank} * Original repository: [CyVerse-learning-materials/foss](https://github.com/CyVerse-learning-materials/foss){target=_blank} (this adaptation starts from commit `86d4cc3` on the `mkdocs` branch) ## Authors and instructors The FOSS course materials are authored by the instructors who also taught the lessons: * **Michele Cosi** — [ORCID 0000-0001-7609-1939](https://orcid.org/0000-0001-7609-1939){target=_blank} * **Jeffrey K. Gillan** — [ORCID 0000-0002-0731-3048](https://orcid.org/0000-0002-0731-3048){target=_blank} * **Tyson L. Swetnam** — [ORCID 0000-0002-6639-7181](https://orcid.org/0000-0002-6639-7181){target=_blank} Other contributors to the source repository over its history include Mary Margaret Sprinkle, Ryan Bartelme, Michael Culshaw-Maurer, Eli Schillinger, Jason Williams, and Shashank Yadav. Lesson 10, *AI Models & Where to Find Them*, is taught from the separate [GPT 101 workshop](https://tyson-swetnam.github.io/intro-gpt/){target=_blank}. Parts of [How to Talk to Computers](https://unm-carc.github.io/foss/lessons/03-talk-to-computers/) follow the [Software Carpentry Unix Shell lesson](https://swcarpentry.github.io/shell-novice/){target=_blank}; third-party figures are credited in their captions. ## How to cite Cite the original course: > Michele Cosi, Jeffrey K. Gillan, Tyson L. Swetnam. (2024, May 2). > *CyVerse-learning-materials/foss: FOSS 2024 Spring*. Zenodo. > [https://doi.org/10.5281/zenodo.11105956](https://doi.org/10.5281/zenodo.11105956){target=_blank} The [CyVerse Zenodo community](https://zenodo.org/communities/cyverse){target=_blank} holds citable versions of other CyVerse materials. When you use CyVerse resources, follow the [CyVerse citation policy](https://cyverse.org/policies/cite-cyverse){target=_blank}. ## Funding CyVerse is funded by the Arizona Board of Regents and the US National Science Foundation under Award Numbers [DBI-0735191](https://www.nsf.gov/awardsearch/showAward?AWD_ID=0735191){target=_blank}, [DBI-1265383](https://www.nsf.gov/awardsearch/showAward?AWD_ID=1265383){target=_blank}, and [DBI-1743442](https://www.nsf.gov/awardsearch/showAward?AWD_ID=1743442){target=_blank}. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation. ## License and changes The original FOSS documentation is Copyright © CyVerse and is available under the [Creative Commons Attribution 4.0 International License (CC BY 4.0)](https://creativecommons.org/licenses/by/4.0/){target=_blank}. This site is derived from that work and is shared under the same license (see `LICENSE` in the repository). CyVerse does not endorse this adaptation. The UNM Center for Advanced Research Computing (CARC) made these changes: * Rebuilt the site with [Zensical](https://zensical.org){target=_blank} as an Open Knowledge Format (OKF) v0.2 bundle, restyled after the [CARC documentation](https://carc.unm.edu/docs/){target=_blank}, and reorganized pages into Course, Lessons, Capstone, and About sections with new URLs. * Added provenance frontmatter and a source footer to every adapted page, fixed broken links, aligned lesson titles, refreshed a few outdated instructions, and added short tips for UNM CARC users. * Dropped archived lessons, the FOSS+ page, the chatbot widget, and unused images. The originals remain in the CyVerse repository. The full change history is in the [update log](https://unm-carc.github.io/foss/log/). ---8<--- https://unm-carc.github.io/foss/about/contributing/ --- title: "Contributing" description: "How to edit pages, the OKF frontmatter contract, verifying adapted content, and building the site locally." type: Guide tags: - About - Contributing generated: by: "claude/opus-5" at: "2026-09-11T00:00:00Z" sources: - id: carc-docs resource: "https://carc.unm.edu/docs/about/contributing/" title: "CARC Documentation: Contributing to these docs" author: "team:unm-carc" --- # Contributing This site is a git repository of Markdown files, built with [Zensical](https://zensical.org){target=_blank} and structured as an [Open Knowledge Format (OKF) v0.2](https://github.com/GoogleCloudPlatform/open-knowledge-format){target=_blank} knowledge bundle — every page is readable by people *and* consumable by AI agents, with provenance and trust signals in its frontmatter. ## Small fixes Every page has an **edit button** (:material-pencil:) in the upper right that opens the source file on GitHub. Fix the text, propose the change, and CI validates and deploys it once merged. ## The frontmatter contract Every content page starts with YAML frontmatter. `type` is required by OKF; the rest make the page trustworthy and discoverable: ```yaml --- title: "Page title" description: "One sentence used by search, indexes, and agents." type: Lesson # Lesson, Guide, Reference, Policy, or Schedule tags: - Containers generated: by: "human:yournetid" # who/what wrote the current content at: "2026-09-11T00:00:00Z" sources: # where the content came from (optional) - id: upstream resource: "https://github.com/..." title: "Original source" author: "team:cyverse" status: stable # draft | stable | deprecated (default: stable) --- ``` Section `index.md` files are OKF directory listings and carry **no frontmatter**; `log.md` is the bundle's dated change log — add an entry when you make a meaningful change. Every page must also be listed in the `nav` of `zensical.toml`. ## Verifying adapted pages Pages adapted from CyVerse materials were migrated by an agent and are intentionally **unverified**. When you review one and confirm it is correct, record it: ```yaml verified: { by: "human:yournetid", at: "2026-09-15T00:00:00Z" } ``` If a page is obsolete, set `status: deprecated`, add a note pointing at the replacement, and log the change in `log.md`. ## Building locally ```bash git clone https://github.com/UNM-CARC/foss.git && cd foss python3 -m venv .venv && source .venv/bin/activate pip install -r requirements.txt # Python 3.11+ zensical serve # live preview at localhost:8000 python3 scripts/okf_validate.py docs # OKF conformance check (runs in CI) python3 scripts/check_links.py docs # links, images, nav coverage (runs in CI) python3 scripts/gen_llms_txt.py # regenerate llms.txt indexes (CI checks drift) ``` ## Style notes * One `#` H1 per page, matching the frontmatter `title`. * Relative links between pages (`../section/page.md`); never site-root links (`/section/page/`), because the site is served under `/foss/`. * External links get `{target=_blank}` — `scripts/externalize_links.py` adds them for you. * Admonitions (`!!! note`, `??? question`) for asides; content tabs (`=== "macOS"`) for OS-specific instructions. * Keep images next to the section that uses them, under `docs/assets/`. ---8<--- https://unm-carc.github.io/foss/about/ai-agents/ --- title: "For AI agents" description: "How agents and harnesses should consume this site: llms.txt, per-page Markdown with OKF frontmatter, and trust signals." type: Reference tags: - About - AI agents - OKF generated: by: "claude/opus-5" at: "2026-09-11T00:00:00Z" sources: - id: okf-spec resource: "https://github.com/GoogleCloudPlatform/open-knowledge-format" title: "Open Knowledge Format (OKF) v0.2 specification" author: "team:google-cloud" - id: llmstxt resource: "https://llmstxt.org" title: "The /llms.txt convention" author: "team:answer-ai" - id: carc-docs resource: "https://carc.unm.edu/docs/about/ai-agents/" title: "CARC Documentation: For AI agents" author: "team:unm-carc" --- # For AI agents This site is published for people **and** for AI agents. Its source is an [Open Knowledge Format (OKF) v0.2](https://github.com/GoogleCloudPlatform/open-knowledge-format){target=_blank} knowledge bundle, and the deployed site exposes that structure directly. If you are an agent (or you are wiring one up), consume the content through these endpoints rather than scraping rendered HTML. ## Entry points | Endpoint | What you get | | -------- | ------------ | | [`llms.txt`](https://unm-carc.github.io/foss/llms.txt) | Linked outline of every page with one-line descriptions ([llms.txt convention](https://llmstxt.org){target=_blank}) | | [`llms-full.txt`](https://unm-carc.github.io/foss/llms-full.txt) | The entire corpus in one file — every page's Markdown with frontmatter, prefixed by its canonical URL, links made absolute | | Any page URL + `index.md` | That page's Markdown source with full OKF frontmatter (e.g. `https://unm-carc.github.io/foss/lessons/05-version-control/index.md`) | | `sitemap.xml`, `robots.txt` | Standard crawl surface; robots.txt repeats these pointers | | [Source repository](https://github.com/UNM-CARC/foss){target=_blank} | The bundle itself, plus `AGENTS.md` with contribution rules for coding agents | Every rendered page also declares its Markdown twin and OKF signals in HTML: ```html ``` ## Reading the OKF frontmatter Each concept page's YAML frontmatter answers the questions agents should ask before relying on content: * **What is this?** — `type` (Lesson, Guide, Reference, Policy, or Schedule), `title`, `description`, `tags`. * **Where did it come from?** — `generated: { by, at }` and `sources`. Most pages were adapted from CyVerse learning materials; `sources[].resource` points at the exact upstream file and commit, with `last_modified` from the upstream git history. * **How much should I trust it?** — the `verified` key (OKF §5.3): absent means **unverified**; `by: "human: