Skip to content

Troubleshooting

Work through the section that matches your symptom. If nothing here fixes it, open a ticket with the details listed at the bottom of that page — cluster, job ID, exact command, and full error message.

I can't log in

  1. Check for maintenance on the cluster status monitor and UNM IT alerts.
  2. Password or OTP problems — reset via the steps in password reset.
  3. Permission denied (publickey) — your SSH key setup is incomplete or has wrong permissions; see SSH keys (~/.ssh must be 700, private keys 600).
  4. Account exists but no access — you may not be on an active project yet; ask your PI to add you in ColdFront (getting started).

"Disk quota exceeded"

You've hit a storage limit (what the limits are):

quotas            # show your usage against each quota
du -sh ~/* | sort -rh | head   # find what's using home space

Clean up, move bulk data to scratch or project space (storage layout), or talk to us about purchasing more. Remember conda environments and pip caches grow quietly — conda clean --all and pip cache purge often free gigabytes.

My job won't start

squeue -u $USER           # state and reason code
squeue --start --job <id> # predicted start time (fairshare-aware)
sinfo                     # partition and node availability

Common reason codes:

Reason Meaning What to do
Priority Others are ahead of you (fairshare) Wait, or request fewer/shorter resources; see fairshare
Resources Not enough free nodes for your request Reduce cores/memory/GPUs or choose another partition
QOSMax* / limits You've hit a partition or account limit Check resource limits
ReqNodeNotAvail Nodes down or reserved (often maintenance) Check the cluster status monitor
InvalidAccount Wrong --account List yours: sacctmgr show assoc user=$USER format=account

My job failed or was killed

sacct -j <id> --format=JobID,State,ExitCode,Elapsed,MaxRSS,ReqMem
seff <id>     # efficiency summary after completion
  • OUT_OF_MEMORY / oom-kill — request more memory (--mem or --mem-per-cpu) or use fewer tasks per node; seff shows what you actually used.
  • TIMEOUT — raise --time within partition limits, or checkpoint and restart.
  • Immediate crash — check the job's .out/.err files in the submit directory; a missing module load or unactivated conda environment is the usual culprit (modules, conda).

Software and environment problems

  • command not found — load the module first (module spider <name> to find it; modules guide).
  • Conda is slow or conflicts — prefer clean per-project environments and the conda-forge channel; see channels and pip.
  • GPU code can't see the GPU — did you request one in the job (--gres=gpu:1 or the cluster's GPU partition)? Verify with nvidia-smi inside the job; see example Slurm scripts.
  • My kernel is missing in JupyterHub — register your environment as a kernel: conda in JupyterHub.

Graphics won't display

X11 applications need forwarding enabled — ssh -Y and a local X server; see X11 forwarding. For heavier visualization, use ParaView client–server or an Open OnDemand session instead.

Transfers are slow or failing

Use rsync with resume (rsync -avP) rather than scp for large trees, and transfer to the right storage tier — see transferring data.

Still stuck?

Open a ticket or bring it to office hours — include your cluster, job ID, command, and the complete error text.