Lesson 3: Swarm Jobs, File Transfers & Downloading from the SRA

Learning Objectives

• Submit parallel jobs using swarm.
• Transfer files between Biowulf and your local computer.
• Understand the NCBI Sequence Read Archive (SRA).
• Download sequencing data using fasterq-dump.
• Scale downloads across multiple accessions.

Part 1 — Swarm Jobs: Running Many Jobs in Parallel

swarm is Biowulf's tool for running a batch of similar commands as independent parallel jobs — sometimes called an "embarrassingly parallel" workload.

Scenario: You have 10 samples to process with the same command. Instead of running them one by one (slow) or writing a complex loop, you write each command on its own line in a text file and let swarm handle the rest.

Create a swarm file

nano mycommands.swarm

echo "Processing sample SRR001"
echo "Processing sample SRR002"
echo "Processing sample SRR003"

When finished, press Ctrl+X, then Y, then Enter to save and exit nano.

Submit the swarm

swarm -f mycommands.swarm --partition=student

Default allocation: Each subjob gets 1.5 GB RAM and 1 core. You can override:

swarm -f mycommands.swarm -g 4 -t 2 --partition=student 
# -g = GB of memory per subjob, -t = threads (CPUs) per subjob

Monitor swarm jobs

squeue -u $USER                     # each subjob appears separately

For loops vs. swarm — when to use which

Part 2 — Transferring Files

At some point you will need to move data between your local computer and Biowulf. Always use Helix (helix.nih.gov) for interactive file transfers — not the Biowulf login node.

ssh username@helix.nih.gov          # connect to Helix for transfers

Copy from Biowulf to your local machine (run from your laptop terminal)

scp username@helix.nih.gov:/data/username/Module_1/results.txt .

Copy a local file to Biowulf (run from your laptop terminal)

scp ./mydata.txt username@helix.nih.gov:/data/username/Module_1/

Copy an entire directory (use -r for recursive)

scp -r username@helix.nih.gov:/data/username/Module_1 .

Alternative: You can also mount HPC directories to your local machine using SSHFS or the HPC SMB share — see the NIH HPC documentation at https://hpc.nih.gov/docs/transfer.html.

Part 3 — Introduction to the SRA

The Sequence Read Archive (SRA) is NCBI's public repository for high-throughput sequencing data (RNA-seq, ChIP-seq, WGS, etc.). Most published sequencing datasets are deposited here.

Key terminology

The ENA (European Nucleotide Archive) mirrors all SRA data — you can use wget or curl to pull gzipped FASTQ files directly from ENA without the SRA Toolkit, which can be faster for some use cases.

Part 4 — Downloading SRA Data with fasterq-dump

fasterq-dump (part of the SRA Toolkit) downloads sequencing data from the SRA and converts it to FASTQ format. It uses multi-threading (default: 6 threads) and is significantly faster than the older fastq-dump.

Copy the SRA accession list

cd /data/$USER/Module_1
cp /data/classes/BTEP/B4B_2025/Module_1/sra_files_PRJNA578488.txt .
less sra_files_PRJNA578488.txt # view the list of SRR accessions

Option A: Download interactively (single file, for testing)

sinteractive --gres=lscratch:20 --cpus-per-task=6
module load sratoolkit
fasterq-dump -t /lscratch/$SLURM_JOBID SRR10314042 -O /data/$USER/Module_1/
exit

-t specifies a temp directory (local scratch space on the compute node — fast and automatically cleaned up). -O specifies the output directory.

Option B: Batch script (single file, unattended)

nano filedownload.sh

#!/bin/bash
#SBATCH --cpus-per-task=6
#SBATCH --gres=lscratch:10
#SBATCH --partition=student
#SBATCH --job-name=sra_download
#SBATCH --output=sra_download_%j.log
module load sratoolkit
fasterq-dump -t /lscratch/$SLURM_JOB_ID SRR10314042 -O /data/$USER/Module_1/

sbatch filedownload.sh
squeue -u $USER

Option C: Swarm (multiple files in parallel — the real power)

First, build your swarm file:

nano download_all.swarm
fasterq-dump -t /lscratch/$SLURM_JOB_ID SRR10314042 -O /data/$USER/Module_1/
fasterq-dump -t /lscratch/$SLURM_JOB_ID SRR10314043 -O /data/$USER/Module_1/
fasterq-dump -t /lscratch/$SLURM_JOB_ID SRR10314044 -O /data/$USER/Module_1/

Press Ctrl+X, then Y, then Enter to save and exit nano.

swarm -f download_all.swarm -t 6 --gres=lscratch:10 --module sratoolkit --partition=student

Using prefetch + fasterq-dump (fastest method)

For large files, downloading with prefetch first is faster overall:

sinteractive --gres=lscratch:50 --cpus-per-task=6
module load sratoolkit
prefetch SRR10314042 -O /lscratch/$SLURM_JOBID/
fasterq-dump /lscratch/$SLURM_JOBID/SRR10314042/ -O /data/$USER/Module_1/

Part 5 — Checking Your Downloads

Use seqkit to get quick statistics about your FASTQ files:

module load seqkit
seqkit stats /data/$USER/Module_1/*.fastq

This reports: file name, format, type, number of sequences, sum of lengths, min/max/average read lengths.

Lesson 3 — End-of-Lesson Quiz

1. What is swarm used for, and how does it differ from a standard sbatch batch job?
2. How do you specify the amount of memory and number of CPUs for each subjob in a swarm?
3. Why should you use Helix (helix.nih.gov) for file transfers rather than the Biowulf login node?
4. Write an scp command to copy a file called results.txt from /data/username/Module_1/ on Helix to your local current directory.
5. What is the SRA, and why is it important for bioinformatics research?
6. What is the difference between a BioProject accession (e.g., PRJNA578488) and an SRR run accession?
7. What does fasterq-dump do, and how is it different from fastq-dump?
8. Why do we specify --gres=lscratch:20 when running fasterq-dump? What is lscratch?
9. What is the advantage of using prefetch before fasterq-dump?
10. You want to download 50 SRA accessions in parallel using swarm. Describe the steps you would take, from creating the swarm file to submitting and monitoring the jobs.