Skip to content
PDF

Lesson 2: Useful Unix & Running Jobs on Biowulf

Learning Objectives

  • Use pipes, grep, and text utilities to process files.
  • Load software modules.
  • Run interactive sessions with sinteractive.
  • Write and submit a batch job with sbatch.

Part 1 — Powerful Unix Tools

Combining commands with pipes |

The pipe (|) sends the output of one command as the input to the next. This lets you chain tools together into a mini-pipeline.

cat sample.fastq | head -8      # view just the first 8 lines of a FASTQ file
ls -l | wc -l                   # count the number of items in a directory

Redirecting output

ls -l > filelist.txt            # write output to a file (overwrite)
ls -l >> filelist.txt           # append output to an existing file

Check-your-learning

What is the difference between > and >>? What happens if you use > on a file that already exists?

Searching files with grep

grep searches a file for lines matching a pattern:

grep "ATCG" sample.fastq            # find lines containing "ATCG"
grep -c "ATCG" sample.fastq         # count matching lines
grep -i "atcg" sample.fastq         # case-insensitive search
grep -v "^@" sample.fastq           # show lines that do NOT start with @

In FASTQ files, sequence identifier lines start with @. Using grep to find or exclude these is a common bioinformatics task.

Check-your-learning

In a FASTQ file, every 4th line starting from line 1 is a sequence header (@). Write a grep command to count only header lines.

sort and uniq

sort filelist.txt               # sort lines alphabetically
sort -k5 -n filelist.txt        # sort by column 5 numerically
sort filelist.txt | uniq        # remove duplicate lines
sort filelist.txt | uniq -c     # count occurrences of each unique line

Part 2 — The Module System

Biowulf has 600+ pre-installed programs. You load them with module:

module avail                    # list all available software (long!)
module avail sratoolkit         # search for a specific tool
module load sratoolkit          # load the tool into your environment
module list                     # see what you have loaded
module unload sratoolkit        # unload a module

Why modules?

Different analyses require different software versions. The module system lets you load exactly what you need without conflicts.

Check-your-learning

Type module avail samtools. What versions are available? How would you load a specific version?

Part 3 — Running Jobs on Biowulf

On a shared HPC system, you cannot simply run heavy programs on the login node. You must submit jobs through the Slurm scheduler (Simple Linux Utility for Resource Management). Slurm manages the queue and allocates resources fairly.

Three ways to run work on Biowulf

Method Command Best for
Interactive session sinteractive Testing, debugging, short tasks
Batch job sbatch script.sh Long-running analyses
Job array (swarm) swarm -f commands.txt Many similar jobs in parallel

Interactive sessions with sinteractive

sinteractive                        # default: 1 core, 1.5 GB RAM, 8 hrs
sinteractive --cpus-per-task=4 --mem=8g    # request more resources
Once inside, your prompt changes — you are now on a compute node, not the login node. You can run computationally intensive commands here.

hostname                            # confirm you're on a compute node
# do your work...
exit                                # return to login node when done

Check-your-learning

Why is it important NOT to run CPU-intensive jobs on the login node?

Writing a batch script with sbatch

Batch jobs run in the background without you needing to stay connected. You write a script that tells Biowulf what resources you need and what to run.

nano myfirstjob.sh

Paste this into the file: 

#!/bin/bash
#SBATCH --cpus-per-task=2
#SBATCH --mem=4g
#SBATCH --time=01:00:00
#SBATCH --partition=student
#SBATCH --job-name=my_first_job
#SBATCH --output=myfirstjob_%j.log
# Your commands go below this line
echo "Hello from Biowulf compute node!"
hostname
date

Save and exit (Ctrl+X, Y, Enter), then submit:

sbatch myfirstjob.sh

Monitoring your jobs

squeue -u $USER                     # list your running/pending jobs
scancel JOBID                       # cancel a job by its ID
cat myfirstjob_JOBID.log            # view job output after it finishes

Check-your-learning

In the script above, what does #SBATCH --time=01:00:00 do? What happens to your job if it exceeds this limit?

Part 4 — Quick Practice

  1. Start an sinteractive session on the student partition.
  2. Load the sratoolkit module.
  3. Run fastq-dump --version to confirm it loaded.
  4. Exit the interactive session.

Lesson 2 — End-of-Lesson Quiz

  1. What does the pipe | do? Give an example using two commands.
  2. What is the difference between > and >> when redirecting output?
  3. Write a grep command to find all lines in sample.fastq that start with@.
  4. What command lists all software modules currently loaded in your environment?
  5. What is Slurm, and why does Biowulf use it?
  6. What is the difference between sinteractive and sbatch?
  7. In a batch script, what does the #SBATCH line do?
  8. After submitting a job with sbatch, how do you check whether it is still running?
  9. What does --partition=student specify in a Slurm job script?
  10. You submitted a job but realize you made a mistake in the script. What command cancels a running job?