Lesson 3: Copy, move, rename, and delete files and folders using Unix commands

Learning objectives

After this lesson, participants will become proficent with copying, moving, renaming, and removing files as well as folders using Unix commands.

Connecting to Biowulf

To get started, open the Command Prompt (Windows) or the Terminal (Mac) and connect to Biowulf. Remember you need to be connected to the NIH network either by being on campus or through VPN. Recall from lesson 1 that you use the ssh command below to connect to Biowulf, where username is the student account ID that was assigned to you (see student assignments). Remember that when prompted to enter your password, you are not going to be able to see it, but keep typing.

ssh username@biowulf.nih.gov

Change into the data directory after connecting. Replace username with the student account ID.

cd /data/username

Copying of files or folders

Copying folders

There is a folder called unix_on_biowulf_2024_documents in the /data/classes/BTEP folder. We can see it using the ls command with the options and arguments below.

ls -l /data/classes/BTEP

drwxrwsr-x.  4 wuz8        GAU               4096 May 22  2023 unix_on_biowulf_2024_documents

The persmission for the unix_on_biowulf_2024_documents folder is set that so others can read it.

Caution

Do not change into and work in /data/classes/BTEP/unix_on_biowulf_2024_documents.

To copy this folder to the data directory, use the command construct below where

-r: tells Unix to copy the folder and items that are in it.
/data/classes/BTEP/unix_on_biowulf_2024_documents: is the argument for the cp command (ie. this is the item that we want to make a copy of)
.: the folder where the copy should be stored. Recall that . denotes here in the present working directory, which should the /data/username folder

cp -r /data/classes/BTEP/unix_on_biowulf_2024_documents .

Use ls to confirm that the unix_on_biowulf_2024_documents has been copied.

Change into the unix_on_biowulf_2024_documents folder and use ls -l to get a detailed view of what is in it.

cd unix_on_biowulf_2024_documents

ls -l

-rwxr-x---. 1 wuz8 wuz8   368 Jan 18 11:20 SRP045416.swarm
drwxr-x---. 2 wuz8 wuz8  4096 Jan 18 11:20 SRR1553606
drwxr-x---. 2 wuz8 wuz8  4096 Jan 18 11:20 unix_on_biowulf_2024
-rwxr-x---. 1 wuz8 wuz8 41734 Jan 18 11:20 unix_on_biowulf_2024.zip

The unix_on_biowulf_2024_documents has two files and two directories.

Change into unix_on_biowulf_2024.

cd unix_on_biowulf_2024

List contents of unix_on_biowulf_2024.

ls -l

-rwxr-x---. 1 wuz8 wuz8  31666 Jan 18 11:20 counts.csv
-rwxr-x---. 1 wuz8 wuz8 104473 Jan 18 11:20 results.csv
-rwxr-x---. 1 wuz8 wuz8     84 Jan 18 11:20 text_1.txt

Copy a file

Earlier, we used cp with the -r option to recursively copy the unix_on_biowulf_2024_documents directory and all of its contents to the data directory. Suppose we want to make a copy of just one file (the counts.csv file) in the unix_on_biowulf_2024 subfolder of unix_on_biowulf_2024_documents, how would we do this? We could use the cp command with the following arguments.

Argument: File to make a copy of (ie. counts.csv)
Argument: Name of the copy (ie. counts_copy.csv)

cp counts.csv counts_copy.csv

Supposed that we want to make a copy of text_1.txt and call it text_1_copy.txt, we can use the cp command again.

cp text_1.txt text_1_copy.txt

Use ls -1 to list the items in unix_on_biowulf_2024.

ls -1

The copies of counts.csv and text_1.txt have been made.

counts.csv
counts_copy.csv
results.csv
text_1.txt
text_1_copy.txt

Now, if we wanted to make a copy of counts.csv and place it one directory up in the unix_on_biowulf_2024_documents folder then we can use the command below, where ../ represents go back one directory.

cp counts.csv ../counts_copy_1.csv

ls -1 ../

SRP045416.swarm
SRR1553606
counts_copy_1.csv
unix_on_biowulf_2024
unix_on_biowulf_2024.zip

Tip

Note that the size of directory contents are listed as bytes. We can get a more human readable form of the size by appending the -h option to ls -l. Again we can use ls --help to find out about the -h option. Of interest is that options for Unix commands can also be written in the long form preceded by "--" (ie. -h is the same as --human-readable). This an example of a command that does not use -h as a short notation for --help.

ls --help

-h, --human-readable  with -l, print sizes in human readable format
                        (e.g., 1K 234M 2G)

ls -lh

-rwxr-x---. 1 wuz8 wuz8  31K Jan 18 11:20 counts.csv
-rwxr-x---. 1 wuz8 wuz8  31K Jan 18 11:58 counts_copy.csv
-rwxr-x---. 1 wuz8 wuz8 103K Jan 18 11:20 results.csv
-rwxr-x---. 1 wuz8 wuz8   84 Jan 18 11:20 text_1.txt
-rwxr-x---. 1 wuz8 wuz8   84 Jan 18 12:06 text_1_copy.txt

Tip

Speaking of file sizes, we can use the checkquota command to check disk usage. This should give the same result as that shown in the user dashboard.

checkquota

Mount                   Used      Quota  Percent    Files    Limit  Percent
/data:                1.5 TB     2.0 TB   75.27%   127141 32000000    0.40%
/home:                3.2 GB    16.0 GB   19.87%    63871      n/a    0.00%

Rename and moving

Stay in unix_on_biowulf_2024 for this exercise.

To rename a file, use the mv command. Below, counts_copy.csv is renamed as rna_seq_counts.csv.

mv counts_copy.csv rna_seq_counts.csv

To move a file into a different directory, we can also rely on the mv command. Below, text_1_copy.txt is moved one directory up to the unix_on_biowulf_2024_documents folder.

mv text_1_copy.txt ../

Confirm that the move was successful.

ls -1 ../

SRP045416.swarm
SRR1553606
counts_copy_1.csv
text_1_copy.txt
unix_on_biowulf_2024
unix_on_biowulf_2024.zip

The mv command can be used to rename and move folders as well.

Make a new folder called lesson2.

mkdir lesson3

Rename the folder lesson2 to lesson3_january_2024_unix_class

mv lesson3 lesson3_january_2024_unix_class

Deleting files and folders

To delete a folder, use the rm command with the -r options, which enables deletion of the folder and everything that is in it.

To delete the lesson2_january_2024_unix_class directory, use the following.

rm -r lesson2_january_2024_unix_class

Tip

The rmdir command can be used to delete empty folders.

To delete a file, just use rm followed by the file to be removed.

rm rna_seq_counts.csv

Snapshots

Biowulf and other high performance computing clusters do not have a recycling bin or trash can to hold deleted files or folders and allow users to restore if they removed by accident. However, users are able to restore from snaphots. See https://hpc.nih.gov/storage/backups.html to learn how to restore from snapshot on Biowulf.

Definition

"A 'snapshot' is a read-only copy of the data at a particular point in time." -- Biowulf

A snapshot of the files or folders have to be made in order for restoration to be possible.

Caution

Be care when deleting items when working on Biowulf.

The way to access snapshots for the data directory depends on the user's storage system. Use ls -ld /data/username to find out. Replace username with your specific Biowulf account ID. In these examples, my Biowulf account ID (wuz8) will be used to demonstrate.

ls -ld /data/wuz8

From the results below, my data directory is on a VAST storage system evident by it beginning with /vf.

lrwxrwxrwx. 1 root root 14 May  5  2023 /data/wuz8 -> /vf/users/wuz8

Note

The /data/wuz8 path is just symlink which points to an actual file storage location. Symlinks are useful as they provide shortcuts for referencing things like really long file paths.

The data directory has daily and weekly snapshots. Choose the the appropriate one and append your username to goto your user specific data folder snapshot.

ls -1 /vf/users/.snapshot/

daily-snap._2024-01-16_04_00_00_UTC
daily-snap._2024-01-17_04_00_00_UTC
daily-snap._2024-01-18_04_00_00_UTC
weekly-snap._2024-01-07_07_00_00_UTC
weekly-snap._2024-01-14_07_00_00_UTC

ls -1 /vf/users/.snapshot/daily-snap._2024-01-18_04_00_00_UTC/wuz8

Suppose that I want to restore a file example_text.txt from snapshot back to my data folder, the following can be used.

cp /vf/users/.snapshot/daily-snap._2024-01-18_04_00_00_UTC/wuz8/example_text.txt /data/wuz8

Snapshots are also made for the home directory.

ls ~/.snapshot

Hourly.2024-01-17_1400  Hourly.2024-01-18_0800   Nightly.2024-01-17_0010  Weekly.2023-12-24_0010
Hourly.2024-01-17_1700  Hourly.2024-01-18_1100   Nightly.2024-01-18_0010  Weekly.2023-12-31_0010
Hourly.2024-01-17_2000  Nightly.2024-01-12_0010  Weekly.2023-11-26_0010   Weekly.2024-01-07_0010
Hourly.2024-01-17_2300  Nightly.2024-01-13_0010  Weekly.2023-12-03_0010   Weekly.2024-01-14_0010
Hourly.2024-01-18_0200  Nightly.2024-01-15_0010  Weekly.2023-12-10_0010
Hourly.2024-01-18_0500  Nightly.2024-01-16_0010  Weekly.2023-12-17_0010