Skip to content

Transferring Data to the NIH Partek Flow Server Using Command Line

Note

Command line uploads can be used when the files are already present on Biowulf. -- Biowulf

Copy from User's Biowulf data Folder to Partek Flow uploads Folder

Copying from user's Biowulf /data directory to the uploads subfolder in PartekFlow is allowed. Be sure to be in an interactive session for this by using the sinteractive command. For instance:

To transfer the folder hbr_uhr_fastq_download in /data/username to the uploads subfolder in PartekFlow, do the following. Replace username with the user's Biowulf user name.

cp -r /data/username/hbr_uhr_fastq_download /data/username/PartekFlow/uploads

Note that the Biowulf user owns the hbr_uhr_fastq_download PartekFlow. This enables users to delete the folder. The ls command lists directory content and the -l option list the content in long or detailed format.

[wuz8@cn4304 username]$ ls -l /data/username/PartekFlow/uploads/
drwxr-s---+ 2 username     partekfl 4096 May 28 11:55 hbr_uhr_fastq_download

It is possible to change into the upload subfolder in /data/username/PartekFlow and make a directory using the mkdir command.

cd /data/username/PartekFlow/uploads

[username@cn4304 uploads]$ mkdir hbr_uhr_fastq

Again the user owns the directory, and thus can delete it. Creating folders is a good way to keep data organized for analyses.

[wuz8@cn4304 uploads]$ ls -l
drwxrws---+ 2 username     partekfl 4096 May 28 13:12 hbr_uhr_fastq

Tip

File transfer using command line:

  • Copy from data folder to Partek Flow uploads folder is OK
  • Copy from data folder to Partek Flow directory is OK
  • Copy from data folder to Partek Flow project folder is not OK
  • Copy from Partek Flow uploads folder to project folder is not OK

Caution

While the above command line methods for data transfer are valid, users should remember that: "Virtually all file transfer activities should be run from within the web interface, rather than from the command line. We have implemented a permissions policy on users' PartekFlow directories to prevent inadvertent file removal mistakes that breaks the old way of moving files around." -- Biowulf staff

scp from Personal Computer to the Partek Flow uploads Folder.

The scp commands enables users to securely transfer data to and from personal computer to a remote computer such as a high performance computing system. To transfer data from personal computer to the Partek Flow uploads folder do the following.

Note

Helix is the node on the Biowulf system that is intended for data transfer.

scp local_file username@helix.nih.gov:/data/username/PartekFlow/uploads

Enter user's Biowulf password.

For example, the following command construct will transfer the folder hbr_uhr_fastq on local computer that contains FASTQ files to the Partek Flow uploads folder on Biowulf. The -r option in scp will copy a folder and its content.

scp -r hbr_uhr_fastq_files username@helix.nih.gov/data/username/PartekFlow/uploads

ls -l /data/username/PartekFlow/uploads

Again, the Biowulf account user owns the directory and can modify and delete it.

drwxr-s---+ 2 username  partekfl 4096 Jun 25 21:13 hbr_uhr_fastq_files