SFTP, scp, rsync

Owned by davidls
Last updated: Jan 24, 2024
3 min read

SFTP, SCP, and rsync are commonly used tools used to transfer data to and from a remote Linux based server.  Both scp and rsync are typically command line tools, while SFTP does have command line variants, graphical interface applications, such as Cyberduck, and Filezilla are capable or also transfering files using SFTP.  While these tools are common and easily accessible, they may not be the best tools to use when performing large scale data movement between Mana and other servers or computing centers with external high speed connections (Speed greater than 1 Gbit/s).  In cases you are performing large data movements, we would recommend looking to see if Globus can be utilized. 

SFTP, SCP and rsync are all be subject to the same authentication requirements as SSH.  This means, that they require MFA/DUO authentication.  Due to the nature of these protocols, they DO NOT provide you with a menu of choices to say where to send your request for the second authentication.  Instead, it will automatically send a push notification to your primary device (the first option you see with SSH login).  Please see DUO/MFA considerations on how one can also correctly configure tools like Cyberduck and FIlezilla to not cause problems with DUO.  Also, be aware as none of these applications support terminal input for entering your second factor, use of the YUBIkey or passcode will not be available.  If you only have a YUBIKey or can only enter a passcode for your second factor, we recommend selecting another mode of transfer, such as Globus.



scp

scp - Secure Copy, is a command line tool that can be used to transfer files to and from Mana.  In order for scp to work, the remote server you are trying to download from or upload files to, needs to be running an SSH server.

If the remote server is going to be Mana, be aware that you will be sent an automatic DUO push to your primary authentication option.   As scp does not have any terminal functionality, YUBIKey or passcode entry is not possible.

scp has a few valid syntaxes depending on how you plan to perform your data transfer.  Can either push or pull data from the client side (the side you are trying to scp command into).  



Pulling data
[user@login001 ~]$ scp [username@remote_host:][path to file on remote_host] [path to where to save file on local host] [user@login001 ~]$ scp user@uhunix.hawaii.edu:test/simple.c .

In the above case, we are downloading the file "simple.c" from uhunix.its.hawaii.edu and placing it in our current directory on the local machine, login001. 

Pushing data
[user@login001 ~]$ scp [path to where to save file on local host] [username@remote_host:][path to file on remote_host] [user@login001 ~]$ scp advance.c user@uhunix.hawaii.edu:test/

In the above case, we are uploading the file "advance.c" from our local machine, login001, to uhunix.its.hawaii.edu and placing it into the remote directory "test". 



In the above, we are transferring a single file, but scp also have a flag that allows you to transfer an entire directory tree.  One would just add the "-r" flag, which stands for recursive and provide a directory instead of a file and scp will then copy the entire directory tree.

Pulling data
[user@login001 ~]$ scp -r [username@remote_host:][path to file on remote_host] [path to where to save file on local host] [user@login001 ~]$ scp -r user@uhunix.hawaii.edu:test c .

In the above case, we are downloading the entire directory called "test" from uhunix.its.hawaii.edu and placing it in our current directory on the local machine, login001. 

Pushing data

In the above case, we are uploading the entire directory "test2" from our local machine, login001, to uhunix.its.hawaii.edu and placing it inside the remote directory "test". 



Manual (man) pages for scp: https://linux.die.net/man/1/scp



rsync

For rsync, please make sure to read our FAQ question regarding rsync and how you should be using checksum instead of file size to compare files on Mana due to file system compression.

Manual (man) pages for rsync: https://linux.die.net/man/1/rsync

https://www.digitalocean.com/community/tutorials/how-to-use-rsync-to-sync-local-and-remote-directories



SFTP (FileZilla and Cyberduck)

For Cyberduck and FileZilla, please make sure to look at our FAQ entry regarding MFA/DUO to make sure you setup your SFTP client to work properly with DUO/MFA.