rclone is a command line tool that is available on Koa, which can connect to multiple different types of external storage systems, such as

A full list of supported storage types can be found here: https://rclone.org/overview/

While the Rclone documentation is complete and covers the configuration for each storage type it supports, we will cover the setup of Google Drive using Koa.

This example will utilize the app ID that the rclone developer provides and we will not be covering the process of how one setup their own App ID to use for rclone. Steps to do this are covered in the rclone documentation and other similar tools may also provide similar instructions on setting up your personal App ID, such as https://www.iperiusbackup.net/en/how-to-enable-google-drive-api-and-get-client-credentials/ .

1 Google Drive Example
- 1.1 Connecting
  - 1.1.1 Login to DTN
  - 1.1.2 rclone command
- 1.2 Initial setup
  - 1.2.1 Setup rclone with Google Drive
  - 1.2.2 Continue Setting up rclone with Google Drive
- 1.3 Listing Files
- 1.4 Transfering Files with the Copy Command
  - 1.4.1 Upload Directory to Google Drive
    - 1.4.1.1 Copy to Google Drive
  - 1.4.2 Download a Directory from Google Drive
    - 1.4.2.1 Copy Data from Google Drive

Google Drive Example

When using Koa, rclone can be used from two locations

Data Transfer Nodes (DTNs)
Computing nodes by loading the rclone module

The following example will use one of the DTNs to setup and use rclone. To utilize it on a computing node, one would start an interactive session and load the rclone module "module load tools/rclone".

Connecting

Frist, you will need to ssh into one of the DTN nodes using your UH credentials

Login to DTN

[user@localhost ~]$ ssh user@koa-dtn.its.hawaii.edu
user@koa-dtn.its.hawaii.edu's password:
Duo two-factor login for user

Enter a passcode or select one of the following options:

1. Duo Push to XXX-XXX-xxxx
2. Phone call to XXX-XXX-xxxx

Passcode or option (1-1): 1

Pushed a login request to your device...
Success. Logging you in...

Last login: Fri Mar 5 06:14:19 UTC 2021 on pts/2
[user@koa-dtn ~]$

Once logged in, rclone si installed on your default PATH, so you should be able to access rclone once you are logged in.

rclone command

[user@koa-dtn ~]$ which rclone 
/bin/rclone
[user@koa-dtn ~]$ rclone 
Usage:
rclone [flags]
rclone [command]

Available Commands:
about              Get quota information from the remote.
authorize          Remote authorization.
backend            Run a backend specific command.
cat                Concatenates any files and sends them to stdout.
check              Checks the files in the source and destination match.
cleanup            Clean up the remote if possible.
config             Enter an interactive configuration session.
copy               Copy files from source to dest, skipping already copied.
copyto             Copy files from source to dest, skipping already copied.
copyurl            Copy url content to dest.
cryptcheck         Cryptcheck checks the integrity of a crypted remote.
cryptdecode        Cryptdecode returns unencrypted file names.
dedupe             Interactively find duplicate filenames and delete/rename them.
delete             Remove the contents of path.
deletefile         Remove a single file from remote.
genautocomplete    Output completion script for a given shell.
gendocs            Output markdown docs for rclone to the directory supplied.
hashsum            Produces a hashsum file for all the objects in the path.
help               Show help for rclone commands, flags and backends.
link               Generate public link to file/folder.
listremotes        List all the remotes in the config file.
ls                 List the objects in the path with size and path.
lsd                List all directories/containers/buckets in the path.
lsf                List directories and objects in remote:path formatted for parsing.
lsjson             List directories and objects in the path in JSON format.
lsl                List the objects in path with modification time, size and path.
md5sum             Produces an md5sum file for all the objects in the path.
mkdir              Make the path if it doesn't already exist.
mount              Mount the remote as file system on a mountpoint.
move               Move files from source to dest.
moveto             Move file or directory from source to dest.
ncdu               Explore a remote with a text based user interface.
obscure            Obscure password for use in the rclone config file.
purge              Remove the path and all of its contents.
rc                 Run a command against a running rclone.
rcat               Copies standard input to file on remote.
rcd                Run rclone listening to remote control commands only.
rmdir              Remove the path if empty.
rmdirs             Remove empty directories under the path.
serve              Serve a remote over a protocol.
settier            Changes storage class/tier of objects in remote.
sha1sum            Produces an sha1sum file for all the objects in the path.
size               Prints the total size and number of objects in remote:path.
sync               Make source and dest identical, modifying destination only.
touch              Create new file or change file modification time.
tree               List the contents of the remote in a tree like fashion.
version            Show the version number.

Use "rclone [command] --help" for more information about a command.
Use "rclone help flags" for to see the global flags.
Use "rclone help backends" for a list of supported services.

[user@koa-dtn ~]$

Initial setup

In order to utilize rclone, a configuration step needs to be completed before you are able to move files to and from google drive.

Setup rclone with Google Drive

[user@koa-dtn ~]$  rclone config
No remotes found - make a new one
n) New remote
s) Set configuration password
q) Quit config
n/s/q> n

name> uh_gdrive

Type of storage to configure.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
1 / 1Fichier
\ "fichier"
2 / Alias for an existing remote
\ "alias"
3 / Amazon Drive
\ "amazon cloud drive"
4 / Amazon S3 Compliant Storage Provider (AWS, Alibaba, Ceph, Digital Ocean, Dreamhost, IBM COS, Minio, Tencent COS, etc)
\ "s3"
5 / Backblaze B2
\ "b2"
6 / Box
\ "box"
7 / Cache a remote
\ "cache"
8 / Citrix Sharefile
\ "sharefile"
9 / Dropbox
\ "dropbox"
10 / Encrypt/Decrypt a remote
\ "crypt"
11 / FTP Connection
\ "ftp"
12 / Google Cloud Storage (this is not Google Drive)
\ "google cloud storage"
13 / Google Drive
\ "drive"
14 / Google Photos
\ "google photos"
15 / Hubic
\ "hubic"
16 / In memory object storage system.
\ "memory"
17 / Jottacloud
\ "jottacloud"
18 / Koofr
\ "koofr"
19 / Local Disk
\ "local"
20 / Mail.ru Cloud
\ "mailru"
21 / Mega
\ "mega"
22 / Microsoft Azure Blob Storage
\ "azureblob"
23 / Microsoft OneDrive
\ "onedrive"
24 / OpenDrive
\ "opendrive"
25 / OpenStack Swift (Rackspace Cloud Files, Memset Memstore, OVH)
\ "swift"
26 / Pcloud
\ "pcloud"
27 / Put.io
\ "putio"
28 / QingCloud Object Storage
\ "qingstor"
29 / SSH/SFTP Connection
\ "sftp"
30 / Sugarsync
\ "sugarsync"
31 / Tardigrade Decentralized Cloud Storage
\ "tardigrade"
32 / Transparently chunk/split large files
\ "chunker"
33 / Union merges the contents of several upstream fs
\ "union"
34 / Webdav
\ "webdav"
35 / Yandex Disk
\ "yandex"
36 / http Connection
\ "http"
37 / premiumize.me
\ "premiumizeme"
38 / seafile
\ "seafile"

Storage> 13

** See help for drive backend at: https://rclone.org/drive/ **

Google Application Client Id
Setting your own is recommended.
See https://rclone.org/drive/#making-your-own-client-id for how to create your own.
If you leave this blank, it will use an internal key which is low performance.
Enter a string value. Press Enter for the default ("").

client_id>
OAuth Client Secret
Leave blank normally.
Enter a string value. Press Enter for the default ("").
client_secret>
Scope that rclone should use when requesting access from drive.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
1 / Full access all files, excluding Application Data Folder.
\ "drive"
2 / Read-only access to file metadata and file contents.
\ "drive.readonly"
/ Access to files created by rclone only.
3 | These are visible in the drive website.
| File authorization is revoked when the user deauthorizes the app.
\ "drive.file"
/ Allows read and write access to the Application Data folder.
4 | This is not visible in the drive website.
\ "drive.appfolder"
/ Allows read-only access to file metadata but
5 | does not allow any access to read or download file content.
\ "drive.metadata.readonly"

scope> 1
ID of the root folder
Leave blank normally.

Fill in to access "Computers" folders (see docs), or for rclone to use
a non root folder as its starting point.

Enter a string value. Press Enter for the default ("").

root_folder_id>

Service Account Credentials JSON file path
Leave blank normally.
Needed only if you want use SA instead of interactive login.

Leading `~` will be expanded in the file name as will environment variables such as `${RCLONE_CONFIG_DIR}`.

Enter a string value. Press Enter for the default ("").
service_account_file>

Edit advanced config? (y/n)
y) Yes
n) No (default)
y/n> n

Remote config
Use auto config?
* Say Y if not sure
* Say N if you are working on a remote or headless machine
y) Yes (default)
n) No
y/n>  y/n> n
Please go to the following link: https://accounts.google.com/o/oauth2/auth .....  Log in and authorize rclone for access
Enter verification code>

When you visit the webpage to authorize rclone for access you will be presented with the following. If you do not allow rclone this access, it will not be able to transfer files to and from your google drive and Koa. Upon allowing, you are given a code you will copy on the "Enter verification code" line of the rclone configuration. At any time you wish to revoke rclones access to your google drive, you can revokes it access from your google account 3rd party access settings. If you do revoke access to rclone, you will also want to remove it from your rclone configuration.

Continue Setting up rclone with Google Drive

Listing Files

The list file command for rclone takes the following form: rclone lsf remote:path [flags]

Remote is the name of the drive added to our accounts rclone configuration, which was UH_drive in the previous example, and the path would be if we plan to navigate to a sub directory in our google drive we linked.

Transfering Files with the Copy Command

The ‘rclone copy’ command is the way to move files to or from GDrive.

The copy command on a folder will overwrite files that have the same name but if a files exists on the destination that isn’t in the folder being copied it will be retained on the destination.

Upload Directory to Google Drive

Let us assume we have a directory named, test in our home directory on Koa, that contains some important data that we want to upload to google drive as a temporary backup. rclone has a command, called copy, which allows you to copy local data to a location in google drive.

Copy to Google Drive

Download a Directory from Google Drive

Let us assume we want to download our Thesis directory from google drive to Koa. We can use the copy command again to do this.