...
For scalability and performance, files written to Scratch use a progressive file layout in which at certain size boundaries switch to different types of storage or recruits more storage targets to storage parts of a file.
Automatic File Purging
| Note |
|---|
Scratch is not a persistent storage location for users data. Scratch provides a 90 day grace period after a file was last written to before the file is removed automatically from the file system. |
| Warning |
|---|
The purge process cannot be paused for individual users and files that are removed cannot be recovered |
Parameters of the automated file purge
Only files under ~/koa_scratch/ or /mnt/lustre/koa/scratch/${USER} will be subject to purge
Files and folders not modified for 90 days will be deleted from scratch
The purge process will run daily The process cannot be paused for individual users and files that are removed cannot be recovered
| Info |
|---|
In the case the file system |
...
nears 85-90% |
...
utilized, ITS-CI will contact users who have large occupancy of |
...
scratch to voluntarily reduce their usage |
...
. If we are unable to reclaim enough space to drop below 70% utilized, we will purge files from oldest to newest, regardless of time on scratch until we |
...
are below 70% usage |
...
. |
Details
Scratch, or more specifically, Koa Storage utilizes Lustre with ZFS as its underlying file system. For transport to Koa, it also utilizes RDMA using multiple servers all connected at 200Gbit Infiniband. The ZFS components are setup to provide the same zstd-3 compression seen on the home file system providing space savings and in some cases faster access to file as you are needing to read less data from slower storage medium. The underlying storage system utilizes a mixture of spinning enterprise hard drives (HDD) and Solid State storage, (SAS SSD and NVMe).
...
File system | Storage Quota | Files and & Directories (inode) Quota | Compression | Persistent |
|---|---|---|---|---|
Lustre + ZFS | 800 TB | 400,000,000 | zstd-3 | No (90 day purge) |
...