...
Each user is provided a home storage location on Koa. Each user's home is provided 50 GB of space that can be used as the user chooses. Upon initialization, user homes are setup with an examples directory, a symlink for Koa Scratch and other basic configurations needed to allow a users to take advantage of Koa.
...
Home is not a high performance filesystem. Network File System (NFS) is tried a and true but it is not designed to handle the stresses an High Performance Computing cluster can place on it. As a result, home should not be used for writing output from multiple jobs or jobs that can generate a lot of data. Jobs of this nature should take advantage of Scratch instead, which is a file system designed for the stresses an HPC can generate.
...
Details
Home is utilizing a Zoned File System (ZFS) over NFS Remote Direct Access (RDMA). Currently, home is served out utilizing a single Virtual Host that is connected to Koa at 200 Gbit. ZFS is configured to use zstd-3 compression allowing users to take full advantage of the inline compression and space savings zstd provides. As of this writing, on average users see a 2.16x compression ratio.
...
Scratch, also known by the symlink “koa_scratch”, provides each user access to a an 800TB pool of storage on which files may live 90 days on from the last time they were modified. In total, this file system can support up to 400,000,000 files and directories. Users are not provided an individual quota allowing for flexibility based on need. Scratch directly access accesses the underlying Koa Storage system providing the high performance possible from the storage system.
...
Scratch provides direct access to the underlying high performance file system: Koa Storage, which utilizes Lustre. Lustre is designed for situations where many servers and workloads are needing need to read and write data as quickly as possible. While Lustre works best with long sequential read/writes, and exhibits poorer performance with small random reads, but methods exist to work around this limitation. Work around Workaround would include the use of squashfs as we cover in our documentation or archive formats that combine multiple small files that align with your workflow, such as HDF5.
For scalability and performance, files written to Scratch use a progressive file layout in which at certain size boundaries switch to different types of storage medium or recruits recruit more storage targets to store parts of a file.
...
| Note |
|---|
Scratch is not a persistent storage location for users data. Scratch provides a 90-day grace period after a file was last written to before the file is removed automatically from the file system. |
| Warning |
|---|
The purge process cannot be paused for individual users and files that are removed cannot be recovered. |
Parameters of the automated file purge
...
Scratch, or more specifically, Koa Storage utilizes Lustre with ZFS as its underlying file system. For transport to Koa, it also utilizes RDMA using multiple servers all connected at 200Gbit Infiniband. The ZFS components are setup to provide the same zstd-3 compression seen on the home file system providing space savings and in some cases faster access to file files as you are needing need to read less data from a slower storage medium. The underlying storage system utilizes a mixture of spinning enterprise hard drives (HDD) and Solid State storage, (SAS SSD and NVMe).
...
Metadata is stored in separate targets from the files file data, which for performance reason reasons is entirely using Solid State Drives. The meta data metadata is split up among multiple targets, which are then served out by different servers. In case of a server failure, “failing-over” the storage to another server is possible allowing for minimal down timedowntime. Each folder is assigned to one of the Metadata storage targets and all files under it are also assigned to that target. Load balancing is done in some cases to try and balancing the different meta data metadata targets so they do not grow too out of sync size wise.
...
| Info |
|---|
Of the current storage (7 PB), about 1/7th of is either solid state (SAS SSD or NVMe) |
Storage targets are split up into different configurations providing at least a 2 disk parity. Current Currently, Koa has 58 Object Storage Targets, of which 10 targets are SAS SSD or NVMe.
Object data is written to Scratch using a progressive file layout (PFL). The currently current PFL setup used by scratch follows the following rules:
...