...
Scratch, or more specifically, Koa Storage utilizes Lustre with ZFS as its underlying file system. For transport to Koa, it also utilizes RDMA using multiple servers all connected at 200Gbit Infiniband. The ZFS components are setup to provide the same zstd-3 compression seen on the home file system providing space savings and in some cases faster access to file as you are needing to read less data from slower storage medium. The underlying storage system utilizes a mixture of spinning enterprise hard drives (HDD) and Solid State storage, (SAS SSD and NVMe).
...
Metadata
Meta data Metadata is stored in separate targets from the files data, which for performance reason is entirely using Solid State Drives. The meta data is split up among multiple targets, which are then served out by different servers. In case of a server failure, “failing-over” the storage to another server is possible allowing for minimal down time. Each folder is assigned to one of the Meta Data Metadata storage targets and all files under it are also assigned to that target. Load balancing is done in some cases to try and balancing the different meta data targets so they do not grow too out of sync size wise.
...
Storage targets are split up into different configurations providing at least a 2 disk parity. Current Koa has 58 Object Storage Targets, of which 10 targets are SAS SSD or NVMe.
Object Data data is written to Scratch using a progressive file layout used for scratch is currently setup to follow (PFL). The currently PFL setup used by scratch follows the following rules:
The first 512K of every file is written to a single solid state (SSD or NVMe) storage target
Next, file data up to 64MB is written to a single HDD target
Next, file data up to 512MB is written to two HDD targets in 4 MB stripes
Next, file data up to 1 GB is written to four HDD targets in 4 MB stripes
Finally, any remaining data for a file is written to eight HDD targets in 4 MB stripes
...