...
The two additional storage locations could be granted access to include Lab storage (free) and KoaStore storage (for fee).
...
Table of Contents | ||||||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
...
Home
Each user is provided a home storage location on Koa. Each user's home is provided 50 GB of space that can be used as the user chooses. Upon initialization user homes are setup with an examples directory, a symlink for Koa Scratch and other basic configurations needed to allow a users to take advantage of Koa.
...
Scratch provides direct access to the underlying high performance file system: Koa Storage, which utilizes Lustre. Lustre is designed for situations where many servers and workloads are needing to read and write data as quickly as possible. While Lustre works best with long sequential reads and writes, read/writes, and exhibits poorer performance with small random reads, but methods exist to work around this limitation when the need to read as input a lot of small files, such as using squashfs. Work around would include the use of squashfs as we cover in our documentation or archive formats that combine multiple small files that align with your workflow, such as HDF5.
For scalability and performance, files written to Scratch use a progressive file layout in which at certain size boundaries switch to different types of storage or recruits more storage targets overall are recruited to storage parts of a file.
...
The first 512K of every file is written to a single solid state (SSD or NVMe) storage target
Next, file data up to 64MB is written to a single HDD target
Next, file data up to 512MB is written to two HDD targets in 4 MB stripes
Next, file data up to 1 GB is written to four HDD targets in 4 MB stripes
Finally, any remaining data for a file is written to eight HDD targets in 4 MB stripes
File system | Total Storage Quota | Max Files and Directories Quota | Compression | Persistent |
---|---|---|---|---|
Lustre + ZFS | 800 TB | 400,000,000 | zstd-3 | No (90 day purge) |
...