...
| Info |
|---|
Usage of squashFS in this manner has also been tested and shown to be beneficial for read only databases utilized by the NCBI bioinformatics tool , Blast. https://arxiv.org/abs/2002.06129 |
Building a squashfs file
On Koa, users cane create a squashfs archive using the mksquashfs on a compute node. For example, let us assume we have a folder located at ~/koa_scratch/database which contains 10K files with an average file size of 512K. I could create a single file, let us call it database.sqfs and also save it in ~/koa_scratch with the following command:
| Code Block | ||
|---|---|---|
| ||
mksquashfs ~/koa_scratch/database ~/koa_scratch/database.sqfs -comp zstd -Xcompression-level 3 -b 1M -no-xattrs |
Mounting and using a squashfs file
Once the archive is created, one would be able to mount and access the archive on a per node basis by use the squashfuse_ll command. While one could use just squashfuse we recommend the _ll variant as it allows you to setup an automatic clean up of the mount when it hasn’t been accessed for a while. This is useful in the case you have multiple jobs using the same node all accessing the same archive from the same location. The automatic unmount via timeout would allow your jobs to not unmount the squash archive yet it will eventually be removed automatically if it no longer see any other file accesses.
...