...

Anaconda

Installing/creating a ipyrad virtual environment

...

On Koa, modules for Anaconda exist.  These modules are provided so that users do not need to install their own copy of anaconda or miniconda.  While this may provide convenience  to many users, for those that require full control over their conda version, please look at the MiniConda guide.

While the modules for Anaconda can simplify using conda install software, it does have at least two things that you should be aware of:

1. The conda version is updated using newer modules, so some packages may be out of date compared to the latest available version
2. Not all of the latest commands work with the Anaconda modules.  The most visible case of this is how one activates and deactivates a conda environment.

To build an conda environment on Koa, one would first start an interactive session and then load the specific Anaconda module they wish to use.  Take note of the module you use, since you will need to load the anaconda module each time you wish to use anaconda and you may want to use the same or newer version.

Code Block
language bash theme Midnight
[user@login001 ~]$ srun -p sandbox --mem=6G -c 2 -t 60 --pty

...

/bin/bash

...

Loading & executing the ipyrad environment

...

srun -p sandbox --mem=6G -c 2 -t 60 --pty /bin/bash
module load lang/Anaconda2
source activate ipyrad
export HOME=/tmp/ (ipyrad specific)
ipcluster start --n ${SLURM_CPUS_PER_TASK} --daemonize (ipyrad specific)
sleep 200 (ipyrad specific)
ipyrad -p params.txt -s 1234567 -c ${SLURM_CPUS_PER_TASK} --ipcluster (ipyrad specific)

NOTE: Contrary to what Anaconda might say after you create a new environment you need to use source activate and NOT conda activate on Mana.

[user@node-0005 ~]$ module load lang/Anaconda3 [user@node-0005 ~]$ mamba create -n ipyrad -c conda-forge -c bioconda ipyrad

Once anaconda is loaded, you will now be able to create an anaconda environment, using the conda  create command.  In the example below, we will be naming our conda environment "test" and temporarily adding the conda channel of "conda-forge", while trying to install "

Code Block
language bash theme Midnight
[user@node-0005 ~]$ mamba create -n test -c conda-forge -c conda python=3.8 matplotlib scipy ipython

Once the environment is installed, it can be activated using "source activate test", or the generic form of "source activate [name of conda environment ]".

Code Block
language bash theme Midnight
[user@node-0005 ~]$ source activate test (test) [user@node-0005 ~]$ conda deactivate [user@node-0005 ~]$

MiniConda

While a base anaconda module exists for users to access and build their own conda environments, this is not the only way.  Below you will find instructions created by Eric Firing covering an alternate method in which you can use miniconda on Mana Koa instead of the Anaconda module.

...

1. The user has no ability to install system-level packages. Instead, the administrators install packages together with modules that the user can load to put those packages on the user’s path, thereby making them accessible.

2. The user cannot write to the user’s own .bashrc file, even though the standard Linux permissions for the file show the file as user-writable. Writing is blocked at a different level.

3. Some libraries must be compiled specifically for HPC use, so packages from conda-forge using the Message Passing Interface (MPI) might not work correctly. See this note on MPI in conda-forge. I don’t know how this applies to the ManaKoa.

The Mana Koa admins have created a module giving access to a base Anaconda environment at the system level, meaning the user cannot change anything in that environment. It is possibly, however, to use that as a base environment for the creation of new working environments in the user’s directory tree (specifically, hidden in the ~/.conda/ subdirectory). This still has some disadvantages compared to using a normal Miniconda installation; primarily, that conda itself (which must live in the base environment) cannot be updated, and that the newer method of activating and deactivating environments with conda is unavailable.

...

. /optusr/ohpcshare/admin/lmod/lmod/init/zsh # Needed to run modules in zsh.

Now log out completely, log back in, and start an interactive session on a compute node. Download the Miniconda installer:

...

conda create -n py38 python=3.8 matplotlib scipy ipython 
conda activate py38

Of course, any time you can use conda to add more packages, use pip to install packages that are not available from conda-forge, make and populate new environments, etc. Just be careful to activate whichever environment you want to work with.

...

for example, from your zsh shell, regardless of whether any other environment is active.

...

External Links

Using Intel optimized packages in Anaconda

Intel Anaconda Channel

...