Use Environment Module to Manage Software Packages and Environment Variables in Linux

Motivation

When using the university’s computing cluster, it is very convenient to use the module load command to load software packages such as the Intel compiler and Python Anaconda. These packages can also be easily discarded when they are not needed, which makes it convenient to select and enable certain environment variables, and prevent certain environment variables from contaminating the compilation environment of other software.

On one’s own computer, in order to install and compile various programs, it is sometimes necessary to download and install many compilers and software packages. As a result of the impact of environment variables, compilers or programs often report errors during compilation or execution. Therefore, I plan to install modules on my own computer to manage these software packages and environment variables.

Installation and Working Mechanism of module Tool

Installation

For Debian-based systems such as Ubuntu, it can be directly installed using apt:

1

sudo apt install environment-modules

Generally, it will be installed by default under /usr/share/modules. The modulefiles folder under this path will store some default module files:

dot module-git module-info modules null use.own

Users can view them by using the module avail command.

If you find that the module command cannot be used, it is because it has not been initialized yet. Under /usr/share/modules/init, there are initialization tools for various shells such as bash, ksh, fish, tcsh, and zsh. For example, if I use the module tool in zsh, I need to run the following command to complete the initialization:

1

source /usr/share/modules/init/zsh

Since the above command needs to be re-entered every time the system or zsh is re-logged in, for convenience, it can be added to “~/.zshrc”.

Working Mechanism

The module tool lists and loads user-specified software packages and environment variables by recognizing module files. A typical module file content is shown below:

#%Module

proc ModulesHelp { } { puts stderr “This module adds solar to your path” }

module-whatis “This module adds solar to your path\n”

set basedir “/home/aturing/software/solar-1.2”
prepend-path PATH “${basedir}/bin”
prepend-path LD_LIBRARY_PATH “${basedir}/lib64”
module load intel/19.1/64/19.1.1.217
module load intel-mpi/intel/2019.7/64

After installing module, it will look for modulefiles in the default path /usr/share/modules/modulefiles. When users use the module avail command to view available modules, the module tool will display the modulefiles under the /usr/share/modules/modulefiles path. After installing the module tool, the default modules are usually displayed as follows:

——- /usr/share/modules/modulefiles ——–
dot module-git module-info modules null use.own

If the user uses the module load use.own command to load use.own, the module tool will create a privatemodules folder in the user’s home directory, and the user can place custom modulefiles in this folder. After that, modulefiles located in the ~/privatemodules directory will also be detected by the module tool.

Installing Packages and Managing with module

Installing Intel’s C++ compiler icc

Intel provides its own C++ compiler icc and Fortran compiler ifort for free to users of Intel chips, which can be downloaded and installed on the official website of Intel. Intel offers multiple installation methods, including oneAPI which integrates multiple Intel compilers and tools, as well as standalone versions of these compilers and tools. The full version of oneAPI toolkit is very large, but since I only need icc and ifort compilers at present, I only downloaded and installed these two standalone compilers.

Download the installation package

The standalone icc compiler installation package can be downloaded from the Intel website at the following URL: https://www.intel.com/content/www/us/en/developer/articles/tool/oneapi-standalone-components.html#dpcpp-cpp. For each of the Linux/Windows/macOS systems, there are both online and offline installation packages available. You can choose the online version of the installation package. For example, the name of the installation package I downloaded is l_dpcpp-cpp-compiler_p_2022.1.0.137.sh.

Alternatively, you can download the package using the command line:
1

wget https://registrationcenter-download.intel.com/akdlm/irc_nas/18717/l_dpcpp-cpp-compiler_p_2022.1.0.137.sh
Run the installation package

After downloading, navigate to the directory where the installation package is saved and use the following command to install:
1

sudo sh ./l_dpcpp-cpp-compiler_p_2022.1.0.137.sh -a -s --eula accept
Here, -a indicates the use of command-line parameters; -s indicates a silent installation, i.e., no installation window is displayed; --eula accept indicates that you accept the user agreement. For more details, please refer to the Intel installation guide.

The compiler is installed by default in the /opt/intel/oneapi directory.
Configure Modulefiles

Intel has integrated a script in the installation package that can automatically generate modulefiles. Users only need to run the script, and the installed compiler or toolkit will be automatically written into modulefiles that can be used by the module tool.

The configuration script is located in the installation directory /opt/intel/oneapi, and the modulefiles-setup.sh file is used. When running this script, users can specify the folder where the generated modulefiles are placed. Here, I choose to put the generated modulefiles in the ~/privatemodules directory:
1

/opt/intel/oneapi/modulefiles-setup.sh --output-dir=$HOME/privatemodules

View and Load the Compiler

After completing the above steps, you can use the module avail command to view the icc compiler that was just installed. After running the module avail command, you will see output similar to the following:

1
2
3
4
5
6
7
8


------------------------------------------------ /usr/share/modules/modulefiles ------------------------------------------------  
dot  module-git  module-info  modules  null  use.own    

------------------------------------------------- /home/lijin/privatemodules ---------------------------------------------------  
compiler-rt/2022.1.0    compiler/2022.1.0    debugger/2021.6.0       icc/2022.1.0    init_opencl/2022.1.0  tbb/2021.6.0    
compiler-rt/latest      compiler/latest      debugger/latest         icc/latest      init_opencl/latest    tbb/latest      
compiler-rt32/2022.1.0  compiler32/2022.1.0  dev-utilities/2021.6.0  icc32/2022.1.0  oclfpga/2022.1.0      tbb32/2021.6.0  
compiler-rt32/latest    compiler32/latest    dev-utilities/latest    icc32/latest    oclfpga/latest        tbb32/latest    

Since I have also installed some other tools, in addition to icc, other modules are also displayed.

Then, use module load icc/latest to load the latest installed version of the icc compiler!

Installing Intel Fortran Compiler ifort

The steps to install the ifort compiler are similar to the above steps, and will not be repeated here. The default installation directory for the ifort compiler is also /opt/intel/oneapi. After installation, run the following command again to update the modulefiles in the ~/privatemodules directory:

1

/opt/intel/oneapi/modulefiles-setup.sh --output-dir=$HOME/privatemodules