Working with Lmod Spider

lmodule provides an API to retrieve results from spider command. To get started you need to import the Spider class as follows

>>> from lmod.spider import Spider

To create a Spider object you can do the following

>>> a = Spider()

Getting Unique Software

If you want to get a list of unique software names, you can use the get_names() method.

>>> a = Spider()
>>> a.get_names()
['Anaconda3', 'Autoconf', 'Automake', 'Autotools', 'Bison', 'CMake', 'CUDA', 'Clang', 'EasyBuild', 'FFTW', 'GCC', 'GCCcore', 'GMP', 'M4', 'OSU-Micro-Benchmarks', 'OpenBLAS', 'OpenMPI', 'PyCharm', 'Python', 'SQLite', 'ScaLAPACK', 'Tcl', 'XZ', 'binutils', 'bzip2', 'diffutils', 'flex', 'foss', 'gdbm', 'gettext', 'gompi', 'help2man', 'hwloc', 'libffi', 'libiconv', 'libpciaccess', 'libreadline', 'libsigsegv', 'libtool', 'libxml2', 'lmod', 'm4', 'ncurses', 'numactl', 'pkgconf', 'readline', 'settarg', 'tar', 'util-macros', 'xorg-macros', 'xz', 'zlib']

This would be all unique software based on value of MODULEPATH. We can also tweak which module trees to search with spider as follows.

>>> b = Spider("/usr/share/lmod/lmod/modulefiles/Core")
>>> b.get_names()
['lmod', 'settarg']

Get Module Trees

You can get all module trees used by spider command using the method get_trees().

>>> a.get_trees()
'/usr/share/lmod/lmod/modulefiles/Core'

Retrieve all Module Names

We can retrieve a list of full canonical module names by using get_modules() method. The output is returned as a dictionary with key representing the modulefile and value is the full canonical module name.

>>> >>> b = Spider("/usr/share/lmod/lmod/modulefiles/Core")
>>> b.get_modules()
{'/usr/share/lmod/lmod/modulefiles/Core/lmod.lua': 'lmod', '/usr/share/lmod/lmod/modulefiles/Core/settarg.lua': 'settarg'}

You may filter result by module name (i.e result from get_names()) when using get_modules(). For instance if you want to see all CUDA and lmod modules you can do the following

>>> c = Spider()
>>> c.get_modules(["lmod","CUDA"])
{'/mxg-hpc/users/ssi29/easybuild/modules/all/CUDA/10.0.130.lua': 'CUDA/10.0.130', '/mxg-hpc/users/ssi29/easybuild/modules/all/CUDA/9.2.148.1.lua': 'CUDA/9.2.148.1', '/usr/share/lmod/lmod/modulefiles/Core/lmod.lua': 'lmod'}

Note

You must pass values as a list when filtering by module names

Retrieve all Parent Modules

We can retrieve parent modules (modules that set MODULEPATH) to other trees by using method get_parents. This is useful for finding modules in Hierarchical Module Naming Scheme.

>>> e = Spider()
>>> e.get_parents()
['GCCcore/8.1.0', 'GCCcore/9.2.0']

In this case we know that loading GCCcore/8.1.0 will set MODULEPATH to another module tree. Notice in command below we see prepend_path("MODULEPATH",... is set to indicate another tree is added to MODULEPATH. This helps users and site-administrator to find all sub-trees in your software stack.

$ module --redirect show GCCcore/8.1.0 | grep MODULEPATH
prepend_path("MODULEPATH","/mxg-hpc/users/ssi29/easybuild-HMNS/modules/all/Compiler/GCCcore/8.1.0")

Getting all versions of a particular software

We can retrieve a list of all versions of a particular software using the get_all_version method which takes an argument for the software name. Shown below we can query all versions of the GCC module.

>>> e.get_all_versions("GCC")
['6.4.0-2.28', '7.1.0-2.28', '9.2.0-2.32', '8.1.0-2.30', '8.1.0-2.30', '8.3.0', '7.4.0-2.31.1']