Module Class

The get started we will show how to use the Module class to interact with the Lmod module system. First off we need to import the Module class using from lmod.module import Module

In this simple example we can pass GCCcore/8.3.0 as module name to the class. The class accepts string or list. We can use the get_command() method to get the module command that would be run.

>>> from lmod.module import Module
>>> a = Module("GCCcore/8.3.0")
>>> a.get_command()
'module purge && module load GCCcore/8.3.0  '

Similarly, we can pass a list of modules to the class

>>> b = Module(["GCCcore/8.3.0", "zlib"])
>>> b.get_command()
'module purge && module load GCCcore/8.3.0 &&  module load zlib  '

You can get the same behavior if you pass modules by a string separated by a space

>>> b = Module("GCCcore/8.3.0 zlib")
>>> b.get_command()
'module purge && module load GCCcore/8.3.0 &&  module load zlib  '

Testing Modules

You can use the test_modules() method to test if modules are loaded properly. The method will return an exit code.

>>> b.test_modules()
0

By default tests are run in a sub-shell if you are interested in running test in a login shell see Testing Modules in Login Shell

Enabling Debug

To enable debug you can pass debug=True to the Module class. This works with most methods.

>>> a = Module("GCCcore/8.3.0",debug=True)
>>> a.test_modules()
[DEBUG] Executing module command: module purge && module load GCCcore/8.3.0
[DEBUG] Return Code: 0
0

Debug works on user collection methods as well

Testing Modules in Login Shell

If you would like to test your module in a login shell you can pass login=True to test_modules method. The test will be conducted using bash

The format of the test will be as follows:

bash -l -c "<command>"

Shown below is GCCcore/8.3.0 tested using a login shell. You may get different results if MODULEPATH is different in your current shell as pose to when you login.

>>> a = Module("GCCcore/8.3.0",debug=True)
>>> a.test_modules(login=True)
[DEBUG] Executing module command: bash -l -c "module purge && module load GCCcore/8.3.0  "
[DEBUG] Return Code: 0
0

Saving Modules to User Collection

Modules can be stored into a user collection using the save() method. If no arguments are passed in, it will save your modules to the default collection. This is equivalent to running module save

>>> b.save()
Saving modules ['GCCcore/8.3.0', 'zlib'] to module collection name: default
Saved current collection of modules to: "default"

Likewise you can specify a collection name by passing a name to the save() method.

>>> b.save("gcc_zlib")
Saving modules ['GCCcore/8.3.0', 'zlib'] to module collection name: gcc_zlib
Saved current collection of modules to: "gcc_zlib"

Show Modules associated to a User Collection

You can view the user collection using describe() method. If no argument is passed in, it will show the default collection

>>> b.describe()
Collection "default" contains:
   1) GCCcore/8.3.0    2) zlib

Similarly, you can pass a collection name to describe() method to view a particular collection.

>>> b.describe("gcc_zlib")
Collection "gcc_zlib" contains:
   1) GCCcore/8.3.0    2) zlib

Get collection command

The get_collection() method can fetch the command to restore the user collection. If no argument is passed in it will resort to the default collection

>>> b.get_collection()
'module restore default'

Likewise you can pass a collection name to get_collection method to fetch any collection name.

>>> b.get_collection("gcc_zlib")
'module restore gcc_zlib'

Testing a User Collection

We can also test if a user collection is working. This can be done using the test_collection method. The method will return the exit code of the command which can be useful for testing output validity.

>>> b.test_collection()
0
>>> b.test_collection("xyz")
1

Tweaking Module Purge Behavior

By default, when you pass modules to Module class, it will purge the modules. You can tweak this behavior by passing the purge=False option to Module. By default purge is set to True

>>> c = Module("OpenMPI/3.0.0", purge=False)
>>> c.get_command()
'module load OpenMPI/3.0.0  '

Enable Force Purge

You can force purge modules by passing force=True. This will purge sticky modules that may be setup in your site.

>>> c = Module("OpenMPI/3.0.0", force=True)
>>> c.get_command()
'module --force purge &&  module load OpenMPI/3.0.0  '

Note if you set purge=False and also pass force=True to the class, it will not purge any modules. Purge takes precedence over force.

>>> c = Module("OpenMPI/3.0.0", purge=False, force=True)
>>> c.get_command()
'module load OpenMPI/3.0.0  '
>>> a.describe()
[DEBUG] Executing module command: module describe default
[DEBUG] Return Code: 0
Collection "default" contains:
   1) GCCcore/8.3.0    2) zlib

>>> a.test_collection()
[DEBUG] Executing command: module restore default
[DEBUG] Return Code: 0
0

>>> a.save("GCC")
[DEBUG] Executing module command: module purge && module load GCCcore/8.3.0   && module save GCC
[DEBUG] Return Code: 0
Saving modules ['GCCcore/8.3.0'] to module collection name: GCC
Saved current collection of modules to: "GCC"

The Module class will throw a TypeError if it detects modules are not of type str or list

>>> a=Module(1)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/mxg-hpc/users/ssi29/lmodule/lmod/module.py", line 44, in __init__
    f"Expecting of type 'list' or 'string' for argument modules. Got of type {type(modules)}"
TypeError: Expecting of type 'list' or 'string' for argument modules. Got of type <class 'int'>

Is Module Available?

The module is-avail command can check if a module file is available in your system. The command will return an exit code either 0 or 1. This could be useful in finding module in system before loading them in your script. To demonstrate, we will use the is_avail() method to check for module files.

>>> a = Module()
>>> a.is_avail("GCC")
0

>>> a.is_avail("cuda")
1

Similarly module avail command is mapped to the method avail(). To check if lmod is available (i.e module avail lmod) you can do the following

>>> a = Module()
>>> a.avail("lmod")
['/usr/share/lmod/lmod/modulefiles/Core:', 'lmod']

If you want to get a listing of all modules (i.e module avail), then don’t pass any argument to avail() method.

>>> a = Module()
>>> a.avail()
['/usr/share/lmod/lmod/modulefiles/Core:', 'lmod', 'settarg']

Module Spider

The module spider command can be used to provide extra details for available modules along with details about specific versions and module description. The spider method can be used to mimic this behavior. Running module spider without any arguments will return all available modules in MODULEPATH.

The following snippet below will mimic module spider command and the output is a string type which we can print.

>>> m = Module()
>>> out = m.spider()
>>> print(out)

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
The following is a list of the modules and extensions currently available:
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  lmod: lmod
    Lmod: An Environment Module System

  settarg: settarg

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

To learn more about a package execute:

   $ module spider Foo

where "Foo" is the name of a module.

To find detailed information about a particular package you
must specify the version if there is more than one version:

   $ module spider Foo/11.1

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

If we specify modules during instance creation time, those modules will be used when invoking spider class. In this next example we mimic module spider lmod command.

>>> m = Module("lmod")
>>> out = m.spider()
>>> print(out)

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  lmod: lmod
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
    Description:
      Lmod: An Environment Module System


    This module can be loaded directly: module load lmod

You may specify modules through spider method which must be a string or list type. If you specify an invalid type when specifying modules to spider method then you will get an exception of TypeError as shown below

>>> m = Module()
>>> m.spider(1)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/siddiq90/Documents/github/lmodule/lmod/module.py", line 182, in spider
    raise TypeError(f"{name} must be a string or list")
TypeError: 1 must be a string or list

If you want to specify multiple modules you can do one of the following

>>> m = Module()
>>> out = m.spider("gcc python")
>>> m = Module()
>>> out = m.spider([gcc, python])

If you specify a list, each item will be converted to string before invoking module spider command. If you create a class instance with a list of modules like m = Module("xyz") then a call m.spider() will run module spider xyz. However, you may specify modules in the spider method as an argument to override modules in the class instance for instance we run m.spider("lmod") and we notice its running module spider lmod.

>>> m = Module("xyz")
>>> m.spider()
>>> print(m.spider())
Lmod has detected the following error: Unable to find: "xyz".

>>> print(m.spider("lmod"))

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  lmod: lmod
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
    Description:
      Lmod: An Environment Module System


    This module can be loaded directly: module load lmod

Module Overview

The module overview command can be used to a summary of available modules in your system. The output is slightly different than module avail it will show a count of each modules next to the module name.

Lmodule supports this via overview method that can be invoked via Module class. The following snippet will show an example where we get a summary of all modules in our system.

>>> m = Module()
>>> print(m.overview())

----------------------------------------------------------------------------- /Users/siddiq90/projects/spack/share/spack/lmod/darwin-catalina-x86_64/Core ------------------------------------------------------------------------------
autoconf-archive (1)   automake    (1)   bzip2     (1)   gcc  (2)   gmp (1)   libiconv   (1)   libtool (1)   mpc  (2)   ncurses (1)   pkgconf  (1)   zlib (1)
autoconf         (1)   berkeley-db (1)   diffutils (1)   gdbm (1)   isl (2)   libsigsegv (1)   m4      (1)   mpfr (2)   perl    (1)   readline (1)   zstd (1)

-------------------------------------------------------------------------------------------- /usr/local/Cellar/lmod/8.6.14/modulefiles/Core --------------------------------------------------------------------------------------------
lmod (1)   settarg (1)

You can specify argument to Module class and invoke the overview method which will show summary for specified module. Likewise, you can specify an argument to overview method which can be a list or string type. If you specify an invalid type then you will get a TypeError exception. Shown below are few example usage

>>> m = Module()
>>> print(m.overview(['gcc', 'lmod']))

----------------------------------------------------------------------------- /Users/siddiq90/projects/spack/share/spack/lmod/darwin-catalina-x86_64/Core ------------------------------------------------------------------------------
gcc (2)

-------------------------------------------------------------------------------------------- /usr/local/Cellar/lmod/8.6.14/modulefiles/Core --------------------------------------------------------------------------------------------
lmod (1)


>>> print(m.overview('lmod'))

-------------------------------------------------------------------------------------------- /usr/local/Cellar/lmod/8.6.14/modulefiles/Core --------------------------------------------------------------------------------------------
lmod (1)

>>> print(m.overview(123))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "lmod/module.py", line 219, in overview
    raise TypeError(f"{name} must be a string or list")
TypeError: 123 must be a string or list

Get Lmod Version

You can get the Lmod version by using the version() method.

>>> a = Module()
>>> a.version()
'7.8.16'

Retrieve User Collections

Lmod user collection are typically found in $HOME/.lmod.d and you can get all collections by running module -t savelist.

Similarly, we have a method get_user_collections that can return a list of user collections as shown below

>>> from lmod.module import get_user_collections
>>> get_user_collections()
['GCC', 'Python', 'default', 'gcc_zlib', 'settarg', 'zlib']

This could be used in conjunction with Module class with options like get_collection, test_collection, describe to perform operation on the user collections.

Shown below is an example of showing all user collections in a simple for-loop using the describe method from Module class

>>> for collection in get_user_collections():
...     Module().describe(collection)
...
Collection "GCC" contains:
   1) GCCcore/8.3.0

Collection "Python" contains:
   1) GCCcore/8.3.0                    7)  SQLite/3.29.0-GCCcore-8.3.0
   2) bzip2/1.0.8-GCCcore-8.3.0        8)  XZ/5.2.4-GCCcore-8.3.0
   3) zlib/1.2.11-GCCcore-8.3.0        9)  GMP/6.1.2-GCCcore-8.3.0
   4) ncurses/6.1-GCCcore-8.3.0        10) libffi/3.2.1-GCCcore-8.3.0
   5) libreadline/8.0-GCCcore-8.3.0    11) Python
   6) Tcl/8.6.9-GCCcore-8.3.0

Collection "default" contains:
   1) settarg

Collection "gcc_zlib" contains:
   1) GCCcore/8.3.0    2) zlib

Collection "settarg" contains:
   1) settarg

Collection "zlib" contains:
   1) zlib

Likewise, we can easily test all user collection using test_collection which gives opportunity to ensure all your user collection are valid before using them in your script

>>> for collection in get_user_collections():
...     Module(debug=True).test_collection(collection)
...
[DEBUG] Executing command: bash -l -c 'module restore GCC'
[DEBUG] Return Code: 0
0
[DEBUG] Executing command: bash -l -c  'module restore Python'
[DEBUG] Return Code: 0
0
[DEBUG] Executing command: bash -l -c  'module restore default'
[DEBUG] Return Code: 0
0
[DEBUG] Executing command: bash -l -c  'module restore gcc_zlib'
[DEBUG] Return Code: 0
0
[DEBUG] Executing command: bash -l -c  'module restore settarg'
[DEBUG] Return Code: 0
0
[DEBUG] Executing command: bash -l -c  'module restore zlib'
[DEBUG] Return Code: 0
0

Check Syntax Error in modulefile

Lmod supports ability to check for syntax error in modulefile via module --checkSyntax load command. This is useful when you are writing a modulefile and you want to ensure there is no syntax error before you load the modulefile. We can do this via Lmodule api as shown below. The checkSyntax method can be used to perform such operation.

>>> from lmod.module import Module
>>> m = Module('lmod')
>>> m.checkSyntax()
0

You can also overide the default modules in the method definition by doing the following. Note that the argument must be a string

>>> from lmod.module import Module
>>> m = Module()
>>> m.checkSyntax('lmod')
0

If you specify an invalid type other than string you will get an exception of TypeError

>>> from lmod.module import Module
>>> m = Module()
>>> m.checkSyntax(1)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/siddiq90/Documents/github/lmodule/lmod/module.py", line 219, in checkSyntax
    raise TypeError(f"{name} must be a string")
TypeError: 1 must be a string