Module Class¶
The get started we will show how to use the Module
class to interact with the Lmod module system. First
off we need to import the Module class using from lmod.module import Module
In this simple example we can pass GCCcore/8.3.0
as module name to the class. The class accepts string or
list. We can use the get_command()
method to get the module command that would be run.
>>> from lmod.module import Module
>>> a = Module("GCCcore/8.3.0")
>>> a.get_command()
'module purge && module load GCCcore/8.3.0 '
Similarly, we can pass a list of modules to the class
>>> b = Module(["GCCcore/8.3.0", "zlib"])
>>> b.get_command()
'module purge && module load GCCcore/8.3.0 && module load zlib '
You can get the same behavior if you pass modules by a string separated by a space
>>> b = Module("GCCcore/8.3.0 zlib")
>>> b.get_command()
'module purge && module load GCCcore/8.3.0 && module load zlib '
Testing Modules¶
You can use the test_modules()
method to test if modules are loaded properly. The method will return an exit
code.
>>> b.test_modules()
0
By default tests are run in a sub-shell if you are interested in running test in a login shell see Testing Modules in Login Shell
Enabling Debug¶
To enable debug you can pass debug=True
to the Module
class. This works with most methods.
>>> a = Module("GCCcore/8.3.0",debug=True)
>>> a.test_modules()
[DEBUG] Executing module command: module purge && module load GCCcore/8.3.0
[DEBUG] Return Code: 0
0
Debug works on user collection methods as well
Testing Modules in Login Shell¶
If you would like to test your module in a login shell you can pass login=True
to test_modules
method. The
test will be conducted using bash
The format of the test will be as follows:
bash -l -c "<command>"
Shown below is GCCcore/8.3.0 tested using a login shell. You may get different results if MODULEPATH is different in your current shell as pose to when you login.
>>> a = Module("GCCcore/8.3.0",debug=True)
>>> a.test_modules(login=True)
[DEBUG] Executing module command: bash -l -c "module purge && module load GCCcore/8.3.0 "
[DEBUG] Return Code: 0
0
Saving Modules to User Collection¶
Modules can be stored into a user collection using the save()
method. If no arguments are passed in, it will
save your modules to the default
collection. This is equivalent to running module save
>>> b.save()
Saving modules ['GCCcore/8.3.0', 'zlib'] to module collection name: default
Saved current collection of modules to: "default"
Likewise you can specify a collection name by passing a name to the save()
method.
>>> b.save("gcc_zlib")
Saving modules ['GCCcore/8.3.0', 'zlib'] to module collection name: gcc_zlib
Saved current collection of modules to: "gcc_zlib"
Show Modules associated to a User Collection¶
You can view the user collection using describe()
method. If no argument is passed in, it will show
the default
collection
>>> b.describe()
Collection "default" contains:
1) GCCcore/8.3.0 2) zlib
Similarly, you can pass a collection name to describe()
method to view a particular collection.
>>> b.describe("gcc_zlib")
Collection "gcc_zlib" contains:
1) GCCcore/8.3.0 2) zlib
Get collection command¶
The get_collection()
method can fetch the command to restore the user collection. If no argument is passed in
it will resort to the default
collection
>>> b.get_collection()
'module restore default'
Likewise you can pass a collection name to get_collection
method to fetch any collection name.
>>> b.get_collection("gcc_zlib")
'module restore gcc_zlib'
Testing a User Collection¶
We can also test if a user collection is working. This can be done using the test_collection
method. The method
will return the exit code of the command which can be useful for testing output validity.
>>> b.test_collection()
0
>>> b.test_collection("xyz")
1
Tweaking Module Purge Behavior¶
By default, when you pass modules to Module
class, it will purge the modules. You can tweak this behavior by passing
the purge=False
option to Module
. By default purge is set to True
>>> c = Module("OpenMPI/3.0.0", purge=False)
>>> c.get_command()
'module load OpenMPI/3.0.0 '
Enable Force Purge¶
You can force purge modules by passing force=True
. This will purge sticky modules that may be setup in your site.
>>> c = Module("OpenMPI/3.0.0", force=True)
>>> c.get_command()
'module --force purge && module load OpenMPI/3.0.0 '
Note if you set purge=False
and also pass force=True
to the class, it will not purge any modules. Purge
takes precedence over force.
>>> c = Module("OpenMPI/3.0.0", purge=False, force=True)
>>> c.get_command()
'module load OpenMPI/3.0.0 '
>>> a.describe()
[DEBUG] Executing module command: module describe default
[DEBUG] Return Code: 0
Collection "default" contains:
1) GCCcore/8.3.0 2) zlib
>>> a.test_collection()
[DEBUG] Executing command: module restore default
[DEBUG] Return Code: 0
0
>>> a.save("GCC")
[DEBUG] Executing module command: module purge && module load GCCcore/8.3.0 && module save GCC
[DEBUG] Return Code: 0
Saving modules ['GCCcore/8.3.0'] to module collection name: GCC
Saved current collection of modules to: "GCC"
The Module class will throw a TypeError
if it detects modules are not of type str
or list
>>> a=Module(1)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/mxg-hpc/users/ssi29/lmodule/lmod/module.py", line 44, in __init__
f"Expecting of type 'list' or 'string' for argument modules. Got of type {type(modules)}"
TypeError: Expecting of type 'list' or 'string' for argument modules. Got of type <class 'int'>
Is Module Available?¶
The module is-avail
command can check if a module file is available in your system. The command will return an
exit code either 0
or 1
. This could be useful in finding module in system before loading them in your script.
To demonstrate, we will use the is_avail()
method to check for module files.
>>> a = Module()
>>> a.is_avail("GCC")
0
>>> a.is_avail("cuda")
1
Similarly module avail
command is mapped to the method avail()
. To check if lmod
is available (i.e module avail lmod
)
you can do the following
>>> a = Module()
>>> a.avail("lmod")
['/usr/share/lmod/lmod/modulefiles/Core:', 'lmod']
If you want to get a listing of all modules (i.e module avail
), then don’t pass any argument to avail()
method.
>>> a = Module()
>>> a.avail()
['/usr/share/lmod/lmod/modulefiles/Core:', 'lmod', 'settarg']
Module Spider¶
The module spider
command can be used to provide extra details for available modules along with details about specific
versions and module description. The spider
method can be used to mimic this behavior. Running module spider
without
any arguments will return all available modules in MODULEPATH.
The following snippet below will mimic module spider
command and the output is a string type which we can print.
>>> m = Module()
>>> out = m.spider()
>>> print(out)
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
The following is a list of the modules and extensions currently available:
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
lmod: lmod
Lmod: An Environment Module System
settarg: settarg
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
To learn more about a package execute:
$ module spider Foo
where "Foo" is the name of a module.
To find detailed information about a particular package you
must specify the version if there is more than one version:
$ module spider Foo/11.1
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
If we specify modules during instance creation time, those modules will be used when invoking spider
class.
In this next example we mimic module spider lmod
command.
>>> m = Module("lmod")
>>> out = m.spider()
>>> print(out)
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
lmod: lmod
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Description:
Lmod: An Environment Module System
This module can be loaded directly: module load lmod
You may specify modules through spider
method which must be a string or list type. If you specify an invalid type when specifying modules
to spider
method then you will get an exception of TypeError as shown below
>>> m = Module()
>>> m.spider(1)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/siddiq90/Documents/github/lmodule/lmod/module.py", line 182, in spider
raise TypeError(f"{name} must be a string or list")
TypeError: 1 must be a string or list
If you want to specify multiple modules you can do one of the following
>>> m = Module()
>>> out = m.spider("gcc python")
>>> m = Module()
>>> out = m.spider([gcc, python])
If you specify a list, each item will be converted to string before invoking module spider
command. If you create
a class instance with a list of modules like m = Module("xyz")
then a call m.spider() will run module spider xyz. However,
you may specify modules in the spider method as an argument to override modules in the class instance for instance we run
m.spider("lmod")
and we notice its running module spider lmod.
>>> m = Module("xyz")
>>> m.spider()
>>> print(m.spider())
Lmod has detected the following error: Unable to find: "xyz".
>>> print(m.spider("lmod"))
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
lmod: lmod
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Description:
Lmod: An Environment Module System
This module can be loaded directly: module load lmod
Module Overview¶
The module overview
command can be used to a summary of available modules in your system. The output is slightly different
than module avail
it will show a count of each modules next to the module name.
Lmodule supports this via overview method that can be invoked via Module class. The following snippet will show an example where we get a summary of all modules in our system.
>>> m = Module()
>>> print(m.overview())
----------------------------------------------------------------------------- /Users/siddiq90/projects/spack/share/spack/lmod/darwin-catalina-x86_64/Core ------------------------------------------------------------------------------
autoconf-archive (1) automake (1) bzip2 (1) gcc (2) gmp (1) libiconv (1) libtool (1) mpc (2) ncurses (1) pkgconf (1) zlib (1)
autoconf (1) berkeley-db (1) diffutils (1) gdbm (1) isl (2) libsigsegv (1) m4 (1) mpfr (2) perl (1) readline (1) zstd (1)
-------------------------------------------------------------------------------------------- /usr/local/Cellar/lmod/8.6.14/modulefiles/Core --------------------------------------------------------------------------------------------
lmod (1) settarg (1)
You can specify argument to Module
class and invoke the overview
method which will
show summary for specified module. Likewise, you can specify an argument to overview
method which can be a list
or string type. If you specify an invalid type then you will get a TypeError exception. Shown below are few example
usage
>>> m = Module()
>>> print(m.overview(['gcc', 'lmod']))
----------------------------------------------------------------------------- /Users/siddiq90/projects/spack/share/spack/lmod/darwin-catalina-x86_64/Core ------------------------------------------------------------------------------
gcc (2)
-------------------------------------------------------------------------------------------- /usr/local/Cellar/lmod/8.6.14/modulefiles/Core --------------------------------------------------------------------------------------------
lmod (1)
>>> print(m.overview('lmod'))
-------------------------------------------------------------------------------------------- /usr/local/Cellar/lmod/8.6.14/modulefiles/Core --------------------------------------------------------------------------------------------
lmod (1)
>>> print(m.overview(123))
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "lmod/module.py", line 219, in overview
raise TypeError(f"{name} must be a string or list")
TypeError: 123 must be a string or list
Get Lmod Version¶
You can get the Lmod version by using the version()
method.
>>> a = Module()
>>> a.version()
'7.8.16'
Retrieve User Collections¶
Lmod user collection are typically found in $HOME/.lmod.d and you can get all collections by running module -t savelist
.
Similarly, we have a method get_user_collections
that can return a list of user collections as shown below
>>> from lmod.module import get_user_collections
>>> get_user_collections()
['GCC', 'Python', 'default', 'gcc_zlib', 'settarg', 'zlib']
This could be used in conjunction with Module
class with options like get_collection
, test_collection
, describe
to perform operation on the user collections.
Shown below is an example of showing all user collections in a simple for-loop using the describe
method from Module class
>>> for collection in get_user_collections():
... Module().describe(collection)
...
Collection "GCC" contains:
1) GCCcore/8.3.0
Collection "Python" contains:
1) GCCcore/8.3.0 7) SQLite/3.29.0-GCCcore-8.3.0
2) bzip2/1.0.8-GCCcore-8.3.0 8) XZ/5.2.4-GCCcore-8.3.0
3) zlib/1.2.11-GCCcore-8.3.0 9) GMP/6.1.2-GCCcore-8.3.0
4) ncurses/6.1-GCCcore-8.3.0 10) libffi/3.2.1-GCCcore-8.3.0
5) libreadline/8.0-GCCcore-8.3.0 11) Python
6) Tcl/8.6.9-GCCcore-8.3.0
Collection "default" contains:
1) settarg
Collection "gcc_zlib" contains:
1) GCCcore/8.3.0 2) zlib
Collection "settarg" contains:
1) settarg
Collection "zlib" contains:
1) zlib
Likewise, we can easily test all user collection using test_collection
which gives opportunity to ensure all your
user collection are valid before using them in your script
>>> for collection in get_user_collections():
... Module(debug=True).test_collection(collection)
...
[DEBUG] Executing command: bash -l -c 'module restore GCC'
[DEBUG] Return Code: 0
0
[DEBUG] Executing command: bash -l -c 'module restore Python'
[DEBUG] Return Code: 0
0
[DEBUG] Executing command: bash -l -c 'module restore default'
[DEBUG] Return Code: 0
0
[DEBUG] Executing command: bash -l -c 'module restore gcc_zlib'
[DEBUG] Return Code: 0
0
[DEBUG] Executing command: bash -l -c 'module restore settarg'
[DEBUG] Return Code: 0
0
[DEBUG] Executing command: bash -l -c 'module restore zlib'
[DEBUG] Return Code: 0
0
Check Syntax Error in modulefile¶
Lmod supports ability to check for syntax error in modulefile via module --checkSyntax load
command. This is useful
when you are writing a modulefile and you want to ensure there is no syntax error before you load the modulefile. We
can do this via Lmodule api as shown below. The checkSyntax method can be used to perform such operation.
>>> from lmod.module import Module
>>> m = Module('lmod')
>>> m.checkSyntax()
0
You can also overide the default modules in the method definition by doing the following. Note that the argument must be a string
>>> from lmod.module import Module
>>> m = Module()
>>> m.checkSyntax('lmod')
0
If you specify an invalid type other than string you will get an exception of TypeError
>>> from lmod.module import Module
>>> m = Module()
>>> m.checkSyntax(1)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/siddiq90/Documents/github/lmodule/lmod/module.py", line 219, in checkSyntax
raise TypeError(f"{name} must be a string")
TypeError: 1 must be a string