Spack Training
Goal
The goal of this training is to provide advice for how one can use Spack to install packages and manage a software stack on Perlmutter. We will cover the following topics:
User Environment
Defining Compilers in Spack
Define Package Preference and Externals
Create a source mirror
Building CUDA packages
Generating modulefiles
After completing the training, one can expect to be familiar with the customizations needed for an optimal Spack experience on Perlmutter.
Pre-Requisite
In order to perform this training, you need a NERSC account and access to Perlmutter. We assume you already have a basic understanding of spack.
Setup
In order to get started, please Connect to Perlmutter via ssh. Once you have access, please
clone the following Git repository into your $HOME directory.
git clone https://github.com/NERSC/spack-infrastructure.git
User Environment
Spack builds can be sensitive to your user environment and any configuration setup in your shell startup files. We recommend you review your startup configuration files. Some things to look out for are the following:
Loading or unloading of any modules
Activating a Python or Conda environment
Any user environment variables such as
$PATH
Note
We have seen that purging modules (module purge) can alter Spack builds and cause most of the Cray programming environment to be removed. For more details see spack/#27124.
When performing Spack builds, we encourage using the default modules. This should look at follows:
elvis@login34> module list
Currently Loaded Modules:
1) craype-x86-milan 4) perftools-base/22.06.0 7) craype/2.7.16 10) cray-libsci/21.08.1.2 13) darshan/3.3.1 (io)
2) libfabric/1.15.0.0 5) xpmem/2.3.2-2.2_7.5__g93dd7ee.shasta 8) cray-dsmml/0.2.2 11) PrgEnv-gnu/8.3.3
3) craype-network-ofi 6) gcc/11.2.0 9) cray-mpich/8.1.17 12) xalt/2.10.2
Where:
io: Input/output software
In order to setup our environment, let’s source the setup script which will create a new Python virtual environment to perform the Spack builds. Please run the following commands:
elvis@login34> cd spack-infrastructure/
elvis@login34> source setup-env.sh
Collecting clingo
Using cached clingo-5.5.2-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.2 MB)
Collecting cffi
Using cached cffi-1.15.1-cp36-cp36m-manylinux_2_5_x86_64.manylinux1_x86_64.whl (402 kB)
Collecting pycparser
Using cached pycparser-2.21-py2.py3-none-any.whl (118 kB)
Installing collected packages: pycparser, cffi, clingo
Successfully installed cffi-1.15.1 clingo-5.5.2 pycparser-2.21
WARNING: You are using pip version 20.2.3; however, version 21.3.1 is available.
You should consider upgrading via the '/global/homes/e/elvis/spack-infrastructure/spack-pyenv/bin/python3 -m pip install --upgrade pip' command.
/global/homes/e/elvis/spack-infrastructure/spack-pyenv/bin/python
Package Version
---------- -------
cffi 1.15.1
clingo 5.5.2
pip 20.2.3
pycparser 2.21
setuptools 44.1.1
WARNING: You are using pip version 20.2.3; however, version 21.3.1 is available.
You should consider upgrading via the '/global/homes/e/elvis/spack-infrastructure/spack-pyenv/bin/python3 -m pip install --upgrade pip' command.
The setup-env.sh script will install clingo in your Python environment which is typically required by Spack along with a few
other configurations relevant for building Spack.
Note
Spack requires clingo in-order to bootstrap clingo however we observed issues where Spack was unable to bootstrap clingo see spack/28315. We found that installing clingo as a Python package addressed the issue.
Acquiring Spack
Clone the following Spack branch from the Git Repository and source the setup script.
git clone -b e4s-22.05 https://github.com/spack/spack.git
source spack/share/spack/setup-env.sh
Once you have acquired Spack and sourced the activation script, please run the following commands to ensure your setup is done correctly. We
have configured the environment, SPACK_PYTHON, to use a Python wrapper in the virtual environment.
(spack-pyenv) elvis@login34> spack --version
0.18.0.dev0 (6040c82740449632aa1d6faab08f93f5e4c54615)
(spack-pyenv) elvis@login34> echo $SPACK_PYTHON
/global/homes/e/elvis/spack-infrastructure/spack-pyenv/bin/python
(spack-pyenv) elvis@login34> which python
/global/homes/e/elvis/spack-infrastructure/spack-pyenv/bin/python
The command below will pass the full path to the Python interpreter used by Spack, which should be the path
set by environment SPACK_PYTHON.
(spack-pyenv) elvis@login34> spack-python --path
/global/homes/e/elvis/spack-infrastructure/spack-pyenv/bin/python
Creating a Spack Environment
When using Spack, you may be tempted to start installing packages via spack install in your Spack instance. Note
that it’s best you organize your Spack stacks in their own spack environment,
similar to how one would organize a Python or Conda environment.
Let’s start by creating a Spack environment named data_viz, and activating it.
spack env create data_viz
spack env activate data_viz
Upon completion you should confirm the output of spack env status matches the following:
(spack-pyenv) elvis@login34> spack env status
==> In environment data_viz
Let’s navigate to the directory for Spack environment data_viz. You will see a file spack.yaml that is used to specify your Spack configuration. This includes configuration options such as which compilers to use in your Spack builds.
(spack-pyenv) elvis@login34> spack cd -e data_viz
(spack-pyenv) elvis@login34> ls -l
total 1
-rw-rw-r-- 1 elvis elvis 199 Aug 3 19:09 spack.yaml
Defining Compilers
In order to use Spack, one must define a list of compilers in order to build packages. On Perlmutter, we have gcc/11.2.0
and cce/13.0.2 compilers available as modulefiles which correspond to the GCC and Cray compiler. In order to specify the
compiler definition we must use the corresponding PrgEnv-* module.
(spack-pyenv) elvis@login34> ml -t av gcc/11.2.0 cce/13.0.2
/opt/cray/pe/lmod/modulefiles/core:
cce/13.0.2
gcc/11.2.0
Let’s add the following content in spack.yaml. Please open the file in your preferred editor and paste the contents. Note that we specify the full path for cc, cxx, f77, and fc which should correspond to the Cray wrappers.
1# This is a Spack Environment file.
2#
3# It describes a set of packages to be installed, along with
4# configuration settings.
5spack:
6 config:
7 view: false
8 concretization: separately
9 build_stage: $spack/var/spack/stage
10 misc_cache: $spack/var/spack/misc_cache
11 concretizer: clingo
12
13 compilers:
14 - compiler:
15 spec: gcc@11.2.0
16 paths:
17 cc: cc
18 cxx: CC
19 f77: ftn
20 fc: ftn
21 flags: {}
22 operating_system: sles15
23 target: any
24 modules:
25 - PrgEnv-gnu
26 - gcc/11.2.0
27 - craype-x86-milan
28 - libfabric
29 extra_rpaths: []
30 - compiler:
31 spec: cce@13.0.2
32 paths:
33 cc: /opt/cray/pe/craype/default/bin/cc
34 cxx: /opt/cray/pe/craype/default/bin/CC
35 f77: /opt/cray/pe/craype/default/bin/ftn
36 fc: /opt/cray/pe/craype/default/bin/ftn
37 flags: {}
38 operating_system: sles15
39 target: any
40 modules:
41 - PrgEnv-cray
42 - cce/13.0.2
43 - craype-x86-milan
44 - libfabric
45 environment: {}
46 extra_rpaths: []
47
48 # add package specs to the `specs` list
49 specs: []
50 packages:
51 all:
52 compiler: [gcc@11.2.0, cce@13.0.2]
53
54 view: true
Note
The directory /opt/cray/pe/craype/default resorts to the default Cray programming environment, craype, in this case its 2.7.16 and
the cc wrapper should be from this corresponding directory.
(spack-pyenv) elvis@login34> ls -ld /opt/cray/pe/craype/default
lrwxrwxrwx 1 root root 6 Jun 1 14:56 /opt/cray/pe/craype/default -> 2.7.16
(spack-pyenv) elvis@login34> which cc
/opt/cray/pe/craype/2.7.16/bin/cc
On Perlmutter, the craype/2.7.16 modulefile is responsible for setting the Cray wrappers which is loaded by default as shown below:
(spack-pyenv) elvis@login34> ml -t list craype/2.7.16
craype/2.7.16
If this modulefile was removed, you will not have access to the Cray wrappers cc, CC or ftn which may result in several errors.
Now let’s check all available compilers by running spack compiler list
(spack-pyenv) elvis@login34> spack compiler list
==> Available compilers
-- cce sles15-any -----------------------------------------------
cce@13.0.2
-- gcc sles15-any -----------------------------------------------
gcc@11.2.0
Package Preference
Now let’s try to run spack spec -Il hdf5, you will notice Spack will try to install all the packages from source, some of which
are dependencies that should not be installed but rather set as external packages.
For instance, utilities like openssl, bzip2, diffutils, openmpi, openssh should not be installed from source. We have documented
Recommended External Packages for Spack that outlines a list
of packages where we recommend using the NERSC system installations.
1(spack-pyenv) elvis@login34> spack spec -Il hdf5
2Input spec
3--------------------------------
4 - hdf5
5
6Concretized
7--------------------------------
8 - z4dfikd hdf5@1.12.2%gcc@11.2.0~cxx~fortran~hl~ipo~java+mpi+shared~szip~threadsafe+tools api=default build_type=RelWithDebInfo arch=cray-sles15-zen3
9 - auepzq2 ^cmake@3.23.1%gcc@11.2.0~doc+ncurses+ownlibs~qt build_type=Release arch=cray-sles15-zen3
10 - 2t22mc5 ^ncurses@6.2%gcc@11.2.0~symlinks+termlib abi=none arch=cray-sles15-zen3
11 - nugfov2 ^pkgconf@1.8.0%gcc@11.2.0 arch=cray-sles15-zen3
12 - i2r3jpl ^openssl@1.1.1o%gcc@11.2.0~docs~shared certs=system arch=cray-sles15-zen3
13 - ekj3iat ^perl@5.34.1%gcc@11.2.0+cpanm+shared+threads arch=cray-sles15-zen3
14 - hafeanv ^berkeley-db@18.1.40%gcc@11.2.0+cxx~docs+stl patches=b231fcc arch=cray-sles15-zen3
15 - blbwwl4 ^bzip2@1.0.8%gcc@11.2.0~debug~pic+shared arch=cray-sles15-zen3
16 - gvbyw6w ^diffutils@3.8%gcc@11.2.0 arch=cray-sles15-zen3
17 - 3xwztgy ^libiconv@1.16%gcc@11.2.0 libs=shared,static arch=cray-sles15-zen3
18 - bxrz7zm ^gdbm@1.19%gcc@11.2.0 arch=cray-sles15-zen3
19 - avhrefq ^readline@8.1%gcc@11.2.0 arch=cray-sles15-zen3
20 - ozmcyfj ^zlib@1.2.12%gcc@11.2.0+optimize+pic+shared patches=0d38234 arch=cray-sles15-zen3
21 - gdm5qma ^openmpi@4.1.3%gcc@11.2.0~atomics~cuda~cxx~cxx_exceptions~gpfs~internal-hwloc~java~legacylaunchers~lustre~memchecker~pmi+pmix+romio+rsh~singularity+static+vt+wrapper-rpath fabrics=none schedulers=none arch=cray-sles15-zen3
22 - 6rkjosk ^hwloc@2.7.1%gcc@11.2.0~cairo~cuda~gl~libudev+libxml2~netloc~nvml~opencl+pci~rocm+shared arch=cray-sles15-zen3
23 - oyeiwvg ^libpciaccess@0.16%gcc@11.2.0 arch=cray-sles15-zen3
24 - 56oycjj ^libtool@2.4.7%gcc@11.2.0 arch=cray-sles15-zen3
25 - flsruli ^m4@1.4.19%gcc@11.2.0+sigsegv patches=9dc5fbd,bfdffa7 arch=cray-sles15-zen3
26 - wcuq435 ^libsigsegv@2.13%gcc@11.2.0 arch=cray-sles15-zen3
27 - koitq65 ^util-macros@1.19.3%gcc@11.2.0 arch=cray-sles15-zen3
28 - u2ai4xj ^libxml2@2.9.13%gcc@11.2.0~python arch=cray-sles15-zen3
29 - tyswlp4 ^xz@5.2.5%gcc@11.2.0~pic libs=shared,static arch=cray-sles15-zen3
30 - w2itznc ^libevent@2.1.12%gcc@11.2.0+openssl arch=cray-sles15-zen3
31 - t4jyphv ^numactl@2.0.14%gcc@11.2.0 patches=4e1d78c,62fc8a8,ff37630 arch=cray-sles15-zen3
32 - al4xc7v ^autoconf@2.69%gcc@11.2.0 patches=35c4492,7793209,a49dd5b arch=cray-sles15-zen3
33 - 2uxxcnx ^automake@1.16.5%gcc@11.2.0 arch=cray-sles15-zen3
34 - w5aq2sc ^openssh@9.0p1%gcc@11.2.0 arch=cray-sles15-zen3
35 - mkoju5b ^libedit@3.1-20210216%gcc@11.2.0 arch=cray-sles15-zen3
36 - t3wpbom ^pmix@4.1.2%gcc@11.2.0~docs+pmi_backwards_compatibility~restful arch=cray-sles15-zen3
Let’s try to update our Spack configuration with the external packages as follows:
1# This is a Spack Environment file.
2#
3# It describes a set of packages to be installed, along with
4# configuration settings.
5spack:
6 config:
7 view: false
8 concretization: separately
9 build_stage: $spack/var/spack/stage
10 misc_cache: $spack/var/spack/misc_cache
11 concretizer: clingo
12
13 compilers:
14 - compiler:
15 spec: gcc@11.2.0
16 paths:
17 cc: cc
18 cxx: CC
19 f77: ftn
20 fc: ftn
21 flags: {}
22 operating_system: sles15
23 target: any
24 modules:
25 - PrgEnv-gnu
26 - gcc/11.2.0
27 - craype-x86-milan
28 - libfabric
29 extra_rpaths: []
30 - compiler:
31 spec: cce@13.0.2
32 paths:
33 cc: /opt/cray/pe/craype/default/bin/cc
34 cxx: /opt/cray/pe/craype/default/bin/CC
35 f77: /opt/cray/pe/craype/default/bin/ftn
36 fc: /opt/cray/pe/craype/default/bin/ftn
37 flags: {}
38 operating_system: sles15
39 target: any
40 modules:
41 - PrgEnv-cray
42 - cce/13.0.2
43 - craype-x86-milan
44 - libfabric
45 environment: {}
46 extra_rpaths: []
47
48 # add package specs to the `specs` list
49 specs: []
50 packages:
51 all:
52 compiler: [gcc@11.2.0, cce@13.0.2]
53 bzip2:
54 version: [1.0.6]
55 externals:
56 - spec: bzip2@1.0.6
57 prefix: /usr
58 diffutils:
59 version: [3.6]
60 externals:
61 - spec: diffutils@3.6
62 prefix: /usr
63 findutils:
64 version: [4.6.0]
65 externals:
66 - spec: findutils@4.6.0
67 prefix: /usr
68 openssl:
69 version: [1.1.0i]
70 buildable: false
71 externals:
72 - spec: openssl@1.1.0i
73 prefix: /usr
74 openssh:
75 version: [7.9p1]
76 buildable: false
77 externals:
78 - spec: openssh@7.9p1
79 prefix: /usr
80 readline:
81 version: [7.0]
82 buildable: false
83 externals:
84 - spec: readline@7.0
85 prefix: /usr
86 tar:
87 version: [1.3]
88 buildable: false
89 externals:
90 - spec: tar@1.30
91 prefix: /usr
92 unzip:
93 version: [6.0]
94 buildable: false
95 externals:
96 - spec: unzip@6.0
97 prefix: /usr
98
99 view: true
Many software packages depend on MPI, BLAS, PMI, and libfabrics, and these packages are typically available on Perlmutter. Shown below is a breakdown of the provider and its corresponding modules typically available on Perlmutter
MPI: cray-mpich
BLAS: cray-libsci
PMI: cray-pmi
libfabrics: libfabrics
Shown below are the corresponding modules that you should consider when setting up external packages.
(spack-pyenv) elvis@login34> ml -d av cray-mpich cray-libsci cray-pmi libfabrics
--------------------------------------------------- Cray Compiler/Network Dependent Packages ----------------------------------------------------
cray-mpich-abi/8.1.17 cray-mpich/8.1.17 (L)
--------------------------------------------------------------- Cray Core Modules ---------------------------------------------------------------
cray-libsci/21.08.1.2 (L) cray-pmi-lib/6.0.17 cray-pmi/6.1.3
Where:
L: Module is loaded
Use "module spider" to find all possible modules and extensions.
Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys".
In Spack, you can use the spack providers command to find the corresponding Spack package that maps to the provider.
In Spack these are referred to as virtual packages which are a collection of Spack packages that provide the same functionality.
(spack-pyenv) elvis@login34> spack providers
Virtual packages:
D daal flame glu iconv jpeg lua-lang mkl mysql-client osmesa pkgconfig sycl unwind yacc
awk elf fuse glx ipp lapack luajit mpe onedal pbs rpc szip uuid ziglang
blas fftw-api gl golang java libllvm mariadb-client mpi opencl pil scalapack tbb xxd
For instance, if you want to see all the MPI providers you can run the following. Note that cray-mpich is in the list.
(spack-pyenv) elvis@login34> spack providers mpi
mpi:
cray-mpich intel-mpi mpich@:1.1 mpich mpt@1: mvapich2@2.3: openmpi spectrum-mpi
cray-mvapich2 intel-oneapi-mpi mpich@:1.2 mpilander mpt@3: mvapich2-gdr openmpi@1.6.5
fujitsu-mpi intel-parallel-studio mpich@:3.1 mpitrampoline mvapich2 mvapich2x openmpi@1.7.5:
hpcx-mpi mpich@:1.0 mpich@:3.2 mpt mvapich2@2.1: nvhpc openmpi@2.0.0:
Now let’s try to update our Spack configuration as follows:
1 # This is a Spack Environment file.
2 #
3 # It describes a set of packages to be installed, along with
4 # configuration settings.
5 spack:
6 config:
7 view: false
8 concretization: separately
9 build_stage: $spack/var/spack/stage
10 misc_cache: $spack/var/spack/misc_cache
11 concretizer: clingo
12
13 compilers:
14 - compiler:
15 spec: gcc@11.2.0
16 paths:
17 cc: cc
18 cxx: CC
19 f77: ftn
20 fc: ftn
21 flags: {}
22 operating_system: sles15
23 target: any
24 modules:
25 - PrgEnv-gnu
26 - gcc/11.2.0
27 - craype-x86-milan
28 - libfabric
29 extra_rpaths: []
30 - compiler:
31 spec: cce@13.0.2
32 paths:
33 cc: /opt/cray/pe/craype/default/bin/cc
34 cxx: /opt/cray/pe/craype/default/bin/CC
35 f77: /opt/cray/pe/craype/default/bin/ftn
36 fc: /opt/cray/pe/craype/default/bin/ftn
37 flags: {}
38 operating_system: sles15
39 target: any
40 modules:
41 - PrgEnv-cray
42 - cce/13.0.2
43 - craype-x86-milan
44 - libfabric
45 environment: {}
46 extra_rpaths: []
47
48 # add package specs to the `specs` list
49 specs: []
50 packages:
51 all:
52 compiler: [gcc@11.2.0, cce@13.0.2]
53 providers:
54 blas: [cray-libsci]
55 mpi: [cray-mpich]
56 bzip2:
57 version: [1.0.6]
58 externals:
59 - spec: bzip2@1.0.6
60 prefix: /usr
61 cray-libsci:
62 buildable: false
63 externals:
64 - spec: cray-libsci@21.08.1.2
65 modules:
66 - cray-libsci/21.08.1.2
67 cray-mpich:
68 buildable: false
69 externals:
70 - spec: cray-mpich@8.1.15 %gcc@11.2.0
71 prefix: /opt/cray/pe/mpich/8.1.15/ofi/gnu/9.1
72 modules:
73 - cray-mpich/8.1.15
74 - cudatoolkit/11.5
75 - spec: cray-mpich@8.1.15 %cce@13.0.2
76 prefix: /opt/cray/pe/mpich/8.1.15/ofi/cray/10.0/
77 modules:
78 - cray-mpich/8.1.15
79 - cudatoolkit/11.5
80 cray-pmi:
81 buildable: false
82 externals:
83 - spec: cray-pmi@6.1.1
84 modules:
85 - cray-pmi/6.1.1
86 diffutils:
87 version: [3.6]
88 externals:
89 - spec: diffutils@3.6
90 prefix: /usr
91 findutils:
92 version: [4.6.0]
93 externals:
94 - spec: findutils@4.6.0
95 prefix: /usr
96 libfabric:
97 buildable: false
98 variants: fabrics=sockets,tcp,udp,rxm
99 externals:
100 - spec: libfabric@1.11.0.4.114
101 prefix: /opt/cray/libfabric/1.11.0.4.114
102 modules:
103 - libfabric/1.11.0.4.114
104 openssl:
105 version: [1.1.0i]
106 buildable: false
107 externals:
108 - spec: openssl@1.1.0i
109 prefix: /usr
110 openssh:
111 version: [7.9p1]
112 buildable: false
113 externals:
114 - spec: openssh@7.9p1
115 prefix: /usr
116 readline:
117 version: [7.0]
118 buildable: false
119 externals:
120 - spec: readline@7.0
121 prefix: /usr
122 tar:
123 version: [1.3]
124 buildable: false
125 externals:
126 - spec: tar@1.30
127 prefix: /usr
128 unzip:
129 version: [6.0]
130 buildable: false
131 externals:
132 - spec: unzip@6.0
133 prefix: /usr
134
135 view: true
Let’s try to run spack spec hypre and notice that Spack will now use cray-libsci and cray-mpich as the dependencies,
because we have set these packages as externals.
(spack-pyenv) elvis@login34> spack spec hypre
Input spec
--------------------------------
hypre@2.24.0
Concretized
--------------------------------
hypre@2.24.0%gcc@11.2.0~complex~cuda~debug+fortran~gptune~int64~internal-superlu~mixedint+mpi~openmp~rocm+shared~superlu-dist~unified-memory arch=cray-sles15-zen3
^cray-libsci@21.08.1.2%gcc@11.2.0~mpi~openmp+shared arch=cray-sles15-zen3
^cray-mpich@8.1.15%gcc@11.2.0+wrappers arch=cray-sles15-zen3
Now let’s try to add some packages to our Spack configuration by adding the following lines:
1# This is a Spack Environment file.
2#
3# It describes a set of packages to be installed, along with
4# configuration settings.
5spack:
6 config:
7 view: false
8 concretization: separately
9 build_stage: $spack/var/spack/stage
10 misc_cache: $spack/var/spack/misc_cache
11 concretizer: clingo
12 compilers:
13 - compiler:
14 spec: gcc@11.2.0
15 paths:
16 cc: cc
17 cxx: CC
18 f77: ftn
19 fc: ftn
20 flags: {}
21 operating_system: sles15
22 target: any
23 modules:
24 - PrgEnv-gnu
25 - gcc/11.2.0
26 - craype-x86-milan
27 - libfabric
28 extra_rpaths: []
29 - compiler:
30 spec: cce@13.0.2
31 paths:
32 cc: /opt/cray/pe/craype/default/bin/cc
33 cxx: /opt/cray/pe/craype/default/bin/CC
34 f77: /opt/cray/pe/craype/default/bin/ftn
35 fc: /opt/cray/pe/craype/default/bin/ftn
36 flags: {}
37 operating_system: sles15
38 target: any
39 modules:
40 - PrgEnv-cray
41 - cce/13.0.2
42 - craype-x86-milan
43 - libfabric
44 environment: {}
45 extra_rpaths: []
46 # add package specs to the `specs` list
47 specs:
48 - papi %gcc
49 - papi %cce
50 - hypre %gcc
51 - hypre %cce
52 - darshan-runtime %gcc
53 - darshan-runtime %cce
54 packages:
55 all:
56 compiler: [gcc@11.2.0, cce@13.0.2]
57 providers:
58 blas: [cray-libsci]
59 mpi: [cray-mpich]
60 bzip2:
61 version: [1.0.6]
62 externals:
63 - spec: bzip2@1.0.6
64 prefix: /usr
65 cray-libsci:
66 buildable: false
67 externals:
68 - spec: cray-libsci@21.08.1.2
69 modules:
70 - cray-libsci/21.08.1.2
71 cray-mpich:
72 buildable: false
73 externals:
74 - spec: cray-mpich@8.1.15 %gcc@11.2.0
75 prefix: /opt/cray/pe/mpich/8.1.15/ofi/gnu/9.1
76 modules:
77 - cray-mpich/8.1.15
78 - cudatoolkit/11.5
79 - spec: cray-mpich@8.1.15 %cce@13.0.2
80 prefix: /opt/cray/pe/mpich/8.1.15/ofi/cray/10.0/
81 modules:
82 - cray-mpich/8.1.15
83 - cudatoolkit/11.5
84 cray-pmi:
85 buildable: false
86 externals:
87 - spec: cray-pmi@6.1.1
88 modules:
89 - cray-pmi/6.1.1
90 diffutils:
91 version: [3.6]
92 externals:
93 - spec: diffutils@3.6
94 prefix: /usr
95 findutils:
96 version: [4.6.0]
97 externals:
98 - spec: findutils@4.6.0
99 prefix: /usr
100 libfabric:
101 buildable: false
102 variants: fabrics=sockets,tcp,udp,rxm
103 externals:
104 - spec: libfabric@1.11.0.4.114
105 prefix: /opt/cray/libfabric/1.11.0.4.114
106 modules:
107 - libfabric/1.11.0.4.114
108 openssl:
109 version: [1.1.0i]
110 buildable: false
111 externals:
112 - spec: openssl@1.1.0i
113 prefix: /usr
114 openssh:
115 version: [7.9p1]
116 buildable: false
117 externals:
118 - spec: openssh@7.9p1
119 prefix: /usr
120 readline:
121 version: [7.0]
122 buildable: false
123 externals:
124 - spec: readline@7.0
125 prefix: /usr
126 tar:
127 version: [1.3]
128 buildable: false
129 externals:
130 - spec: tar@1.30
131 prefix: /usr
132 unzip:
133 version: [6.0]
134 buildable: false
135 externals:
136 - spec: unzip@6.0
137 prefix: /usr
138 view: true
Next, we will concretize the environment, you should see papi, hypre and darshan-runtime built with each compiler.
(spack-pyenv) elvis@login34> spack concretize
==> Starting concretization pool with 6 processes
==> Environment concretized in 18.58 seconds.
==> Concretized papi%gcc
- s2y4nrv papi@6.0.0.1%gcc@11.2.0~cuda+example~infiniband~lmsensors~nvml~powercap~rapl~rocm~rocm_smi~sde+shared~static_tools arch=cray-sles15-zen3
==> Concretized papi%cce
- 3aprcx5 papi@6.0.0.1%cce@13.0.2~cuda+example~infiniband~lmsensors~nvml~powercap~rapl~rocm~rocm_smi~sde+shared~static_tools patches=b6d6caa arch=cray-sles15-zen3
==> Concretized hypre%gcc
- mbn7bum hypre@2.24.0%gcc@11.2.0~complex~cuda~debug+fortran~gptune~int64~internal-superlu~mixedint+mpi~openmp~rocm+shared~superlu-dist~unified-memory arch=cray-sles15-zen3
- jzbnd6y ^cray-libsci@21.08.1.2%gcc@11.2.0~mpi~openmp+shared arch=cray-sles15-zen3
- 3zy6uvs ^cray-mpich@8.1.15%gcc@11.2.0+wrappers arch=cray-sles15-zen3
==> Concretized hypre%cce
- 62ofdsf hypre@2.24.0%cce@13.0.2~complex~cuda~debug+fortran~gptune~int64~internal-superlu~mixedint+mpi~openmp~rocm+shared~superlu-dist~unified-memory arch=cray-sles15-zen3
- 7uzhxpv ^cray-libsci@21.08.1.2%cce@13.0.2~mpi~openmp+shared arch=cray-sles15-zen3
- tb5uxwe ^cray-mpich@8.1.15%cce@13.0.2+wrappers arch=cray-sles15-zen3
==> Concretized darshan-runtime%gcc
- hkxzwvt darshan-runtime@3.3.1%gcc@11.2.0~apmpi~apmpi_sync~apxc~hdf5+mpi scheduler=NONE arch=cray-sles15-zen3
- 3zy6uvs ^cray-mpich@8.1.15%gcc@11.2.0+wrappers arch=cray-sles15-zen3
- ozmcyfj ^zlib@1.2.12%gcc@11.2.0+optimize+pic+shared patches=0d38234 arch=cray-sles15-zen3
==> Concretized darshan-runtime%cce
- uj3wa4a darshan-runtime@3.3.1%cce@13.0.2~apmpi~apmpi_sync~apxc~hdf5+mpi scheduler=NONE arch=cray-sles15-zen3
- tb5uxwe ^cray-mpich@8.1.15%cce@13.0.2+wrappers arch=cray-sles15-zen3
- e2hl6cx ^zlib@1.2.12%cce@13.0.2+optimize+pic+shared patches=0d38234 arch=cray-sles15-zen3
Let’s install all the packages via spack install. This would be a good time to get a cup of coffee since it will likely
take a few minutes.
(spack-pyenv) elvis@login34> spack install
==> Installing environment data_viz
==> Installing papi-6.0.0.1-s2y4nrvu6whr6hhgi63aa3nqwz2d35af
==> No binary for papi-6.0.0.1-s2y4nrvu6whr6hhgi63aa3nqwz2d35af found: installing from source
==> Fetching https://mirror.spack.io/_source-cache/archive/3c/3cd7ed50c65b0d21d66e46d0ba34cd171178af4bbf9d94e693915c1aca1e287f.tar.gz
==> No patches needed for papi
==> papi: Executing phase: 'autoreconf'
==> papi: Executing phase: 'configure'
==> papi: Executing phase: 'build'
==> papi: Executing phase: 'install'
==> papi: Successfully installed papi-6.0.0.1-s2y4nrvu6whr6hhgi63aa3nqwz2d35af
Fetch: 1.49s. Build: 28.94s. Total: 30.43s.
[+] /global/u1/e/elvis/spack-infrastructure/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/papi-6.0.0.1-s2y4nrvu6whr6hhgi63aa3nqwz2d35af
==> Installing papi-6.0.0.1-3aprcx5klzafe7xt6aq57jx5sequpue2
==> No binary for papi-6.0.0.1-3aprcx5klzafe7xt6aq57jx5sequpue2 found: installing from source
==> Using cached archive: /global/u1/e/elvis/spack-infrastructure/spack/var/spack/cache/_source-cache/archive/3c/3cd7ed50c65b0d21d66e46d0ba34cd171178af4bbf9d94e693915c1aca1e287f.tar.gz
==> Applied patch /global/u1/e/elvis/spack-infrastructure/spack/var/spack/repos/builtin/packages/papi/crayftn-fixes.patch
==> papi: Executing phase: 'autoreconf'
==> papi: Executing phase: 'configure'
==> papi: Executing phase: 'build'
==> papi: Executing phase: 'install'
==> papi: Successfully installed papi-6.0.0.1-3aprcx5klzafe7xt6aq57jx5sequpue2
Fetch: 0.01s. Build: 28.94s. Total: 28.95s.
[+] /global/u1/e/elvis/spack-infrastructure/spack/opt/spack/cray-sles15-zen3/cce-13.0.2/papi-6.0.0.1-3aprcx5klzafe7xt6aq57jx5sequpue2
==> cray-libsci@21.08.1.2 : has external module in ['cray-libsci/21.08.1.2']
[+] /opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64 (external cray-libsci-21.08.1.2-jzbnd6ycupy2ycs5jiavwyvkxv3rpuru)
==> cray-mpich@8.1.15 : has external module in ['cray-mpich/8.1.15', 'cudatoolkit/11.5']
[+] /opt/cray/pe/mpich/8.1.15/ofi/gnu/9.1 (external cray-mpich-8.1.15-3zy6uvszbd5a3rniq2xd2v5a3d27qstw)
==> cray-libsci@21.08.1.2 : has external module in ['cray-libsci/21.08.1.2']
[+] /opt/cray/pe/libsci/21.08.1.2/CRAY/9.0/x86_64 (external cray-libsci-21.08.1.2-7uzhxpvoka7ixfxs44354dkishquwyhq)
==> cray-mpich@8.1.15 : has external module in ['cray-mpich/8.1.15', 'cudatoolkit/11.5']
[+] /opt/cray/pe/mpich/8.1.15/ofi/cray/10.0/ (external cray-mpich-8.1.15-tb5uxwezfzx4xth7azefyrhzlvf7koqb)
==> Installing zlib-1.2.12-ozmcyfjfv7i5gjjgklfsh43h67vzsuc5
==> No binary for zlib-1.2.12-ozmcyfjfv7i5gjjgklfsh43h67vzsuc5 found: installing from source
==> Fetching https://mirror.spack.io/_source-cache/archive/91/91844808532e5ce316b3c010929493c0244f3d37593afd6de04f71821d5136d9.tar.gz
==> Applied patch /global/u1/e/elvis/spack-infrastructure/spack/var/spack/repos/builtin/packages/zlib/configure-cc.patch
==> zlib: Executing phase: 'install'
==> zlib: Successfully installed zlib-1.2.12-ozmcyfjfv7i5gjjgklfsh43h67vzsuc5
Fetch: 0.62s. Build: 2.10s. Total: 2.72s.
[+] /global/u1/e/elvis/spack-infrastructure/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.12-ozmcyfjfv7i5gjjgklfsh43h67vzsuc5
==> Installing zlib-1.2.12-e2hl6cxmzbg5psoh5upqmqqltjftc3pb
==> No binary for zlib-1.2.12-e2hl6cxmzbg5psoh5upqmqqltjftc3pb found: installing from source
==> Using cached archive: /global/u1/e/elvis/spack-infrastructure/spack/var/spack/cache/_source-cache/archive/91/91844808532e5ce316b3c010929493c0244f3d37593afd6de04f71821d5136d9.tar.gz
==> Applied patch /global/u1/e/elvis/spack-infrastructure/spack/var/spack/repos/builtin/packages/zlib/configure-cc.patch
==> zlib: Executing phase: 'install'
==> zlib: Successfully installed zlib-1.2.12-e2hl6cxmzbg5psoh5upqmqqltjftc3pb
Fetch: 0.00s. Build: 2.45s. Total: 2.45s.
[+] /global/u1/e/elvis/spack-infrastructure/spack/opt/spack/cray-sles15-zen3/cce-13.0.2/zlib-1.2.12-e2hl6cxmzbg5psoh5upqmqqltjftc3pb
==> Installing hypre-2.24.0-mbn7bumcoqmjhf5y2sm3hnr64vml4dvf
==> No binary for hypre-2.24.0-mbn7bumcoqmjhf5y2sm3hnr64vml4dvf found: installing from source
==> Fetching https://mirror.spack.io/_source-cache/archive/f4/f480e61fc25bf533fc201fdf79ec440be79bb8117650627d1f25151e8be2fdb5.tar.gz
==> No patches needed for hypre
==> hypre: Executing phase: 'autoreconf'
==> hypre: Executing phase: 'configure'
==> hypre: Executing phase: 'build'
==> hypre: Executing phase: 'install'
==> hypre: Successfully installed hypre-2.24.0-mbn7bumcoqmjhf5y2sm3hnr64vml4dvf
Fetch: 0.77s. Build: 37.43s. Total: 38.20s.
[+] /global/u1/e/elvis/spack-infrastructure/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-2.24.0-mbn7bumcoqmjhf5y2sm3hnr64vml4dvf
==> Installing hypre-2.24.0-62ofdsfxckay53ewpiidg4nlamhnzq3b
==> No binary for hypre-2.24.0-62ofdsfxckay53ewpiidg4nlamhnzq3b found: installing from source
==> Using cached archive: /global/u1/e/elvis/spack-infrastructure/spack/var/spack/cache/_source-cache/archive/f4/f480e61fc25bf533fc201fdf79ec440be79bb8117650627d1f25151e8be2fdb5.tar.gz
==> No patches needed for hypre
==> hypre: Executing phase: 'autoreconf'
==> hypre: Executing phase: 'configure'
==> hypre: Executing phase: 'build'
==> hypre: Executing phase: 'install'
==> hypre: Successfully installed hypre-2.24.0-62ofdsfxckay53ewpiidg4nlamhnzq3b
Fetch: 0.01s. Build: 1m 5.86s. Total: 1m 5.87s.
[+] /global/u1/e/elvis/spack-infrastructure/spack/opt/spack/cray-sles15-zen3/cce-13.0.2/hypre-2.24.0-62ofdsfxckay53ewpiidg4nlamhnzq3b
==> Installing darshan-runtime-3.3.1-hkxzwvtw5rlmsvwt4irwnxxuwzwbuzoj
==> No binary for darshan-runtime-3.3.1-hkxzwvtw5rlmsvwt4irwnxxuwzwbuzoj found: installing from source
==> Fetching https://mirror.spack.io/_source-cache/archive/28/281d871335977d0592a49d053df93d68ce1840f6fdec27fea7a59586a84395f7.tar.gz
==> No patches needed for darshan-runtime
==> darshan-runtime: Executing phase: 'autoreconf'
==> darshan-runtime: Executing phase: 'configure'
==> darshan-runtime: Executing phase: 'build'
==> darshan-runtime: Executing phase: 'install'
==> darshan-runtime: Successfully installed darshan-runtime-3.3.1-hkxzwvtw5rlmsvwt4irwnxxuwzwbuzoj
Fetch: 1.07s. Build: 9.24s. Total: 10.31s.
[+] /global/u1/e/elvis/spack-infrastructure/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/darshan-runtime-3.3.1-hkxzwvtw5rlmsvwt4irwnxxuwzwbuzoj
==> Installing darshan-runtime-3.3.1-uj3wa4au7kphj52syka4w3dxiadosagh
==> No binary for darshan-runtime-3.3.1-uj3wa4au7kphj52syka4w3dxiadosagh found: installing from source
==> Using cached archive: /global/u1/e/elvis/spack-infrastructure/spack/var/spack/cache/_source-cache/archive/28/281d871335977d0592a49d053df93d68ce1840f6fdec27fea7a59586a84395f7.tar.gz
==> No patches needed for darshan-runtime
==> darshan-runtime: Executing phase: 'autoreconf'
==> darshan-runtime: Executing phase: 'configure'
==> darshan-runtime: Executing phase: 'build'
==> darshan-runtime: Executing phase: 'install'
==> darshan-runtime: Successfully installed darshan-runtime-3.3.1-uj3wa4au7kphj52syka4w3dxiadosagh
Fetch: 0.01s. Build: 9.58s. Total: 9.58s.
[+] /global/u1/e/elvis/spack-infrastructure/spack/opt/spack/cray-sles15-zen3/cce-13.0.2/darshan-runtime-3.3.1-uj3wa4au7kphj52syka4w3dxiadosagh
==> Updating view at /global/u1/e/elvis/spack-infrastructure/spack/var/spack/environments/data_viz/.spack-env/view
==> Warning: Skipping external package: cray-libsci@21.08.1.2%gcc@11.2.0~mpi~openmp+shared arch=cray-sles15-zen3/jzbnd6y
==> Warning: Skipping external package: cray-mpich@8.1.15%gcc@11.2.0+wrappers arch=cray-sles15-zen3/3zy6uvs
==> Warning: Skipping external package: cray-libsci@21.08.1.2%cce@13.0.2~mpi~openmp+shared arch=cray-sles15-zen3/7uzhxpv
==> Warning: Skipping external package: cray-mpich@8.1.15%cce@13.0.2+wrappers arch=cray-sles15-zen3/tb5uxwe
==> Error: 178 fatal error(s) when merging prefixes:
`/global/u1/e/elvis/spack-infrastructure/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/papi-6.0.0.1-s2y4nrvu6whr6hhgi63aa3nqwz2d35af/.spack/archived-files/src/removed_la_files.txt` and `/global/u1/e/elvis/spack-infrastructure/spack/opt/spack/cray-sles15-zen3/cce-13.0.2/papi-6.0.0.1-3aprcx5klzafe7xt6aq57jx5sequpue2/.spack/archived-files/src/removed_la_files.txt` both project to `.spack/papi/archived-files/src/removed_la_files.txt`
`/global/u1/e/elvis/spack-infrastructure/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/papi-6.0.0.1-s2y4nrvu6whr6hhgi63aa3nqwz2d35af/.spack/install_environment.json` and `/global/u1/e/elvis/spack-infrastructure/spack/opt/spack/cray-sles15-zen3/cce-13.0.2/papi-6.0.0.1-3aprcx5klzafe7xt6aq57jx5sequpue2/.spack/install_environment.json` both project to `.spack/papi/install_environment.json`
`/global/u1/e/elvis/spack-infrastructure/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/papi-6.0.0.1-s2y4nrvu6whr6hhgi63aa3nqwz2d35af/.spack/install_manifest.json` and `/global/u1/e/elvis/spack-infrastructure/spack/opt/spack/cray-sles15-zen3/cce-13.0.2/papi-6.0.0.1-3aprcx5klzafe7xt6aq57jx5sequpue2/.spack/install_manifest.json` both project to `.spack/papi/install_manifest.json`
Upon completion you can run spack find to see all installed packages.
(spack-pyenv) elvis@login34> spack find
==> In environment data_viz
==> Root specs
-- no arch / cce ------------------------------------------------
darshan-runtime%cce hypre%cce papi%cce
-- no arch / gcc ------------------------------------------------
darshan-runtime%gcc hypre%gcc papi%gcc
==> 12 installed packages
-- cray-sles15-zen3 / cce@13.0.2 --------------------------------
cray-libsci@21.08.1.2 cray-mpich@8.1.15 darshan-runtime@3.3.1 hypre@2.24.0 papi@6.0.0.1 zlib@1.2.12
-- cray-sles15-zen3 / gcc@11.2.0 --------------------------------
cray-libsci@21.08.1.2 cray-mpich@8.1.15 darshan-runtime@3.3.1 hypre@2.24.0 papi@6.0.0.1 zlib@1.2.12
Defining a Source Mirror
You may have noticed Spack will fetch tarballs from the web when installing packages and this can be time-consuming when downloading large tarballs. It is a good idea to store tarballs on the filesystem once and then let Spack use them for any Spack builds. You should have one location where tarballs. Let’s run the following command:
(spack-pyenv) elvis@login34> spack mirror create -d $CI_PROJECT_DIR/spack_mirror -a
==> Adding package cray-libsci@21.08.1.2 to mirror
==> Adding package cray-libsci@21.08.1.2 to mirror
==> Adding package cray-mpich@8.1.15 to mirror
==> Adding package cray-mpich@8.1.15 to mirror
==> Adding package darshan-runtime@3.3.1 to mirror
==> Using cached archive: /global/u1/e/elvis/spack-infrastructure/spack/var/spack/cache/_source-cache/archive/28/281d871335977d0592a49d053df93d68ce1840f6fdec27fea7a59586a84395f7.tar.gz
==> Adding package darshan-runtime@3.3.1 to mirror
==> Adding package hypre@2.24.0 to mirror
==> Using cached archive: /global/u1/e/elvis/spack-infrastructure/spack/var/spack/cache/_source-cache/archive/f4/f480e61fc25bf533fc201fdf79ec440be79bb8117650627d1f25151e8be2fdb5.tar.gz
==> Adding package hypre@2.24.0 to mirror
==> Adding package papi@6.0.0.1 to mirror
==> Using cached archive: /global/u1/e/elvis/spack-infrastructure/spack/var/spack/cache/_source-cache/archive/3c/3cd7ed50c65b0d21d66e46d0ba34cd171178af4bbf9d94e693915c1aca1e287f.tar.gz
==> Fetching https://mirror.spack.io/_source-cache/archive/64/64c57b3ad4026255238cc495df6abfacc41de391a0af497c27d0ac819444a1f8
==> Adding package papi@6.0.0.1 to mirror
==> Adding package zlib@1.2.12 to mirror
==> Using cached archive: /global/u1/e/elvis/spack-infrastructure/spack/var/spack/cache/_source-cache/archive/91/91844808532e5ce316b3c010929493c0244f3d37593afd6de04f71821d5136d9.tar.gz
==> Adding package zlib@1.2.12 to mirror
==> Successfully created mirror in file:///global/homes/e/elvis/spack-infrastructure/spack_mirror
Archive stats:
4 already present
4 added
0 failed to fetch.
If you inspect the directory you will notice the tarballs are present in this directory.
(spack-pyenv) elvis@login34> ls -l $CI_PROJECT_DIR/spack_mirror/*
/global/homes/e/elvis/spack-infrastructure/spack_mirror/darshan-runtime:
total 1
lrwxrwxrwx 1 elvis elvis 99 Aug 4 08:28 darshan-runtime-3.3.1.tar.gz -> ../_source-cache/archive/28/281d871335977d0592a49d053df93d68ce1840f6fdec27fea7a59586a84395f7.tar.gz
/global/homes/e/elvis/spack-infrastructure/spack_mirror/hypre:
total 1
lrwxrwxrwx 1 elvis elvis 99 Aug 4 08:28 hypre-2.24.0.tar.gz -> ../_source-cache/archive/f4/f480e61fc25bf533fc201fdf79ec440be79bb8117650627d1f25151e8be2fdb5.tar.gz
/global/homes/e/elvis/spack-infrastructure/spack_mirror/papi:
total 2
lrwxrwxrwx 1 elvis elvis 99 Aug 4 08:28 papi-6.0.0.1.tar.gz -> ../_source-cache/archive/3c/3cd7ed50c65b0d21d66e46d0ba34cd171178af4bbf9d94e693915c1aca1e287f.tar.gz
lrwxrwxrwx 1 elvis elvis 92 Aug 4 08:28 raw-64c57b3 -> ../_source-cache/archive/64/64c57b3ad4026255238cc495df6abfacc41de391a0af497c27d0ac819444a1f8
/global/homes/e/elvis/spack-infrastructure/spack_mirror/_source-cache:
total 1
drwxrwxr-x 7 elvis elvis 512 Aug 4 08:28 archive
/global/homes/e/elvis/spack-infrastructure/spack_mirror/zlib:
total 1
lrwxrwxrwx 1 elvis elvis 99 Aug 4 08:28 zlib-1.2.12.tar.gz -> ../_source-cache/archive/91/91844808532e5ce316b3c010929493c0244f3d37593afd6de04f71821d5136d9.tar.gz
Building CUDA Packages
On Perlmutter, the standalone CUDA package is available by loading the following modulefile:
(spack-pyenv) elvis@login34> ml -t av cudatoolkit
/opt/cray/pe/lmod/modulefiles/core:
cudatoolkit/11.5
cudatoolkit/11.7
NVIDIA provides CUDA as part of the NVHPC compiler which is installed on Perlmutter and accessible via the nvhpc modulefile.
(spack-pyenv) elvis@login34> ml -t av nvhpc
/opt/cray/pe/lmod/modulefiles/mix_compilers:
nvhpc-mixed/21.11
nvhpc-mixed/22.5
/opt/cray/pe/lmod/modulefiles/core:
nvhpc/21.11
nvhpc/22.5
The root of nvhpc/21.11 is available at /opt/nvidia/hpc_sdk/Linux_x86_64/21.11. You can see content of this modulefile by running
module show nvhpc/21.11 and inspecting the modulefile. Shown below is the directory structure for root of NVHPC stack.
(spack-pyenv) elvis@login34> ls -l /opt/nvidia/hpc_sdk/Linux_x86_64/21.11
total 0
drwxr-xr-x 2 root root 72 Aug 1 07:03 cmake
drwxrwxr-x 6 root root 144 Aug 1 07:07 comm_libs
drwxrwxr-x 14 root root 235 Aug 1 07:07 compilers
drwxrwxr-x 3 root root 78 Aug 1 07:07 cuda
drwxrwxr-x 11 root root 205 Aug 1 07:05 examples
drwxrwxr-x 3 root root 55 Aug 1 07:07 math_libs
drwxrwxr-x 4 root root 71 Aug 1 07:07 profilers
drwxrwxr-x 6 root root 90 Aug 1 07:03 REDIST
cuda/11.5 is installed in following directory, which can be activated by loading the cudatoolkit/11.5 modulefile.
(spack-pyenv) elvis@login34> ls -l /opt/nvidia/hpc_sdk/Linux_x86_64/21.11/cuda/11.5
total 65
drwxrwxr-x 3 root root 335 Aug 1 07:04 bin
drwxrwxr-x 4 root root 385 Aug 1 07:04 compute-sanitizer
-rw-r--r-- 1 root root 160 Dec 8 2021 DOCS
-rw-r--r-- 1 root root 61727 Dec 8 2021 EULA.txt
drwxrwxr-x 4 root root 44 Aug 1 07:04 extras
lrwxrwxrwx 1 root root 28 Dec 8 2021 include -> targets/x86_64-linux/include
lrwxrwxrwx 1 root root 24 Dec 8 2021 lib64 -> targets/x86_64-linux/lib
drwxrwxr-x 7 root root 242 Aug 1 07:04 libnvvp
drwxrwxr-x 3 root root 30 Aug 1 07:04 nvml
drwxrwxr-x 7 root root 106 Aug 1 07:04 nvvm
drwxrwxr-x 7 root root 94 Aug 1 07:04 nvvm-prev
-rw-r--r-- 1 root root 524 Dec 8 2021 README
drwxrwxr-x 3 root root 26 Aug 1 07:04 share
drwxrwxr-x 3 root root 35 Aug 1 07:04 targets
drwxrwxr-x 2 root root 52 Aug 1 07:05 tools
-rw-r--r-- 1 root root 2669 Dec 8 2021 version.json
We can confirm the nvcc compiler provided by CUDA is available in this directory along with the libcudart.so (CUDA Runtime) library
(spack-pyenv) elvis@login34> /opt/nvidia/hpc_sdk/Linux_x86_64/21.11/cuda/11.5/bin/nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2021 NVIDIA Corporation
Built on Thu_Nov_18_09:45:30_PST_2021
Cuda compilation tools, release 11.5, V11.5.119
Build cuda_11.5.r11.5/compiler.30672275_0
(spack-pyenv) elvis@login34> ls /opt/nvidia/hpc_sdk/Linux_x86_64/21.11/cuda/11.5/lib64/libcudart.so
/opt/nvidia/hpc_sdk/Linux_x86_64/21.11/cuda/11.5/lib64/libcudart.so
Let’s define our CUDA package preference in our Spack configuration. To
illustrate, we will install papi with the spec papi +cuda %gcc.
This indicates that we want PAPI installed with CUDA support using the GCC compiler.
Please copy the following content in your spack.yaml.
1 # This is a Spack Environment file.
2 #
3 # It describes a set of packages to be installed, along with
4 # configuration settings.
5 spack:
6 config:
7 view: false
8 concretization: separately
9 build_stage: $spack/var/spack/stage
10 misc_cache: $spack/var/spack/misc_cache
11 concretizer: clingo
12 compilers:
13 - compiler:
14 spec: gcc@11.2.0
15 paths:
16 cc: cc
17 cxx: CC
18 f77: ftn
19 fc: ftn
20 flags: {}
21 operating_system: sles15
22 target: any
23 modules:
24 - PrgEnv-gnu
25 - gcc/11.2.0
26 - craype-x86-milan
27 - libfabric
28 extra_rpaths: []
29 - compiler:
30 spec: cce@13.0.2
31 paths:
32 cc: /opt/cray/pe/craype/default/bin/cc
33 cxx: /opt/cray/pe/craype/default/bin/CC
34 f77: /opt/cray/pe/craype/default/bin/ftn
35 fc: /opt/cray/pe/craype/default/bin/ftn
36 flags: {}
37 operating_system: sles15
38 target: any
39 modules:
40 - PrgEnv-cray
41 - cce/13.0.2
42 - craype-x86-milan
43 - libfabric
44 environment: {}
45 extra_rpaths: []
46
47 # add package specs to the `specs` list
48 specs:
49 - papi %gcc
50 - papi %cce
51 - hypre %gcc
52 - hypre %cce
53 - darshan-runtime %gcc
54 - darshan-runtime %cce
55 - papi +cuda %gcc
56 packages:
57 all:
58 compiler: [gcc@11.2.0, cce@13.0.2]
59 providers:
60 blas: [cray-libsci]
61 mpi: [cray-mpich]
62 bzip2:
63 version: [1.0.6]
64 externals:
65 - spec: bzip2@1.0.6
66 prefix: /usr
67 cray-libsci:
68 buildable: false
69 externals:
70 - spec: cray-libsci@21.08.1.2
71 modules:
72 - cray-libsci/21.08.1.2
73 cray-mpich:
74 buildable: false
75 externals:
76 - spec: cray-mpich@8.1.15 %gcc@11.2.0
77 prefix: /opt/cray/pe/mpich/8.1.15/ofi/gnu/9.1
78 modules:
79 - cray-mpich/8.1.15
80 - cudatoolkit/11.5
81 - spec: cray-mpich@8.1.15 %cce@13.0.2
82 prefix: /opt/cray/pe/mpich/8.1.15/ofi/cray/10.0/
83 modules:
84 - cray-mpich/8.1.15
85 - cudatoolkit/11.5
86 cray-pmi:
87 buildable: false
88 externals:
89 - spec: cray-pmi@6.1.1
90 modules:
91 - cray-pmi/6.1.1
92 cuda:
93 buildable: false
94 version: [11.5.0]
95 externals:
96 - spec: cuda@11.5.0
97 prefix: /opt/nvidia/hpc_sdk/Linux_x86_64/21.11/cuda/11.5
98 modules:
99 - cudatoolkit/11.5
100 diffutils:
101 version: [3.6]
102 externals:
103 - spec: diffutils@3.6
104 prefix: /usr
105 findutils:
106 version: [4.6.0]
107 externals:
108 - spec: findutils@4.6.0
109 prefix: /usr
110 libfabric:
111 buildable: false
112 variants: fabrics=sockets,tcp,udp,rxm
113 externals:
114 - spec: libfabric@1.11.0.4.114
115 prefix: /opt/cray/libfabric/1.11.0.4.114
116 modules:
117 - libfabric/1.11.0.4.114
118 openssl:
119 version: [1.1.0i]
120 buildable: false
121 externals:
122 - spec: openssl@1.1.0i
123 prefix: /usr
124 openssh:
125 version: [7.9p1]
126 buildable: false
127 externals:
128 - spec: openssh@7.9p1
129 prefix: /usr
130 readline:
131 version: [7.0]
132 buildable: false
133 externals:
134 - spec: readline@7.0
135 prefix: /usr
136 tar:
137 version: [1.3]
138 buildable: false
139 externals:
140 - spec: tar@1.30
141 prefix: /usr
142 unzip:
143 version: [6.0]
144 buildable: false
145 externals:
146 - spec: unzip@6.0
147 prefix: /usr
148 view: true
Now let’s try to install.
(spack-pyenv) elvis@login34> spack install
==> Installing environment data_viz
==> cuda@11.5.0 : has external module in ['cudatoolkit/11.5']
[+] /opt/nvidia/hpc_sdk/Linux_x86_64/21.11/cuda/11.5 (external cuda-11.5.0-puekfe32hbj72iftffa3etecesmlqwqg)
==> Installing papi-6.0.0.1-x43djbqgyb64susljh3vu4czlqapbyie
==> No binary for papi-6.0.0.1-x43djbqgyb64susljh3vu4czlqapbyie found: installing from source
==> Using cached archive: /global/u1/e/elvis/spack-infrastructure/spack/var/spack/cache/_source-cache/archive/3c/3cd7ed50c65b0d21d66e46d0ba34cd171178af4bbf9d94e693915c1aca1e287f.tar.gz
==> No patches needed for papi
==> papi: Executing phase: 'autoreconf'
==> papi: Executing phase: 'configure'
==> papi: Executing phase: 'build'
==> papi: Executing phase: 'install'
==> papi: Successfully installed papi-6.0.0.1-x43djbqgyb64susljh3vu4czlqapbyie
Fetch: 0.01s. Build: 4m 46.76s. Total: 4m 46.76s.
[+] /global/u1/e/elvis/spack-infrastructure/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/papi-6.0.0.1-x43djbqgyb64susljh3vu4czlqapbyie
==> Updating view at /global/u1/e/elvis/spack-infrastructure/spack/var/spack/environments/data_viz/.spack-env/view
==> Warning: Skipping external package: cray-libsci@21.08.1.2%gcc@11.2.0~mpi~openmp+shared arch=cray-sles15-zen3/jzbnd6y
==> Warning: Skipping external package: cray-mpich@8.1.15%gcc@11.2.0+wrappers arch=cray-sles15-zen3/3zy6uvs
==> Warning: Skipping external package: cray-libsci@21.08.1.2%cce@13.0.2~mpi~openmp+shared arch=cray-sles15-zen3/7uzhxpv
==> Warning: Skipping external package: cray-mpich@8.1.15%cce@13.0.2+wrappers arch=cray-sles15-zen3/tb5uxwe
==> Warning: Skipping external package: cuda@11.5.0%gcc@11.2.0~allow-unsupported-compilers~dev arch=cray-sles15-zen3/puekfe3
==> Error: 193 fatal error(s) when merging prefixes:
`/global/u1/e/elvis/spack-infrastructure/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/papi-6.0.0.1-s2y4nrvu6whr6hhgi63aa3nqwz2d35af/.spack/archived-files/src/removed_la_files.txt` and `/global/u1/e/elvis/spack-infrastructure/spack/opt/spack/cray-sles15-zen3/cce-13.0.2/papi-6.0.0.1-3aprcx5klzafe7xt6aq57jx5sequpue2/.spack/archived-files/src/removed_la_files.txt` both project to `.spack/papi/archived-files/src/removed_la_files.txt`
`/global/u1/e/elvis/spack-infrastructure/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/papi-6.0.0.1-s2y4nrvu6whr6hhgi63aa3nqwz2d35af/.spack/install_environment.json` and `/global/u1/e/elvis/spack-infrastructure/spack/opt/spack/cray-sles15-zen3/cce-13.0.2/papi-6.0.0.1-3aprcx5klzafe7xt6aq57jx5sequpue2/.spack/install_environment.json` both project to `.spack/papi/install_environment.json`
`/global/u1/e/elvis/spack-infrastructure/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/papi-6.0.0.1-s2y4nrvu6whr6hhgi63aa3nqwz2d35af/.spack/install_manifest.json` and `/global/u1/e/elvis/spack-infrastructure/spack/opt/spack/cray-sles15-zen3/cce-13.0.2/papi-6.0.0.1-3aprcx5klzafe7xt6aq57jx5sequpue2/.spack/install_manifest.json` both project to `.spack/papi/install_manifest.json`
Generating Modulefiles
In this section we let Spack generate modulefiles for the Spack packages we installed. Perlmutter is using Lmod as the module system which supports both
tcl and lua modules. You may want to refer to Modules for more information.
(spack-pyenv) elvis@login34> module --version
Modules based on Lua: Version 8.3.1 2020-02-16 19:46 :z
by Robert McLay mclay@tacc.utexas.edu
For this training we will cover how to generate tcl modules in a flat hierarchy. To get started, let’s add the following
to our Spack configuration:
1# This is a Spack Environment file.
2#
3# It describes a set of packages to be installed, along with
4# configuration settings.
5spack:
6 config:
7 view: false
8 concretization: separately
9 build_stage: $spack/var/spack/stage
10 misc_cache: $spack/var/spack/misc_cache
11 concretizer: clingo
12 compilers:
13 - compiler:
14 spec: gcc@11.2.0
15 paths:
16 cc: cc
17 cxx: CC
18 f77: ftn
19 fc: ftn
20 flags: {}
21 operating_system: sles15
22 target: any
23 modules:
24 - PrgEnv-gnu
25 - gcc/11.2.0
26 - craype-x86-milan
27 - libfabric
28 extra_rpaths: []
29 - compiler:
30 spec: cce@13.0.2
31 paths:
32 cc: /opt/cray/pe/craype/default/bin/cc
33 cxx: /opt/cray/pe/craype/default/bin/CC
34 f77: /opt/cray/pe/craype/default/bin/ftn
35 fc: /opt/cray/pe/craype/default/bin/ftn
36 flags: {}
37 operating_system: sles15
38 target: any
39 modules:
40 - PrgEnv-cray
41 - cce/13.0.2
42 - craype-x86-milan
43 - libfabric
44 environment: {}
45 extra_rpaths: []
46
47 # add package specs to the `specs` list
48 specs:
49 - papi %gcc
50 - papi %cce
51 - hypre %gcc
52 - hypre %cce
53 - darshan-runtime %gcc
54 - darshan-runtime %cce
55 - papi +cuda %gcc
56 packages:
57 all:
58 compiler: [gcc@11.2.0, cce@13.0.2]
59 providers:
60 blas: [cray-libsci]
61 mpi: [cray-mpich]
62 bzip2:
63 version: [1.0.6]
64 externals:
65 - spec: bzip2@1.0.6
66 prefix: /usr
67 cray-libsci:
68 buildable: false
69 externals:
70 - spec: cray-libsci@21.08.1.2
71 modules:
72 - cray-libsci/21.08.1.2
73 cray-mpich:
74 buildable: false
75 externals:
76 - spec: cray-mpich@8.1.15 %gcc@11.2.0
77 prefix: /opt/cray/pe/mpich/8.1.15/ofi/gnu/9.1
78 modules:
79 - cray-mpich/8.1.15
80 - cudatoolkit/11.5
81 - spec: cray-mpich@8.1.15 %cce@13.0.2
82 prefix: /opt/cray/pe/mpich/8.1.15/ofi/cray/10.0/
83 modules:
84 - cray-mpich/8.1.15
85 - cudatoolkit/11.5
86 cray-pmi:
87 buildable: false
88 externals:
89 - spec: cray-pmi@6.1.1
90 modules:
91 - cray-pmi/6.1.1
92 cuda:
93 buildable: false
94 version: [11.5.0]
95 externals:
96 - spec: cuda@11.5.0
97 prefix: /opt/nvidia/hpc_sdk/Linux_x86_64/21.11/cuda/11.5
98 modules:
99 - cudatoolkit/11.5
100 diffutils:
101 version: [3.6]
102 externals:
103 - spec: diffutils@3.6
104 prefix: /usr
105 findutils:
106 version: [4.6.0]
107 externals:
108 - spec: findutils@4.6.0
109 prefix: /usr
110 libfabric:
111 buildable: false
112 variants: fabrics=sockets,tcp,udp,rxm
113 externals:
114 - spec: libfabric@1.11.0.4.114
115 prefix: /opt/cray/libfabric/1.11.0.4.114
116 modules:
117 - libfabric/1.11.0.4.114
118 openssl:
119 version: [1.1.0i]
120 buildable: false
121 externals:
122 - spec: openssl@1.1.0i
123 prefix: /usr
124 openssh:
125 version: [7.9p1]
126 buildable: false
127 externals:
128 - spec: openssh@7.9p1
129 prefix: /usr
130 readline:
131 version: [7.0]
132 buildable: false
133 externals:
134 - spec: readline@7.0
135 prefix: /usr
136 tar:
137 version: [1.3]
138 buildable: false
139 externals:
140 - spec: tar@1.30
141 prefix: /usr
142 unzip:
143 version: [6.0]
144 buildable: false
145 externals:
146 - spec: unzip@6.0
147 prefix: /usr
148 modules:
149 default:
150 enable:
151 - tcl
152 tcl:
153 blacklist_implicits: true
154 hash_length: 0
155 naming_scheme: '{name}/{version}-{compiler.name}-{compiler.version}'
156 all:
157 autoload: direct
158 conflict:
159 - '{name}'
160 environment:
161 set:
162 '{name}_ROOT': '{prefix}'
163 suffixes:
164 ^cuda: cuda
165
166 view: true
The blacklist_implicits: true will ignore module generation for dependencies which is useful when you are building a large
software stack, you don’t want an explosion of modulefiles for utilities that you would never use. The hash_length: 0 will
avoid adding hash characters at end of modulefile, the naming_scheme will instruct Spack how to format the modulefiles
being written on the file-system. Now let’s generate the modulefiles. It is generally a good idea to run this in debug mode to understand how
files are being generated. The spack module tcl refresh command will generate tcl modules, it is good idea to specify --delete-tree -y
which will delete the root of module tree and -y will accept confirmation. In the output take note of where modulefiles are being written. You
will see a list of specs as BLACKLISTED_AS_IMPLICIT which are specs that will not generate modulefiles
1(spack-pyenv) elvis@login34> spack -d module tcl refresh --delete-tree -y
2==> [2022-08-04-09:42:35.558437] Reading config file /global/u1/e/elvis/spack-infrastructure/spack/etc/spack/defaults/config.yaml
3==> [2022-08-04-09:42:35.708144] Reading config file /global/u1/e/elvis/spack-infrastructure/spack/var/spack/environments/data_viz/spack.yaml
4==> [2022-08-04-09:42:35.767338] Using environment 'data_viz'
5==> [2022-08-04-09:42:35.968497] Imported module from built-in commands
6==> [2022-08-04-09:42:35.975354] Imported module from built-in commands
7==> [2022-08-04-09:42:35.991742] Reading config file /global/u1/e/elvis/spack-infrastructure/spack/etc/spack/defaults/bootstrap.yaml
8==> [2022-08-04-09:42:36.044748] DATABASE LOCK TIMEOUT: 3s
9==> [2022-08-04-09:42:36.044959] PACKAGE LOCK TIMEOUT: No timeout
10==> [2022-08-04-09:42:36.161175] Reading config file /global/u1/e/elvis/spack-infrastructure/spack/etc/spack/defaults/repos.yaml
11==> [2022-08-04-09:42:36.634555] Reading config file /global/u1/e/elvis/spack-infrastructure/spack/etc/spack/defaults/modules.yaml
12==> [2022-08-04-09:42:36.691668] Reading config file /global/u1/e/elvis/spack-infrastructure/spack/etc/spack/defaults/cray/modules.yaml
13==> [2022-08-04-09:42:38.077573] BLACKLISTED_AS_IMPLICIT : cray-libsci@21.08.1.2%cce@13.0.2~mpi~openmp+shared arch=cray-sles15-zen3/7uzhxpv
14==> [2022-08-04-09:42:38.079387] BLACKLISTED_AS_IMPLICIT : cray-libsci@21.08.1.2%gcc@11.2.0~mpi~openmp+shared arch=cray-sles15-zen3/jzbnd6y
15==> [2022-08-04-09:42:38.081189] BLACKLISTED_AS_IMPLICIT : cray-mpich@8.1.15%cce@13.0.2+wrappers arch=cray-sles15-zen3/tb5uxwe
16==> [2022-08-04-09:42:38.082661] BLACKLISTED_AS_IMPLICIT : cray-mpich@8.1.15%gcc@11.2.0+wrappers arch=cray-sles15-zen3/3zy6uvs
17==> [2022-08-04-09:42:38.084601] BLACKLISTED_AS_IMPLICIT : cuda@11.5.0%gcc@11.2.0~allow-unsupported-compilers~dev arch=cray-sles15-zen3/puekfe3
18==> [2022-08-04-09:42:38.097284] BLACKLISTED_AS_IMPLICIT : zlib@1.2.12%cce@13.0.2+optimize+pic+shared patches=0d38234 arch=cray-sles15-zen3/e2hl6cx
19==> [2022-08-04-09:42:38.099494] BLACKLISTED_AS_IMPLICIT : zlib@1.2.12%gcc@11.2.0+optimize+pic+shared patches=0d38234 arch=cray-sles15-zen3/ozmcyfj
20==> [2022-08-04-09:44:22.697989] Regenerating tcl module files
21==> [2022-08-04-09:44:22.872234] WRITE: darshan-runtime@3.3.1%cce@13.0.2~apmpi~apmpi_sync~apxc~hdf5+mpi scheduler=NONE arch=cray-sles15-zen3/uj3wa4a [/global/u1/e/elvis/spack-infrastructure/spack/share/spack/modules/cray-sles15-zen3/darshan-runtime/3.3.1-cce-13.0.2]
22==> [2022-08-04-09:44:23.696894] Module name: cce/13.0.2
23==> [2022-08-04-09:44:23.697138] Package directory variable prefix: CCE
24==> [2022-08-04-09:44:23.959854] Module name: cce/13.0.2
25==> [2022-08-04-09:44:23.960027] Package directory variable prefix: CCE
26==> [2022-08-04-09:44:24.183730] Module name: cce/13.0.2
27==> [2022-08-04-09:44:24.183920] Package directory variable prefix: CCE
28==> [2022-08-04-09:44:24.810258] Module name: cce/13.0.2
29==> [2022-08-04-09:44:24.810473] Package directory variable prefix: CCE
30==> [2022-08-04-09:44:25.037930] Module name: cce/13.0.2
31==> [2022-08-04-09:44:25.038163] Package directory variable prefix: CCE
32==> [2022-08-04-09:44:25.052737] BLACKLISTED_AS_IMPLICIT : cray-mpich@8.1.15%cce@13.0.2+wrappers arch=cray-sles15-zen3/tb5uxwe
33==> [2022-08-04-09:44:25.056012] BLACKLISTED_AS_IMPLICIT : zlib@1.2.12%cce@13.0.2+optimize+pic+shared patches=0d38234 arch=cray-sles15-zen3/e2hl6cx
34==> [2022-08-04-09:44:25.060927] Reading config file /global/u1/e/elvis/spack-infrastructure/spack/etc/spack/defaults/packages.yaml
35==> [2022-08-04-09:44:25.113314] WRITE: darshan-runtime@3.3.1%gcc@11.2.0~apmpi~apmpi_sync~apxc~hdf5+mpi scheduler=NONE arch=cray-sles15-zen3/hkxzwvt [/global/u1/e/elvis/spack-infrastructure/spack/share/spack/modules/cray-sles15-zen3/darshan-runtime/3.3.1-gcc-11.2.0]
36==> [2022-08-04-09:44:25.219719] BLACKLISTED_AS_IMPLICIT : cray-mpich@8.1.15%gcc@11.2.0+wrappers arch=cray-sles15-zen3/3zy6uvs
37==> [2022-08-04-09:44:25.222960] BLACKLISTED_AS_IMPLICIT : zlib@1.2.12%gcc@11.2.0+optimize+pic+shared patches=0d38234 arch=cray-sles15-zen3/ozmcyfj
38==> [2022-08-04-09:44:25.258546] WRITE: hypre@2.24.0%cce@13.0.2~complex~cuda~debug+fortran~gptune~int64~internal-superlu~mixedint+mpi~openmp~rocm+shared~superlu-dist~unified-memory arch=cray-sles15-zen3/62ofdsf [/global/u1/e/elvis/spack-infrastructure/spack/share/spack/modules/cray-sles15-zen3/hypre/2.24.0-cce-13.0.2]
39==> [2022-08-04-09:44:25.550468] Module name: cce/13.0.2
40==> [2022-08-04-09:44:25.550681] Package directory variable prefix: CCE
41==> [2022-08-04-09:44:25.785678] Module name: cce/13.0.2
42==> [2022-08-04-09:44:25.785853] Package directory variable prefix: CCE
43==> [2022-08-04-09:44:25.995944] Module name: cce/13.0.2
44==> [2022-08-04-09:44:25.996162] Package directory variable prefix: CCE
45==> [2022-08-04-09:44:26.212011] Module name: cce/13.0.2
46==> [2022-08-04-09:44:26.212283] Package directory variable prefix: CCE
47==> [2022-08-04-09:44:26.225681] BLACKLISTED_AS_IMPLICIT : cray-libsci@21.08.1.2%cce@13.0.2~mpi~openmp+shared arch=cray-sles15-zen3/7uzhxpv
48==> [2022-08-04-09:44:26.230079] BLACKLISTED_AS_IMPLICIT : cray-mpich@8.1.15%cce@13.0.2+wrappers arch=cray-sles15-zen3/tb5uxwe
49==> [2022-08-04-09:44:26.238876] WRITE: hypre@2.24.0%gcc@11.2.0~complex~cuda~debug+fortran~gptune~int64~internal-superlu~mixedint+mpi~openmp~rocm+shared~superlu-dist~unified-memory arch=cray-sles15-zen3/mbn7bum [/global/u1/e/elvis/spack-infrastructure/spack/share/spack/modules/cray-sles15-zen3/hypre/2.24.0-gcc-11.2.0]
50==> [2022-08-04-09:44:26.385208] BLACKLISTED_AS_IMPLICIT : cray-libsci@21.08.1.2%gcc@11.2.0~mpi~openmp+shared arch=cray-sles15-zen3/jzbnd6y
51==> [2022-08-04-09:44:26.388329] BLACKLISTED_AS_IMPLICIT : cray-mpich@8.1.15%gcc@11.2.0+wrappers arch=cray-sles15-zen3/3zy6uvs
52==> [2022-08-04-09:44:26.398423] WRITE: papi@6.0.0.1%cce@13.0.2~cuda+example~infiniband~lmsensors~nvml~powercap~rapl~rocm~rocm_smi~sde+shared~static_tools patches=b6d6caa arch=cray-sles15-zen3/3aprcx5 [/global/u1/e/elvis/spack-infrastructure/spack/share/spack/modules/cray-sles15-zen3/papi/6.0.0.1-cce-13.0.2]
53==> [2022-08-04-09:44:26.749919] Module name: cce/13.0.2
54==> [2022-08-04-09:44:26.750092] Package directory variable prefix: CCE
55==> [2022-08-04-09:44:26.762459] WRITE: papi@6.0.0.1%gcc@11.2.0~cuda+example~infiniband~lmsensors~nvml~powercap~rapl~rocm~rocm_smi~sde+shared~static_tools arch=cray-sles15-zen3/s2y4nrv [/global/u1/e/elvis/spack-infrastructure/spack/share/spack/modules/cray-sles15-zen3/papi/6.0.0.1-gcc-11.2.0]
56==> [2022-08-04-09:44:26.897249] WRITE: papi@6.0.0.1%gcc@11.2.0+cuda+example~infiniband~lmsensors~nvml~powercap~rapl~rocm~rocm_smi~sde+shared~static_tools arch=cray-sles15-zen3/x43djbq [/global/u1/e/elvis/spack-infrastructure/spack/share/spack/modules/cray-sles15-zen3/papi/6.0.0.1-gcc-11.2.0-cuda]
57==> [2022-08-04-09:44:27.240985] Module name: gcc/11.2.0
58==> [2022-08-04-09:44:27.241199] Package directory variable prefix: GCC
59==> [2022-08-04-09:44:27.316093] BLACKLISTED_AS_IMPLICIT : cuda@11.5.0%gcc@11.2.0~allow-unsupported-compilers~dev arch=cray-sles15-zen3/puekfe3
Spack will generate the modulefiles, in its default location $SPACK_ROOT/share/spack/modules which is organized by architecture (spack arch) as shown below:
(spack-pyenv) elvis@login34> ls $SPACK_ROOT/share/spack/modules/$(spack arch)/*
/global/homes/e/elvis/spack-infrastructure/spack/share/spack/modules/cray-sles15-zen3/darshan-runtime:
3.3.1-cce-13.0.2 3.3.1-gcc-11.2.0
/global/homes/e/elvis/spack-infrastructure/spack/share/spack/modules/cray-sles15-zen3/hypre:
2.24.0-cce-13.0.2 2.24.0-gcc-11.2.0
/global/homes/e/elvis/spack-infrastructure/spack/share/spack/modules/cray-sles15-zen3/papi:
6.0.0.1-cce-13.0.2 6.0.0.1-gcc-11.2.0 6.0.0.1-gcc-11.2.0-cuda
Let’s change the directory path such that modulefiles are not inside Spack’s root directory and they are easy to remember. For this
exercise let’s generate the modulefiles in your $HOME/spack-infrastructure/modules directory.
1# This is a Spack Environment file.
2#
3# It describes a set of packages to be installed, along with
4# configuration settings.
5spack:
6 config:
7 view: false
8 concretization: separately
9 build_stage: $spack/var/spack/stage
10 misc_cache: $spack/var/spack/misc_cache
11 concretizer: clingo
12 compilers:
13 - compiler:
14 spec: gcc@11.2.0
15 paths:
16 cc: cc
17 cxx: CC
18 f77: ftn
19 fc: ftn
20 flags: {}
21 operating_system: sles15
22 target: any
23 modules:
24 - PrgEnv-gnu
25 - gcc/11.2.0
26 - craype-x86-milan
27 - libfabric
28 extra_rpaths: []
29 - compiler:
30 spec: cce@13.0.2
31 paths:
32 cc: /opt/cray/pe/craype/default/bin/cc
33 cxx: /opt/cray/pe/craype/default/bin/CC
34 f77: /opt/cray/pe/craype/default/bin/ftn
35 fc: /opt/cray/pe/craype/default/bin/ftn
36 flags: {}
37 operating_system: sles15
38 target: any
39 modules:
40 - PrgEnv-cray
41 - cce/13.0.2
42 - craype-x86-milan
43 - libfabric
44 environment: {}
45 extra_rpaths: []
46
47 # add package specs to the `specs` list
48 specs:
49 - papi %gcc
50 - papi %cce
51 - hypre %gcc
52 - hypre %cce
53 - darshan-runtime %gcc
54 - darshan-runtime %cce
55 - papi +cuda %gcc
56 packages:
57 all:
58 compiler: [gcc@11.2.0, cce@13.0.2]
59 providers:
60 blas: [cray-libsci]
61 mpi: [cray-mpich]
62 bzip2:
63 version: [1.0.6]
64 externals:
65 - spec: bzip2@1.0.6
66 prefix: /usr
67 cray-libsci:
68 buildable: false
69 externals:
70 - spec: cray-libsci@21.08.1.2
71 modules:
72 - cray-libsci/21.08.1.2
73 cray-mpich:
74 buildable: false
75 externals:
76 - spec: cray-mpich@8.1.15 %gcc@11.2.0
77 prefix: /opt/cray/pe/mpich/8.1.15/ofi/gnu/9.1
78 modules:
79 - cray-mpich/8.1.15
80 - cudatoolkit/11.5
81 - spec: cray-mpich@8.1.15 %cce@13.0.2
82 prefix: /opt/cray/pe/mpich/8.1.15/ofi/cray/10.0/
83 modules:
84 - cray-mpich/8.1.15
85 - cudatoolkit/11.5
86 cray-pmi:
87 buildable: false
88 externals:
89 - spec: cray-pmi@6.1.1
90 modules:
91 - cray-pmi/6.1.1
92 cuda:
93 buildable: false
94 version: [11.5.0]
95 externals:
96 - spec: cuda@11.5.0
97 prefix: /opt/nvidia/hpc_sdk/Linux_x86_64/21.11/cuda/11.5
98 modules:
99 - cudatoolkit/11.5
100 diffutils:
101 version: [3.6]
102 externals:
103 - spec: diffutils@3.6
104 prefix: /usr
105 findutils:
106 version: [4.6.0]
107 externals:
108 - spec: findutils@4.6.0
109 prefix: /usr
110 libfabric:
111 buildable: false
112 variants: fabrics=sockets,tcp,udp,rxm
113 externals:
114 - spec: libfabric@1.11.0.4.114
115 prefix: /opt/cray/libfabric/1.11.0.4.114
116 modules:
117 - libfabric/1.11.0.4.114
118 openssl:
119 version: [1.1.0i]
120 buildable: false
121 externals:
122 - spec: openssl@1.1.0i
123 prefix: /usr
124 openssh:
125 version: [7.9p1]
126 buildable: false
127 externals:
128 - spec: openssh@7.9p1
129 prefix: /usr
130 readline:
131 version: [7.0]
132 buildable: false
133 externals:
134 - spec: readline@7.0
135 prefix: /usr
136 tar:
137 version: [1.3]
138 buildable: false
139 externals:
140 - spec: tar@1.30
141 prefix: /usr
142 unzip:
143 version: [6.0]
144 buildable: false
145 externals:
146 - spec: unzip@6.0
147 prefix: /usr
148 modules:
149 default:
150 enable:
151 - tcl
152 roots:
153 tcl: /global/homes/e/elvis/spack-infrastructure/modules
154 tcl:
155 blacklist_implicits: true
156 hash_length: 0
157 naming_scheme: '{name}/{version}-{compiler.name}-{compiler.version}'
158 all:
159 autoload: direct
160 conflict:
161 - '{name}'
162 environment:
163 set:
164 '{name}_ROOT': '{prefix}'
165 suffixes:
166 ^cuda: cuda
167
168 view: true
Now you will see the modulefiles are written in $HOME/spack-infrastructure/modules.
(spack-pyenv) elvis@login34> spack -d module tcl refresh --delete-tree -y
==> [2022-08-04-09:53:00.452047] Reading config file /global/u1/e/elvis/spack-infrastructure/spack/etc/spack/defaults/config.yaml
==> [2022-08-04-09:53:00.563502] Reading config file /global/u1/e/elvis/spack-infrastructure/spack/var/spack/environments/data_viz/spack.yaml
==> [2022-08-04-09:53:00.617365] Using environment 'data_viz'
==> [2022-08-04-09:53:00.625951] Imported module from built-in commands
==> [2022-08-04-09:53:00.632039] Imported module from built-in commands
==> [2022-08-04-09:53:00.637512] Reading config file /global/u1/e/elvis/spack-infrastructure/spack/etc/spack/defaults/bootstrap.yaml
==> [2022-08-04-09:53:00.654001] DATABASE LOCK TIMEOUT: 3s
==> [2022-08-04-09:53:00.654065] PACKAGE LOCK TIMEOUT: No timeout
==> [2022-08-04-09:53:00.657750] Reading config file /global/u1/e/elvis/spack-infrastructure/spack/etc/spack/defaults/repos.yaml
==> [2022-08-04-09:53:00.670487] Reading config file /global/u1/e/elvis/spack-infrastructure/spack/etc/spack/defaults/modules.yaml
==> [2022-08-04-09:53:00.687615] Reading config file /global/u1/e/elvis/spack-infrastructure/spack/etc/spack/defaults/cray/modules.yaml
==> [2022-08-04-09:53:00.891563] BLACKLISTED_AS_IMPLICIT : cray-libsci@21.08.1.2%cce@13.0.2~mpi~openmp+shared arch=cray-sles15-zen3/7uzhxpv
==> [2022-08-04-09:53:00.892858] BLACKLISTED_AS_IMPLICIT : cray-libsci@21.08.1.2%gcc@11.2.0~mpi~openmp+shared arch=cray-sles15-zen3/jzbnd6y
==> [2022-08-04-09:53:00.894129] BLACKLISTED_AS_IMPLICIT : cray-mpich@8.1.15%cce@13.0.2+wrappers arch=cray-sles15-zen3/tb5uxwe
==> [2022-08-04-09:53:00.895334] BLACKLISTED_AS_IMPLICIT : cray-mpich@8.1.15%gcc@11.2.0+wrappers arch=cray-sles15-zen3/3zy6uvs
==> [2022-08-04-09:53:00.896502] BLACKLISTED_AS_IMPLICIT : cuda@11.5.0%gcc@11.2.0~allow-unsupported-compilers~dev arch=cray-sles15-zen3/puekfe3
==> [2022-08-04-09:53:00.904007] BLACKLISTED_AS_IMPLICIT : zlib@1.2.12%cce@13.0.2+optimize+pic+shared patches=0d38234 arch=cray-sles15-zen3/e2hl6cx
==> [2022-08-04-09:53:00.905394] BLACKLISTED_AS_IMPLICIT : zlib@1.2.12%gcc@11.2.0+optimize+pic+shared patches=0d38234 arch=cray-sles15-zen3/ozmcyfj
==> [2022-08-04-09:53:03.555915] Regenerating tcl module files
==> [2022-08-04-09:53:03.577058] WRITE: darshan-runtime@3.3.1%cce@13.0.2~apmpi~apmpi_sync~apxc~hdf5+mpi scheduler=NONE arch=cray-sles15-zen3/uj3wa4a [/global/homes/e/elvis/spack-infrastructure/modules/cray-sles15-zen3/darshan-runtime/3.3.1-cce-13.0.2]
==> [2022-08-04-09:53:04.003818] Module name: cce/13.0.2
==> [2022-08-04-09:53:04.004044] Package directory variable prefix: CCE
==> [2022-08-04-09:53:04.248393] Module name: cce/13.0.2
==> [2022-08-04-09:53:04.248675] Package directory variable prefix: CCE
==> [2022-08-04-09:53:04.484157] Module name: cce/13.0.2
==> [2022-08-04-09:53:04.484420] Package directory variable prefix: CCE
==> [2022-08-04-09:53:04.766465] Module name: cce/13.0.2
==> [2022-08-04-09:53:04.766692] Package directory variable prefix: CCE
==> [2022-08-04-09:53:05.024080] Module name: cce/13.0.2
==> [2022-08-04-09:53:05.024335] Package directory variable prefix: CCE
==> [2022-08-04-09:53:05.043781] BLACKLISTED_AS_IMPLICIT : cray-mpich@8.1.15%cce@13.0.2+wrappers arch=cray-sles15-zen3/tb5uxwe
==> [2022-08-04-09:53:05.048836] BLACKLISTED_AS_IMPLICIT : zlib@1.2.12%cce@13.0.2+optimize+pic+shared patches=0d38234 arch=cray-sles15-zen3/e2hl6cx
==> [2022-08-04-09:53:05.055298] Reading config file /global/u1/e/elvis/spack-infrastructure/spack/etc/spack/defaults/packages.yaml
==> [2022-08-04-09:53:05.111091] WRITE: darshan-runtime@3.3.1%gcc@11.2.0~apmpi~apmpi_sync~apxc~hdf5+mpi scheduler=NONE arch=cray-sles15-zen3/hkxzwvt [/global/homes/e/elvis/spack-infrastructure/modules/cray-sles15-zen3/darshan-runtime/3.3.1-gcc-11.2.0]
==> [2022-08-04-09:53:05.161578] BLACKLISTED_AS_IMPLICIT : cray-mpich@8.1.15%gcc@11.2.0+wrappers arch=cray-sles15-zen3/3zy6uvs
==> [2022-08-04-09:53:05.164707] BLACKLISTED_AS_IMPLICIT : zlib@1.2.12%gcc@11.2.0+optimize+pic+shared patches=0d38234 arch=cray-sles15-zen3/ozmcyfj
==> [2022-08-04-09:53:05.171012] WRITE: hypre@2.24.0%cce@13.0.2~complex~cuda~debug+fortran~gptune~int64~internal-superlu~mixedint+mpi~openmp~rocm+shared~superlu-dist~unified-memory arch=cray-sles15-zen3/62ofdsf [/global/homes/e/elvis/spack-infrastructure/modules/cray-sles15-zen3/hypre/2.24.0-cce-13.0.2]
==> [2022-08-04-09:53:05.469562] Module name: cce/13.0.2
==> [2022-08-04-09:53:05.469791] Package directory variable prefix: CCE
==> [2022-08-04-09:53:05.767046] Module name: cce/13.0.2
==> [2022-08-04-09:53:05.767239] Package directory variable prefix: CCE
==> [2022-08-04-09:53:06.050449] Module name: cce/13.0.2
==> [2022-08-04-09:53:06.050663] Package directory variable prefix: CCE
==> [2022-08-04-09:53:06.295722] Module name: cce/13.0.2
==> [2022-08-04-09:53:06.295923] Package directory variable prefix: CCE
==> [2022-08-04-09:53:06.307895] BLACKLISTED_AS_IMPLICIT : cray-libsci@21.08.1.2%cce@13.0.2~mpi~openmp+shared arch=cray-sles15-zen3/7uzhxpv
==> [2022-08-04-09:53:06.313024] BLACKLISTED_AS_IMPLICIT : cray-mpich@8.1.15%cce@13.0.2+wrappers arch=cray-sles15-zen3/tb5uxwe
==> [2022-08-04-09:53:06.321590] WRITE: hypre@2.24.0%gcc@11.2.0~complex~cuda~debug+fortran~gptune~int64~internal-superlu~mixedint+mpi~openmp~rocm+shared~superlu-dist~unified-memory arch=cray-sles15-zen3/mbn7bum [/global/homes/e/elvis/spack-infrastructure/modules/cray-sles15-zen3/hypre/2.24.0-gcc-11.2.0]
==> [2022-08-04-09:53:06.366559] BLACKLISTED_AS_IMPLICIT : cray-libsci@21.08.1.2%gcc@11.2.0~mpi~openmp+shared arch=cray-sles15-zen3/jzbnd6y
==> [2022-08-04-09:53:06.369882] BLACKLISTED_AS_IMPLICIT : cray-mpich@8.1.15%gcc@11.2.0+wrappers arch=cray-sles15-zen3/3zy6uvs
==> [2022-08-04-09:53:06.377335] WRITE: papi@6.0.0.1%cce@13.0.2~cuda+example~infiniband~lmsensors~nvml~powercap~rapl~rocm~rocm_smi~sde+shared~static_tools patches=b6d6caa arch=cray-sles15-zen3/3aprcx5 [/global/homes/e/elvis/spack-infrastructure/modules/cray-sles15-zen3/papi/6.0.0.1-cce-13.0.2]
==> [2022-08-04-09:53:06.656390] Module name: cce/13.0.2
==> [2022-08-04-09:53:06.656565] Package directory variable prefix: CCE
==> [2022-08-04-09:53:06.670466] WRITE: papi@6.0.0.1%gcc@11.2.0~cuda+example~infiniband~lmsensors~nvml~powercap~rapl~rocm~rocm_smi~sde+shared~static_tools arch=cray-sles15-zen3/s2y4nrv [/global/homes/e/elvis/spack-infrastructure/modules/cray-sles15-zen3/papi/6.0.0.1-gcc-11.2.0]
==> [2022-08-04-09:53:06.719655] WRITE: papi@6.0.0.1%gcc@11.2.0+cuda+example~infiniband~lmsensors~nvml~powercap~rapl~rocm~rocm_smi~sde+shared~static_tools arch=cray-sles15-zen3/x43djbq [/global/homes/e/elvis/spack-infrastructure/modules/cray-sles15-zen3/papi/6.0.0.1-gcc-11.2.0-cuda]
==> [2022-08-04-09:53:07.034250] Module name: gcc/11.2.0
==> [2022-08-04-09:53:07.034531] Package directory variable prefix: GCC
==> [2022-08-04-09:53:07.055549] BLACKLISTED_AS_IMPLICIT : cuda@11.5.0%gcc@11.2.0~allow-unsupported-compilers~dev arch=cray-sles15-zen3/puekfe3
We can see that Spack has generated the modulefiles in the format of {name}/{version}-{compiler.name}-{compiler.version}. For example,
the -cuda suffix was added for the PAPI module that has CUDA-enabled features.
(spack-pyenv) elvis@login34> ls -l $CI_PROJECT_DIR/modules/$(spack arch)/*
/global/homes/e/elvis/spack-infrastructure/modules/cray-sles15-zen3/darshan-runtime:
total 8
-rw-r--r-- 1 elvis elvis 2245 Aug 4 09:53 3.3.1-cce-13.0.2
-rw-r--r-- 1 elvis elvis 2243 Aug 4 09:53 3.3.1-gcc-11.2.0
/global/homes/e/elvis/spack-infrastructure/modules/cray-sles15-zen3/hypre:
total 8
-rw-r--r-- 1 elvis elvis 1951 Aug 4 09:53 2.24.0-cce-13.0.2
-rw-r--r-- 1 elvis elvis 1943 Aug 4 09:53 2.24.0-gcc-11.2.0
/global/homes/e/elvis/spack-infrastructure/modules/cray-sles15-zen3/papi:
total 12
-rw-r--r-- 1 elvis elvis 2441 Aug 4 09:53 6.0.0.1-cce-13.0.2
-rw-r--r-- 1 elvis elvis 2425 Aug 4 09:53 6.0.0.1-gcc-11.2.0
-rw-r--r-- 1 elvis elvis 2503 Aug 4 09:53 6.0.0.1-gcc-11.2.0-cuda
We can add this directory to MODULEPATH by running the following:
(spack-pyenv) elvis@login34> module use $CI_PROJECT_DIR/modules/$(spack arch)
Next, if we run ml av we will see the modules generated from Spack that correspond to the installed Spack packages.
(spack-pyenv) elvis@login34> ml av
------------------------------------ /global/homes/e/elvis/spack-infrastructure/modules/cray-sles15-zen3 -------------------------------------
darshan-runtime/3.3.1-cce-13.0.2 hypre/2.24.0-cce-13.0.2 papi/6.0.0.1-cce-13.0.2 papi/6.0.0.1-gcc-11.2.0
darshan-runtime/3.3.1-gcc-11.2.0 (D) hypre/2.24.0-gcc-11.2.0 (D) papi/6.0.0.1-gcc-11.2.0-cuda
This concludes the Spack training.