Spack Infrastructure

The spack infrastructure repository contains spack configuration in the form of spack.yaml required to build spack stacks on Cori and Perlmutter system. We leverage gitlab to automate software stack deployment which is configured using the .gitlab-ci.yml file. The documentation is available at

Spack Configuration

The spack configuration can be found in spack-configs directory with subdirectory for each deployment. Each pipeline can be run if one sets the variable PIPELINE_NAME to a unique value in order to run a pipeline. You can check the .gitlab-ci.yml for the gitlab configuration. The pipeline can be run via web interface, if you chose this route, you must set PIPELINE_NAME to the appropriate value.

If you want to trigger pipeline via web-interface you will need to define PIPELINE_NAME variable to trigger the appropriate pipeline.

Running CI Pipelines

This project is configured with several scheduled pipelines that will run at different times.

Currently, we have a shell runner installed on Perlmutter using e4s account which is configured with following settings. You can find list of runners and their runner status under Settings > CI/CD > Runners. Please make sure you login to the appropriate hostname when starting the gitlab runner.


Runner Name














The runner configuration files are located in ~/.gitlab-runner for user e4s.

The production pipelines are triggered via web-interface which requires approval from a project maintainer. Production pipelines should be run when we need to do full redeployment of stack.

Current Challenges

There are several challenges with building spack stack at NERSC which can be summarized as follows

  • System OS + Cray Programming Environment (CPE) changes: A system upgrade such as change to glibc or upgrades in CPE can lead to full software stack rebuild, especially if you have external packages set for packages like cray-mpich, cray-libsci which generally change between versions

  • Incompatibile compilers: Some packages can’t be built with certain compilers (nvhpc, aocc) which could be due to several factors.

    • An application doesn’t have support though it was be added in newer version but you don’t have it in your spack release used for deployment

    • Lack of support in spack package recipe or spack-core base including spack-cray detection. This may require getting fix and cherry-pick commit or waiting for new version

    • Spack Cray detection is an important part in build errors including how one specifies externals via modules vs prefix both could be provided and it requires experimentation. An example of this is trying to get cray-mpich external one could set something like this with modules or prefix

        buildable: false
        - spec: cray-mpich@8.1.11 %gcc@9.3.0
          prefix: /opt/cray/pe/mpich/8.1.11/ofi/gnu/9.1
          - cray-mpich/8.1.11
          - cudatoolkit/21.9_11.4
    • Spack concretizer prevent one from chosing a build configration for a spec. This requires a few troubleshooting step but usually boils down to:

      • Read the spack package file spack edit <package> for conflicts and try spack spec to see concretized spec.

      • Try different version, different compiler, different dependency. Some packages have conflicting variant for instance one can’t enable +openmp and +pthread it is mutually exclusive.

There is a document Spack E4S Issues on Permlutter outlining current issues with spack. If you need access to document please contact Shahzeb Siddiqui.


If you need elevated privledge or assistance with this project please contact one of the maintainers:

  • Shahzeb Siddiqui (

  • Erik Palmer (

  • Justin Cook (

  • E4S Team: Sameer Shende (, Christopher Peyralans (, Wyatt Spear (, Nicholas Chaimov (