2. Quickstart

git clone --recursive https://github.com/NOAA-EMC/spack-stack.git
cd spack-stack

# Ensure Python 3.7+ is available and the default before sourcing spack

# Sources Spack from submodule and sets ${SPACK_STACK_DIR}
source setup.sh

# Basic usage of spack stack create
spack stack create -h

Note

We recommend using external Python versions 3.8 or later with spack-stack.

2.1. Using spack to create environments and containers

2.1.1. Create local environment

The following instructions install a new spack environment on a pre-configured site (see Section 3.1 for a list of pre-configured sites and site-specific notes). Instructions for creating a new site config on an configurable system (i.e. a generic Linux or macOS system) can be found in Section 3.2. The options for the spack stack extension are explained in Section 6.

# See a list of sites and templates
spack stack create env -h

# Create a pre-configured Spack environment in envs/<template>.<site>
# (copies site-specific, application-specific, and common config files into the environment directory)
spack stack create env --site hera --template ufs-weather-model --name ufs-weather-model.hera

# Activate the newly created environment
# Optional: decorate the command line prompt using -p
#     Note: in some cases, this can mess up long lines in bash
#     because color codes are not escaped correctly. In this
#     case, use export SPACK_COLOR='never' first.
spack env activate [-p] envs/jedi-fv3.hera

# Optionally edit config files (spack.yaml, packages.yaml compilers.yaml, site.yaml)
emacs envs/jedi-fv3.hera/spack.yaml
emacs envs/jedi-fv3.hera/common/*.yaml
emacs envs/jedi-fv3.hera/site/*.yaml

# Process/concretize the specs
spack concretize

# Optional step for systems with a pre-configured spack mirror, see below.

# Install the environment, recommended to always use --source
# to install the source code with the compiled binary package
spack install --source [--verbose] [--fail-fast]

# Create lua module files
spack module lmod refresh

# Create meta-modules for compiler, mpi, python
spack stack setup-meta-modules

Note

You may want to capture the output from spack concretize and spack install comands in log files. For example:

spack concretize 2>&1 | tee log.concretize
spack install [--verbose] [--fail-fast] 2>&1 | tee log.install

Note

For platforms with multiple compilers in the site config, make sure that the correct compiler and corresponding MPI library are set correctly in envs/jedi-fv3.hera/site/packages.yaml before running spack concretize. Also, check the output of spack concretize to make sure that the correct compiler is used (e.g. %intel-2022.0.1). If not, edit envs/jedi-fv3.hera/site/compilers.yaml and remove the offending compiler. Then, remove envs/jedi-fv3.hera/spack.lock and rerun spack concretize.

2.1.1.1. Optional step for sites with a preconfigured spack mirror

To check if a mirror is configured, look for local-source in the output of

spack mirror list

If a mirror exists, add new packages to the mirror. Here, /path/to/mirror is the location from the above list command without the leading file://

spack mirror create -a -d /path/to/mirror

If this fails with git lfs errors, check the site config for which module to load for git lfs support. Load the module, then run the spack mirror add command, then unload the module and proceed with the installation.

2.1.2. Create container

In this example, a container is created with an empty template, and specs are added manually. It is also possible to start with a different template, but it is important to know that container builds do not allow for multiple versions of the same package (e.g., fms@2022.01 and fms@release-jcsda), therefore not all templates will work (one can remove certain specs from the build, as long as this does not impact the usability of the container).

# See a list of preconfigured containers
spack stack create ctr -h

# Create container spack definition (spack.yaml) in directory envs/<container-config>
spack stack create ctr docker-ubuntu-gcc-openmpi --template=empty

# Descend into container environment directory
cd envs/docker-ubuntu-gcc-openmpi

# Edit config file and add the required specs in section "specs:"
emacs spack.yaml

# Docker: create Dockerfile and build container
# See section "container" in spack.yaml for additional information
spack containerize > Dockerfile
docker build -t myimage .
docker run -it myimage

2.1.3. Extending environments

Additional packages (and their dependencies) or new versions of packages can be added to existing environments. It is recommended to take a backup of the existing environment directory (e.g. using rsync) or test this first as described in Section 8.2, especially if new versions of packages are added that act themselves as dependencies for other packages. In some cases, adding new versions of packages will require rebuilding large portions of the stack, for example if a new version of hdf5 is needed. In this case, it is recommended to start over with an entirely new environment.

In the simplest case, a new package (and its basic dependencies) or a new version of an existing package that is not a dependency for other packages can be added as described in the following example for a new version of ecmwf-atlas.

  1. Check if the package has any variants defined in the common (env_dir/common/packages.yaml) or site (env_dir/site/packages.yaml) package config and make sure that these are reflected correctly in the spec command:

spack spec ecmwf-atlas@0.29.0

  1. Add package to environment specs:

spack add ecmwf-atlas@0.29.0

  1. Run concretize step

spack concretize

  1. Install

spack install [--verbose] [--fail-fast]

Further information on how to define variants for new packages, how to use these non-standard versions correctly as dependencies, …, can be found in the Spack Documentation. Details on the spack stack extension of the spack are provided in Section 6.

Note

Instead of spack add ecmwf-atlas@0.29.0, spack concretize and spack install, one can also just use spack install ecmwf-atlas@0.29.0 after checking in the first step (spack spec) that the package will be installed as desired.

2.2. Using a spack environment to compile and run code

Spack environments are used by loading the modulefiles that generated at the end of the installation process. The spack command itself is not needed in this setup, hence the instructions for creating new environments (source setup.sh etc.) can be ignored. The following is sufficient for loading the modules and using them to compile and run user code.

2.2.1. Pre-configured sites

For pre-configured sites, follow the instructions in Section 3.1 to set the basic environment.

Note

Customizations of the user environment in .bashrc, .bash_profile, …, that load certain modules automatically may interfere with the setup. It is highly advised to avoid “polluting” the standard environment, i.e. to keep the default environment as clean as possible, and create shell scripts that can be sourced to conveniently configure a user environment for a specific task instead.

Next, load the spack meta-modules directory into the module path using

module use $LOCATION/modulefiles/Core

where $LOCATION refers to the install location listed in the table in Section 3.1. Loading the compiler meta-module will give access to the Python and MPI provider module and to packages that only depend on the compiler, not on the MPI provider. Loading the MPI meta-module will then add the MPI-dependent packages to the module path. Use module available to look for the exact names of the meta-modules.

module load stack-compiler-name/compiler-version
module load stack-python-name/python-version
module load stack-mpi-name/mpi-version

After that, list all available modules via module available. For the environment packages described in Section Section 4, convenience modules are created that can be loaded and that automatically load the required dependency modules.

Note

When using lua modules, loading a different module will automatically switch the dependency modules. This is not the case for tcl modules. For the latter, it is recommended to start over with a clean shell and repeat the above steps.

2.2.2. Configurable sites (generic macOS, Linux)

The process for configurable sites is identical to that for pre-configured sites described above. $LOCATION in this case needs to be replaced with the install directory for the spack packages, which by default is subdirectory install in the environment directory.