11. Known Issues

11.1. General

1. gcc@13 (gcc, g++, gfortran) not yet supported

   Our software stack doesn’t build with gcc@13 yet. This is also true when combining the LLVM or Apple clang compiler with gfortran@13.

2. Build errors for mapl@2.35.2 with mpich@4.1.1

   This problem is described in https://github.com/JCSDA/spack-stack/issues/608.

3. Issues starting/finding ecflow_server due to a mismatch of hostnames

   On some systems, ecflow_server gets confused by multiple hostnames, e.g. localhost and MYORG-L-12345. The ecflow_start.sh script reports the hostname it wants to use. This name (or both) must be in /etc/hosts in the correct address line, often the loopback address (127.0.0.1).

4. Installation of duplicate packages ecbuild, hdf5

   One reason for this is an external cmake@3.20 installation, which confuses the concretizer when building a complex environment such as the skylab-dev or `unified-dev environment. For certain packages (and thus their dependencies), a newer version than cmake@3.20 is required, for others cmake@3.20 works, and spack then thinks that it needs to build two identical versions of the same package with different versions of cmake. The solution is to remove any external cmake@3.20 package (and best also earlier versions) in the site config and run the concretization step again. Another reason on Ubuntu 20 is the presence of external openssl packages, which should be removed before re-running the concretization step.

5. Installation of duplicate package nco

   We tracked this down to multiple versions of bison being used. The best solution is to remove external bison versions earlier than 3.8 from the site config (packages.yaml).

6. Installing/using graphical applications after switching user using sudo su

   When using a role account to install spack-stack, it is sometimes necessary to run graphical applications such as the qt online installer. The following website describes in detail how this can be done: https://www.thegeekdiary.com/how-to-set-x11-forwarding-export-remote-display-for-users-who-switch-accounts-using-sudo/

11.2. MSU Hercules

1. wgrib2@2.0.8 doesn’t build on Hercules, use wgrib2@3.1.1 instead.

11.3. NASA Discover

1. Timeout when fetching software during spack installs.

   Discover’s connection to the outside world can be very slow and spack sometimes aborts with fetch timeouts. Try again until it works, sometimes have to wait for a bit.

11.4. NCAR-Wyoming Casper

1. py-scipy is missing the Pythran backend, because older versions of py-pythran (up to 0.11.x) cause compilation errors in py-scipy for all Intel compilers, and newer py-pythran versions (0.12.x and later) do not build with the old Intel compiler used on Casper.

11.5. NCAR-Wyoming Cheyenne

1. py-scipy is missing the Pythran backend, because older versions of py-pythran (up to 0.11.x) cause compilation errors in py-scipy for all Intel compilers, and newer py-pythran versions (0.12.x and later) do not build with the old Intel compiler used on Cheyenne.

11.6. NOAA Parallel Works

1. With the default module path, spack will detect the system as Cray, therefore one needs to remove it when building or using spack environments

2. libxml2 won’t untar during the spack install step, because of an issue with the filesystem. This can be avoided by making libxml2 an external package

3. The /contrib filesystem can be very, very slow

11.7. NOAA RDHPCS Gaea

1. Random “permission denied” errors during the spack install phase

   If random errors during the spack install phase occur related to “permission denied” when building packages, edit envs/env_name/config.yaml and comment out the lines build_stage and test_stage.

2. Random “git-lfs not found” errors during the spack install phase

   If random errors during the spack install phase occur related to “git-lfs not found” when building packages (e.g. crtm), simply load the module and try again (module load git-lfs).

11.8. UW (Univ. of Wisconsin) S4

1. Compiler errors when using too many threads for parallel builds

   Using more than two threads when running make (e.g. make -j4) can lead to compiler errors like the following:

[94%] Linking CXX executable test_ufo_parameters
icpc: error #10106: Fatal error in /home/opt/intel/oneapi/2022.1/compiler/2022.0.1/linux/bin/intel64/../../bin/intel64/mcpcom, terminated by kill signal
...

11.9. NAVY HPCMP Narwhal

1. On Narwhal (like on any other Cray), the spack build environment depends on the currently loaded modules. It is therefore necessary to build separate environments for different compilers while having the correct modules for that setup loaded.

2. mapl@2.35.2 does not build on Narwhal, see https://github.com/JCSDA/spack-stack/issues/524. When using the unified-dev template, one has to manually remove jedi-ufs-env and ufs-weather-model-env from the environment’s spack.yaml.

11.10. NAVY HPCMP Nautilus

1. wgrib2@2.0.8 doesn’t build on Nautilus, use wgrib2@3.1.1 instead.

11.11. macOS

1. Error invalid argument '-fgnu89-inline' not allowed with 'C++'

   This error occurs on macOS Monterey with mpich-3.4.3 installed via Homebrew when trying to build the jedi bundles that use ecbuild. The reason was that the C compiler flag -fgnu89-inline from /usr/local/Cellar/mpich/3.4.3/lib/pkgconfig/mpich.pc was added to the C++ compiler flags by ecbuild. The solution was to set CC=mpicc FC=mpif90 CXX=mpicxx when calling ecbuild for those bundles. Note that it is recommended to install mpich or openmpi with spack-stack, not with Homebrew.

2. Installation of gdal fails with error xcode-select: error: tool 'xcodebuild' requires Xcode, but active developer directory '/Library/Developer/CommandLineTools' is a command line tools instance.

   If this happens, install the full Xcode application in addition to the Apple command line utilities, and switch xcode-select over with sudo xcode-select -s /Applications/Xcode.app/Contents/Developer (change the path if you installed Xcode somewhere else).

3. Error AttributeError: Can't get attribute 'Mark' on <module 'ruamel.yaml.error' from ... when running spack install

   Some users are seeing this with Python 3.10 installed via Homebrew on macOS. Run export | grep SPACK_PYTHON to verify the Python version used, then run brew list to check if there are alternative Python versions available. Manually setting SPACK_PYTHON to a different version, for example via export SPACK_PYTHON=/usr/local/bin/python3.9, solves the problem.

4. Errors handling exceptions on macOS.

   A large number of errors related to handling exceptions thrown by applications was found when using default builds or Homebrew installations of mpich or openmpi, which use flat namespaces. With our spack version, mpich and openmpi are installed with a +two_level_namespace option that fixes the problem.

5. Errors such as Symbol not found: __cg_png_create_info_struct

   Can happen when trying to use the raster plotting scripts in fv3-jedi-tools. In that case, exporting DYLD_LIBRARY_PATH=/usr/lib/:$DYLD_LIBRARY_PATH can help. If git commands fail after this, you might need to verify where which git points to (Homebrew vs module) and unload the git module.

11.12. Ubuntu

1. The lmod version in Ubuntu 22.04 LTS breaks spack modules.

   Ubuntu 22.04 LTS will install lmod 6.6 from official apt repositories. Module files authored by spack use the depends_on directive that was introduced in lmod 7.0. The new site config instructions in Section 6.2 circumvent the issue by using tcl/tk environment modules. If you attempt to use lmod 6.6 you will get the following error:

   $ module load stack-python
   Lmod has detected the following error:  Unable to load module: python/3.10.8
   /home/ubuntu/spack-stack-1.3.1/envs/skylab-4/install/modulefiles/gcc/11.3.0/python/3.10.8.lua : [string "-- -*- lua -*-..."]:16: attempt to call global 'depends_on' (a nil value)