JEDI skylab plot steps are not working with oneapi environments on Orion and Hercules

[srherbener]

Description

When attempting to create plots using the python3 simobs utilities, the following error shows up (Hercules example using spack-stack-1.9.2):

>>> from simobs.plotting.skylab_monitor import main as skylab_monitor
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/work2/noaa/jcsda/herbener/jedi-herc/jedi-workflow/simobs/src/simobs/plotting/skylab_monitor.py", line 16, in <module>
    from simobs.plotting.plot_maps import \
  File "/work2/noaa/jcsda/herbener/jedi-herc/jedi-workflow/simobs/src/simobs/plotting/plot_maps.py", line 9, in <module>
    import scipy.stats as st
  File "/apps/contrib/spack-stack/spack-stack-1.9.2/envs/ue-oneapi-2024.1.0/install/gcc/13.3.0/py-scipy-1.14.1-xhcqbl7/lib/python3.11/site-packages/scipy/stats/__init__.py", line 610, in <module>
    from ._stats_py import *
  File "/apps/contrib/spack-stack/spack-stack-1.9.2/envs/ue-oneapi-2024.1.0/install/gcc/13.3.0/py-scipy-1.14.1-xhcqbl7/lib/python3.11/site-packages/scipy/stats/_stats_py.py", line 38, in <module>
    from scipy.spatial import distance_matrix
  File "/apps/contrib/spack-stack/spack-stack-1.9.2/envs/ue-oneapi-2024.1.0/install/gcc/13.3.0/py-scipy-1.14.1-xhcqbl7/lib/python3.11/site-packages/scipy/spatial/__init__.py", line 110, in <module>
    from ._kdtree import *
  File "/apps/contrib/spack-stack/spack-stack-1.9.2/envs/ue-oneapi-2024.1.0/install/gcc/13.3.0/py-scipy-1.14.1-xhcqbl7/lib/python3.11/site-packages/scipy/spatial/_kdtree.py", line 4, in <module>
    from ._ckdtree import cKDTree, cKDTreeNode
ImportError: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.32' not found (required by /apps/contrib/spack-stack/spack-stack-1.9.2/envs/ue-oneapi-2024.1.0/install/gcc/13.3.0/py-scipy-1.14.1-xhcqbl7/lib/python3.11/site-packages/scipy/spatial/_ckdtree.cpython-311-x86_64-linux-gnu.so)
>>> 

The the stdc++ library (libstdc++.so.6) is being loaded during runtime. However the compiled python library in scipy (/apps/contrib/spack-stack/spack-stack-1.9.2/envs/ue-oneapi-2024.1.0/install/gcc/13.3.0/py-scipy-1.14.1-xhcqbl7/lib/python3.11/site-packages/scipy/spatial/_ckdtree.cpython-311-x86_64-linux-gnu.so) was linked with a stdc++ library from a different location as revealed by the ldd command. Here's the output of running ldd on the scipy compiled library:

ldd /apps/contrib/spack-stack/spack-stack-1.9.2/envs/ue-oneapi-2024.1.0/install/gcc/13.3.0/py-scipy-1.14.1-xhcqbl7/lib/python3.11/site-packages/scipy/spatial/_ckdtree.cpython-311-x86_64-linux-gnu.so
	linux-vdso.so.1 (0x00007ffdb334d000)
	libstdc++.so.6 => /apps/contrib/spack-stack/spack-stack-1.9.2/envs/ue-oneapi-2024.1.0/install/gcc/13.3.0/gcc-runtime-13.3.0-rmyeldf/lib/libstdc++.so.6 (0x00007fedfe603000)
	libm.so.6 => /usr/lib64/libm.so.6 (0x00007fedfe528000)
	libgcc_s.so.1 => /apps/contrib/spack-stack/spack-stack-1.9.2/envs/ue-oneapi-2024.1.0/install/gcc/13.3.0/gcc-runtime-13.3.0-rmyeldf/lib/libgcc_s.so.1 (0x00007fedfe500000)
	libc.so.6 => /usr/lib64/libc.so.6 (0x00007fedfe2f7000)
	/lib64/ld-linux-x86-64.so.2 (0x00007fedfe94c000)

where you can see that /apps/contrib/spack-stack/spack-stack-1.9.2/envs/ue-oneapi-2024.1.0/install/gcc/13.3.0/gcc-runtime-13.3.0-rmyeldf/lib/libstdc++.so.6 was linked at build time, whereas /usr/lib64/libstdc++.so.6 was loaded at runtime.

This is happening because LD_LIBRARY_PATH is not being setup correctly. The desired path (/apps/contrib/spack-stack/spack-stack-1.9.2/envs/ue-oneapi-2024.1.0/install/gcc/13.3.0/gcc-runtime-13.3.0-rmyeldf/lib) is not included but /usr/lib64 is included.

It looks like oneapi provides a standard C++ library that needs to be used instead of the system provided library. Placing the desired path in LD_LIBRARY_PATH before /usr/lib64 provides a workaround, and perhaps this is the way to address this.

The same situation is ocurring on Orion, spack-stack-1.9.3 with the desired path /apps/contrib/spack-stack/spack-stack-1.9.3/envs/ue-oneapi-2024.2.1/install/gcc/12.2.0/gcc-runtime-12.2.0-bilscpd/lib not being included in LD_LIBRARY_PATH.

To Reproduce

1. set up the environment

• On Hercules:

module purge
module use /work/noaa/epic/role-epic/spack-stack/hercules/modulefiles
module load ecflow/5.8.4
module load git-lfs/3.1.2

module use /apps/contrib/spack-stack/spack-stack-1.9.2/envs/ue-oneapi-2024.1.0/install/modulefiles/Core
module use /apps/contrib/spack-stack/spack-stack-1.9.2/envs/ue-oneapi-2024.1.0/install/modulefiles/gcc/13.3.0
module load stack-oneapi/2024.2.1
module load stack-intel-oneapi-mpi/2021.13
module load stack-python/3.11.7
module load jedi-fv3-env

• On Orion:

SPACK_STACK_INTEL_ENV=/apps/contrib/spack-stack/spack-stack-1.9.3/envs/ue-oneapi-2024.2.1

# load modules
module purge
module use $SPACK_STACK_INTEL_ENV/install/modulefiles/Core
module load stack-oneapi/2024.2.1
module load stack-intel-oneapi-mpi/2021.13
module load stack-python/3.11.7

module load singularity

# This is a fix for the issue where a handful of spack-stack-1.9.1
# oneapi modules (eg, py-scipy) were not being found because
# these modules were being built with the gcc compilers instead
# of the oneapi compilers.
export LMOD_TMOD_FIND_FIRST=yes
module use $SPACK_STACK_INTEL_ENV/install/modulefiles/gcc/12.2.0

module load jedi-fv3-env

2. run python3 and try importing the following

linux_prompt> python3
>>> import pandas
>>> import scipy.stats as st

Expected behavior

The example import (above) successfully completes

System, compiler, code, ...

• using the JEDI skylab environment (at the least source the setup script and run build_workflow_apps.sh from jedi-tools)
• intel oneapi 2024.1.0 for both orion and hercules

Additional context

I suspect that there is a process we are missing in our JEDI skylab setup scripts that gets LD_LIBRARY_PATH set correctly, so the solution might simply be to add this process to the JEDI skylab scripts.

[rickgrubin-noaa]

The the stdc++ library (libstdc++.so.6) is being loaded during runtime.

/apps/contrib/spack-stack/spack-stack-1.9.2/envs/ue-oneapi-2024.1.0/install/modulefiles/Core/stack-oneapi/2024.2.1.lua

-- compiler flags and other environment variables
[...]
prepend_path("LD_LIBRARY_PATH", "/usr/lib:/usr/lib64")

which is set in the config files when the env is created / built. I cannot (creatively or otherwise) think of a way to list a "to be built in the future" path for a package library here.

That said, the env on orion specifies gcc@12.2.0 and indeed, the install log shows just that for building py-scipy:

  Project name: scipy
  Project version: 1.14.1
  C compiler for the host machine: /apps/contrib/spack-stack/spack-stack-1.9.2/spack/lib/spack/env/gcc/gcc (gcc 12.2.0 "gcc (GCC) 12.2.0")
  C linker for the host machine: /apps/contrib/spack-stack/spack-stack-1.9.2/spack/lib/spack/env/gcc/gcc ld.bfd 2.35.2-24
  C++ compiler for the host machine: /apps/contrib/spack-stack/spack-stack-1.9.2/spack/lib/spack/env/gcc/g++ (gcc 12.2.0 "g++ (GCC) 12.2.0")

However the compiled python library in scipy (/apps/contrib/spack-stack/spack-stack-1.9.2/envs/ue-oneapi-2024.1.0/install/gcc/13.3.0/py-scipy-1.14.1-xhcqbl7/lib/python3.11/site-packages/scipy/spatial/_ckdtree.cpython-311-x86_64-linux-gnu.so) was linked with a stdc++ library from a different location as revealed by the ldd command. Here's the output of running ldd on the scipy compiled library:

ldd /apps/contrib/spack-stack/spack-stack-1.9.2/envs/ue-oneapi-2024.1.0/install/gcc/13.3.0/py-scipy-1.14.1-xhcqbl7/lib/python3.11/site-packages/scipy/spatial/_ckdtree.cpython-311-x86_64-linux-gnu.so
	linux-vdso.so.1 (0x00007ffdb334d000)
	libstdc++.so.6 => /apps/contrib/spack-stack/spack-stack-1.9.2/envs/ue-oneapi-2024.1.0/install/gcc/13.3.0/gcc-runtime-13.3.0-rmyeldf/lib/libstdc++.so.6 (0x00007fedfe603000)
	libm.so.6 => /usr/lib64/libm.so.6 (0x00007fedfe528000)
	libgcc_s.so.1 => /apps/contrib/spack-stack/spack-stack-1.9.2/envs/ue-oneapi-2024.1.0/install/gcc/13.3.0/gcc-runtime-13.3.0-rmyeldf/lib/libgcc_s.so.1 (0x00007fedfe500000)
	libc.so.6 => /usr/lib64/libc.so.6 (0x00007fedfe2f7000)
	/lib64/ld-linux-x86-64.so.2 (0x00007fedfe94c000)

If I try:

ldd /apps/contrib/spack-stack/spack-stack-1.9.2/envs/ue-oneapi-2024.1.0/install/gcc/12.2.0/py-scipy-1.14.1-o7brx52/lib/python3.11/site-packages/scipy/spatial/_ckdtree.cpython-311-x86_64-linux-gnu.so

I see:

	linux-vdso.so.1 (0x00007ffd68338000)
	libstdc++.so.6 => /apps/contrib/spack-stack/spack-stack-1.9.2/envs/ue-oneapi-2024.1.0/install/gcc/12.2.0/gcc-runtime-12.2.0-bilscpd/lib/libstdc++.so.6 (0x0000153619c7d000)
	libm.so.6 => /usr/lib64/libm.so.6 (0x0000153619ba0000)
	libgcc_s.so.1 => /apps/contrib/spack-stack/spack-stack-1.9.2/envs/ue-oneapi-2024.1.0/install/gcc/12.2.0/gcc-runtime-12.2.0-bilscpd/lib/libgcc_s.so.1 (0x0000153619b7f000)
	libc.so.6 => /usr/lib64/libc.so.6 (0x0000153619976000)
	/lib64/ld-linux-x86-64.so.2 (0x0000153619f7d000)

which still references a stack-built libstdc++, but not the one you're seeing.

Further, referencing the important file path from your comment:

file /apps/contrib/spack-stack/spack-stack-1.9.2/envs/ue-oneapi-2024.1.0/install/gcc/13.3.0/py-scipy-1.14.1-xhcqbl7/lib/python3.11/site-packages/scipy/spatial/_ckdtree.cpython-311-x86_64-linux-gnu.so

/apps/contrib/spack-stack/spack-stack-1.9.2/envs/ue-oneapi-2024.1.0/install/gcc/13.3.0/py-scipy-1.14.1-xhcqbl7/lib/python3.11/site-packages/scipy/spatial/_ckdtree.cpython-311-x86_64-linux-gnu.so: cannot open `/apps/contrib/spack-stack/spack-stack-1.9.2/envs/ue-oneapi-2024.1.0/install/gcc/13.3.0/py-scipy-1.14.1-xhcqbl7/lib/python3.11/site-packages/scipy/spatial/_ckdtree.cpython-311-x86_64-linux-gnu.so' (No such file or directory)

I think that you've inadvertently confused paths on orion with paths on hercules, because on hercules:

/apps/contrib/spack-stack/spack-stack-1.9.3/envs/ue-oneapi-2024.2.1/site/compilers.yaml

- compiler:
    spec: gcc@13.3.0
    paths:
      cc: /apps/contrib/spack-stack/gnu/installs/gnu/13.3.0/bin/gcc
      cxx: /apps/contrib/spack-stack/gnu/installs/gnu/13.3.0/bin/g++
      f77: /apps/contrib/spack-stack/gnu/installs/gnu/13.3.0/bin/gfortran
      fc: /apps/contrib/spack-stack/gnu/installs/gnu/13.3.0/bin/gfortran
    flags: {}
    operating_system: rocky9
    target: x86_64
    modules:
    - gnu/13.3.0

It needs to be clearly stated that this installation for GNU compilers was built by EPIC; owner / group settings show this, and the location of the installation further gives away the story's ending.

The "how and why" of this installation doesn't matter as much as the fact that widely used stack envs on the MSU hosts (which are more or less functionally equivalent) should be the same; that is, on hercules, the oneAPI stack should use the system-supplied gcc/12.2.0 module for GNU compilers. Doing so would not fix the problem per se, but I'll argue that being consistent across hosts when building stack envs -- where consistent means "use system-supplied tools / modules where possible / provided" -- is a practice that EPIC would do well to follow.

---

Anyway, this general scenario happens because

/apps/contrib/spack-stack/spack-stack-1.9.2/envs/ue-oneapi-2024.1.0/site/packages.yaml

    [....]
    py-scipy:
      require: '%gcc'

It looks like oneapi provides a standard C++ library that needs to be used instead of the system provided library. Placing the desired path in LD_LIBRARY_PATH before /usr/lib64 provides a workaround, and perhaps this is the way to address this.

The module file for stack-oneapi/2024.2.1 could be modified to place the "built with" libstdc++ library prior to other locations. This would be a manual, after-the-fact edit of module files.

I suspect that there is a process we are missing in our JEDI skylab setup scripts that gets LD_LIBRARY_PATH set correctly, so the solution might simply be to add this process to the JEDI skylab scripts.

To accomplish this, one would need to go looking for the location of libstdc++ for a given package (e.g. py-scipy) and adjust accordingly. While that's a solution, it seems better if your JEDI skylab setup scripts worked "as is."

@climbfuji:

This situation is a result of having to build py-scipy with gcc on hercules and orion (see here) because py-scipy's meson build would not accommodate building with oneAPI (and this has been noted when trying to build envs with spack-stack@v2 / spack@v1) -- something peculiarly limited to these hosts, as py-scipy builds / installs on other Tier-1 hosts with oneAPI.

Also: since the system-supplied GNU modules do specify, e.g.

prepend_path("LD_LIBRARY_PATH","/apps/spack-managed/gcc-11.3.1/gcc-12.2.0-7cu3qahzhsxpauy4jlnsbcqmlbkxbbbo/lib64")
prepend_path("LD_LIBRARY_PATH","/apps/spack-managed/gcc-11.3.1/gcc-12.2.0-7cu3qahzhsxpauy4jlnsbcqmlbkxbbbo/lib")

is it advisable to include these paths in a stack env config file as part of setting env vars for a compiler / package? I don't know that it would solve this problem, but asking nonetheless.

---

TL;DR for @srherbener and @climbfuji:

I'd think the best interim solution is to manually edit stack-oneapi/2024.2.1.lua -- thoughts, please and thanks.

[srherbener]

@rickgrubin-noaa thanks for your help with this!

Here's what I am successfully using for workarounds on Hercules and Orion.

On Hercules after the all of the spack-stack modules are loaded, I modify LD_LIBRARY_PATH like this:

# workaround for the LD_LIBRARY_PATH issue
LD_LIBRARY_PATH="/apps/contrib/spack-stack/spack-stack-1.9.2/envs/ue-oneapi-2024.1.0/install/gcc/13.3.0/gcc-runtime-13.3.0-rmyeldf/lib:$LD_LIBRARY_PATH"

On Orion, after the spack-stack module loads, I do this:

# workaround for LD_LIBRARY_PATH bug
LD_LIBRARY_PATH="/apps/contrib/spack-stack/spack-stack-1.9.3/envs/ue-oneapi-2024.2.1/install/gcc/12.2.0/gcc-runtime-12.2.0-bilscpd/lib:$LD_LIBRARY_PATH"

I think your suggestion for a fix is good, but I am pointing out the above workarounds since they are using different paths than what you are suggesting with the lua prepend_path example above. The paths that I'm using match the paths that ldd says were used during the link step in the build. I'm using those paths so that during runtime the same exact stdc++ library file will be loaded.

The reason I'm seeing the error is that LD_LIBRARY_PATH is being used to find the libstdc++.so.6 file during runtime and there are multiple copies of that file around (/usr/lib64, /apps/spack-managed/gcc-11.3.1/gcc-12.2.0-7cu3qahzhsxpauy4jlnsbcqmlbkxbbbo/lib64, etc.) and the wrong one is being found and loaded because the path that is needed for loading the correct file (/apps/contrib/spack-stack/spack-stack-1.9.2/envs/ue-oneapi-2024.1.0/install/gcc/13.3.0/gcc-runtime-13.3.0-rmyeldf/lib) is not in LD_LIBRARY_PATH.

By the way I found a much simpler way to expose this issue (that doesn't need the JEDI skylab environment). I'll modify the description section with the new instructions.

Thanks again for your help!

[climbfuji]

@srherbener Is still an issue with spack-stack-2.1.0 ?

[srherbener]

I'm having trouble getting jedi-bundle to build on Hercules using spack-stack-2.1.0. I'm getting help from the JCSDA infra team for now. I'll let you know if I need further help from the spack-stack team.

[rickgrubin-noaa]

I'm having trouble getting jedi-bundle to build on Hercules using spack-stack-2.1.0. I'm getting help from the JCSDA infra team for now. I'll let you know if I need further help from the spack-stack team.

@srherbener do you have an update? The stacks on hercules need to be rebuilt to accommodate a netcdf-c plugins fix.

[srherbener]

I'm having trouble getting jedi-bundle to build on Hercules using spack-stack-2.1.0. I'm getting help from the JCSDA infra team for now. I'll let you know if I need further help from the spack-stack team.

@srherbener do you have an update? The stacks on hercules need to be rebuilt to accommodate a netcdf-c plugins fix.

@rickgrubin-noaa thanks for checking in. Right now @eap is working on resolving this issue so he would know whether we need another package upgrade.

[rickgrubin-noaa]

I'm having trouble getting jedi-bundle to build on Hercules using spack-stack-2.1.0. I'm getting help from the JCSDA infra team for now. I'll let you know if I need further help from the spack-stack team.

@srherbener do you have an update? The stacks on hercules need to be rebuilt to accommodate a netcdf-c plugins fix.

@rickgrubin-noaa thanks for checking in. Right now @eap is working on resolving this issue so he would know whether we need another package upgrade.

@eap -- status please? Environments on hercules require rebuilding to apply the fix from #2010. Thanks.

[srherbener]

@eap - this is the issue we discussed today. Please provide guidance for @rickgrubin-noaa and thank you for your help with this.