diff --git a/containers/eic/Dockerfile b/containers/eic/Dockerfile index 1d212b74a..1a3b85460 100644 --- a/containers/eic/Dockerfile +++ b/containers/eic/Dockerfile @@ -46,6 +46,7 @@ ARG ENV=xl ENV SPACK_ENV=/opt/spack-environment/${ENV} ARG SPACK_FLAGS="--backtrace" ARG SPACK_INSTALL_FLAGS="--no-check-signature --show-log-on-error --yes-to-all" +ARG SPACK_BUILDER_INSTALL_FLAGS="${SPACK_INSTALL_FLAGS}" ENV SPACK_COLOR="always" ENV GIT_TERMINAL_PROMPT=0 @@ -79,7 +80,13 @@ RUN --mount=type=cache,target=/ccache,id=ccache-${TARGETPLATFORM} \ <filesystem] + A2[debuginfod
scans it directly] + A3[gdb fetches over HTTP] + A1 --> A2 --> A3 + end + + subgraph Spack["Spack, before this framework"] + direction TB + B1[Binaries in OCI registry
GHCR / eicweb] + B2[debuginfod: can't scan
a registry] + B3[gdb: no source, no symbols] + B1 --> B2 --> B3 + end + + Debian ~~~ Spack +``` + +This framework closes that gap: + +* Makes DWARF paths machine-agnostic. +* Captures source and symbol data at install time and after the fact. +* Stores it in the same OCI registries used by the build cache, tagged by build ID. +* Resolves GDB build-ID lookups against OCI data over HTTPS via a lightweight debuginfod-compatible adapter. + +## Cherry-picks + +Enabled via `spack.sh` and `spack-packages.sh`: + +```bash +# spack.sh +## 2ba3505dd8985a0fc86695e43cae0020fc50daa8: feat: debuggable installations (source hook, symbol +## splitting, gdbinit, OCI autopush) plus debuginfod, squashed and cherry-picked via open draft +## PR spack/spack#52949 + +# spack-packages.sh +## fdd30418cfd404a8de135c5fcfc349d5de87f84b: compiler-wrapper: add 1.1.0-build-id prototype version (spack-packages#6214) +## 5945d81a8359eed559ec60b1be9151de57473f51: elfutils: patch debuginfod_find_source to accept ./-relative filenames (spack-packages#6259) +``` + +## Environment wiring + +`compiler-wrapper@1.1.0-build-id` and `RelWithDebInfo` overrides for ROOT/Geant4 and dependents in `xl/spack.yaml` and `xl/epic/spack.yaml`, with `elfutils@0.194+debuginfod` GDB support: + +```yaml +packages: + compiler-wrapper: + require: + - '@1.1.0-build-id' + root: + require: + - build_type=RelWithDebInfo + geant4: + require: + - build_type=RelWithDebInfo + acts: + require: + - build_type=RelWithDebInfo + # ...similarly for celeritas, dd4hep, edm4hep, hepmc3, podio, sherpa + professor: + require: + - cflags=-g + - cxxflags=-g + pythia8: + require: + - cflags=-g + - cxxflags=-g +specs: +- gdb ^elfutils@0.194+debuginfod +- ... +``` + +## Full pipeline + +```mermaid +flowchart TB + subgraph Build["scripts/build-eic.sh (ENV=dbg or xl)"] + F1["SPACK_BUILDER_INSTALL_FLAGS =
SPACK_INSTALL_FLAGS + --debug-source --debug-symbols"] + end + + Build -->|--build-arg| Docker + + subgraph Env["spack.yaml (dbg/xl)"] + E1["compiler-wrapper:
require '@1.1.0-build-id'
(cherry-pick spack/spack-packages#6214)"] + end + + subgraph Wrapper["spack/compiler-wrapper#19, cc.sh"] + W1["intercepts every compile/link call"] + W2["injects -ffile-prefix-map"] + W3["injects --build-id / -Wl,--build-id"] + W1 --> W2 + W1 --> W3 + end + + E1 -.->|pins version used by| Wrapper + + subgraph Docker["containers/eic/Dockerfile — builder track"] + D1["spack install $SPACK_BUILDER_INSTALL_FLAGS
(compiles for real, dbg/xl-scoped)"] + D2["new_installer.py phase.execute()
(cherry-pick spack/spack#52949)
routed through cc.sh"] + D3["install_debug_artifacts()
split_debug_symbols()
write_gdbinit()
(cherry-pick spack/spack#52949)"] + D1 --> D2 + D2 -->|"machine-agnostic DWARF paths + build-id already embedded)"| D3 + end + + Wrapper -.->|"cc.sh invoked for
every compile unit"| D2 + + D3 -->|writes to| Cache["~/.spack/debug-sources/<pkg>-<ver>-<hash>/
captured source tree, symbols/.build-id/, gdbinit"] + + Cache -->|install completes| Hook["hooks/autopush.py :: post_install()
(cherry-pick spack/spack#52949)"] + + subgraph Hook_detail["for each autopush:true mirror (eicweb, ghcr)"] + H1["1. uploader.push_or_raise()
tag: pkg-ver-hash.spack (always)"] + H2["2. push_debug_artifacts()
tag: debuginfo-build-id (cherry-pick spack/spack#52949)"] + end + + Hook --> Hook_detail + Hook_detail -->|OCI push| Registry[("GHCR / eicweb
OCI registry")] + + Registry --> R1["pkg-ver-hash.spack
(regular buildcache)"] + Registry --> R2["debuginfo-build-id
layers: .debug + source.tar.gz (new)"] + + R1 -->|"--use-buildcache only
(unchanged behavior)"| Runtime["Runtime image stages
(all environments)"] + + R2 -->|"spack debug fetch
or spack debug serve"| Adapter + + subgraph Adapter["debuginfod-compatible adapter (spack debug serve)"] + A1["ThreadingHTTPServer
127.0.0.1:8002"] + A2["resolves /buildid/<id>/{debuginfo,source}
against OCI manifest"] + A1 --> A2 + end + + subgraph Elfutils["elfutils, libdebuginfod client"] + EL1["patched: accepts ./-relative
DWARF filenames
(cherry-pick spack/spack-packages#6259)"] + end + + Elfutils -.->|"required on the gdb host
to even send the source request"| GDB + + Adapter -->|"HTTPS: symbols + source"| GDB["gdb, DEBUGINFOD_URLS=http://127.0.0.1:8002
no rebuild needed"] +``` + +## Why this is safe for shared infrastructure + +* **Opt-in only.** `--debug-source` and `--debug-symbols` are enabled only for `dbg`/`xl`; `push_debug_artifacts` runs only when `debug_source_dir(spec)` exists. `ci`, `prod`, and other environments are unaffected. +* **Reuses existing infrastructure.** Uses the existing OCI registries (`eicweb`, `ghcr`), `autopush` hook, and credentials from `mirrors.yaml.in`. No new services, registries, or access models. +* **Build-ID-based keying.** Debug artifacts are keyed by build ID, so the same binary always resolves to the same debug data, regardless of concretization. + + +## Current scope and open questions + +* `spack debug serve` is local/on-demand (`--start-daemon`, `--stop-daemon`, `--status`), not a persistent service. +* A shared persistent debuginfod service vs. local `spack debug serve` instances using shared OCI registries remains an open infrastructure decision. + +## Related Documentation + +- [Architecture Overview](architecture.md) - Build system structure +- [Spack Environment](spack-environment.md) - Spack configuration and packages +- [Build Pipeline](build-pipeline.md) - CI workflow details diff --git a/scripts/build-eic.sh b/scripts/build-eic.sh index 8ebdb6325..63ab33071 100755 --- a/scripts/build-eic.sh +++ b/scripts/build-eic.sh @@ -204,6 +204,15 @@ for build_type in "${BUILD_TYPES[@]}"; do # shellcheck disable=SC2206 # word splitting is intentional: BUILD_OPTIONS is a space-separated list build_cmd+=(${BUILD_OPTIONS}) + ## Compute install flags: base flags plus debug capture for the dbg environment + SPACK_INSTALL_FLAGS="--no-check-signature --show-log-on-error --yes-to-all" + SPACK_BUILDER_INSTALL_FLAGS="${SPACK_INSTALL_FLAGS}" + if [ "${ENV}" = "dbg" ] || [ "${ENV}" = "xl" ]; then + SPACK_BUILDER_INSTALL_FLAGS="${SPACK_BUILDER_INSTALL_FLAGS} --debug-source --debug-symbols" + fi + build_cmd+=(--build-arg "SPACK_INSTALL_FLAGS=${SPACK_INSTALL_FLAGS}") + build_cmd+=(--build-arg "SPACK_BUILDER_INSTALL_FLAGS=${SPACK_BUILDER_INSTALL_FLAGS}") + ## Output mode: push-by-digest in all CI modes; load locally if [ "${CI_MODE}" != "local" ]; then ## Push by digest; CI wrapper creates final tags via imagetools create. diff --git a/spack-environment/dbg/epic/spack.yaml b/spack-environment/dbg/epic/spack.yaml index 26fe88789..578ab4135 100644 --- a/spack-environment/dbg/epic/spack.yaml +++ b/spack-environment/dbg/epic/spack.yaml @@ -5,6 +5,10 @@ spack: - ../../config.yaml - ../../packages.yaml - ../../view.yaml + packages: + compiler-wrapper: + require: + - '@1.1.0-build-id' specs: - algorithms build_type=Debug - edm4eic build_type=Debug diff --git a/spack-environment/dbg/spack.yaml b/spack-environment/dbg/spack.yaml index b544d7fb3..54a8fa4f0 100644 --- a/spack-environment/dbg/spack.yaml +++ b/spack-environment/dbg/spack.yaml @@ -5,12 +5,16 @@ spack: - ../packages.yaml - ../packages_root_without_opengl.yaml - ../view.yaml + packages: + compiler-wrapper: + require: + - '@1.1.0-build-id' specs: - acts build_type=Debug - cmake - dd4hep build_type=Debug - edm4hep build_type=Debug - - gdb + - gdb ^elfutils@0.194+debuginfod - irt build_type=Debug - jana2 build_type=Debug - valgrind diff --git a/spack-environment/xl/epic/spack.yaml b/spack-environment/xl/epic/spack.yaml index 9b1893160..bb2dc6e25 100644 --- a/spack-environment/xl/epic/spack.yaml +++ b/spack-environment/xl/epic/spack.yaml @@ -11,9 +11,34 @@ spack: max_dupes: epic: 10 packages: + compiler-wrapper: + require: + - '@1.1.0-build-id' + root: + require: + - build_type=RelWithDebInfo geant4: require: - +opengl + - build_type=RelWithDebInfo + acts: + require: + - build_type=RelWithDebInfo + celeritas: + require: + - build_type=RelWithDebInfo + dd4hep: + require: + - build_type=RelWithDebInfo + edm4hep: + require: + - build_type=RelWithDebInfo + hepmc3: + require: + - build_type=RelWithDebInfo + podio: + require: + - build_type=RelWithDebInfo specs: - algorithms - edm4eic diff --git a/spack-environment/xl/spack.yaml b/spack-environment/xl/spack.yaml index 4a4452b45..024d09b2d 100644 --- a/spack-environment/xl/spack.yaml +++ b/spack-environment/xl/spack.yaml @@ -5,6 +5,45 @@ spack: - ../packages.yaml - ../packages_root_with_opengl.yaml - ../view.yaml + packages: + compiler-wrapper: + require: + - '@1.1.0-build-id' + root: + require: + - build_type=RelWithDebInfo + geant4: + require: + - build_type=RelWithDebInfo + acts: + require: + - build_type=RelWithDebInfo + celeritas: + require: + - build_type=RelWithDebInfo + dd4hep: + require: + - build_type=RelWithDebInfo + edm4hep: + require: + - build_type=RelWithDebInfo + hepmc3: + require: + - build_type=RelWithDebInfo + podio: + require: + - build_type=RelWithDebInfo + professor: + require: + - cflags=-g + - cxxflags=-g + pythia8: + require: + - cflags=-g + - cxxflags=-g + sherpa: + require: + - build_type=RelWithDebInfo specs: - acts - actsvg @@ -38,7 +77,7 @@ spack: - fjcontrib - fmt - g4occt - - gdb + - gdb ^elfutils@0.194+debuginfod - geant4 +opengl - gfal2 - gfal2-util diff --git a/spack-packages.sh b/spack-packages.sh index ffe3a8697..817309bbc 100644 --- a/spack-packages.sh +++ b/spack-packages.sh @@ -48,6 +48,8 @@ c44cd71db91d6e6c68cb604dcb87311b49b1bedb 9385426b49d2c49d026629cb5e96bb5893824e3d d1bfcea155bb51220a4baddfcc75a0ba4d4b972c f3085a0fd9c327692475fbb72b8b9f738c541358 +fdd30418cfd404a8de135c5fcfc349d5de87f84b +5945d81a8359eed559ec60b1be9151de57473f51 --- ## Optional hash table with comma-separated file list ## For these commits, the cherry-pick will be restricted to the listed files only. @@ -93,3 +95,5 @@ read -r -d '' SPACKPACKAGES_CHERRYPICKS_FILES <<- \ ## 9385426b49d2c49d026629cb5e96bb5893824e3d: opencascade: add v7.9.2, v7.9.3 ## d1bfcea155bb51220a4baddfcc75a0ba4d4b972c: opencascade: add v8.0.0, v8.0.0.p1 ## f3085a0fd9c327692475fbb72b8b9f738c541358: opencascade: pass tcl library paths to the cmake build to fix mac builds +## fdd30418cfd404a8de135c5fcfc349d5de87f84b: compiler-wrapper: add 1.1.0-build-id prototype version (spack-packages#6214) +## 5945d81a8359eed559ec60b1be9151de57473f51: elfutils: patch debuginfod_find_source to accept ./-relative filenames (spack-packages #6259) diff --git a/spack.sh b/spack.sh index f3509ca49..6fa11ea4c 100644 --- a/spack.sh +++ b/spack.sh @@ -12,6 +12,7 @@ read -r -d '' SPACK_CHERRYPICKS <<- \ --- || true 292b0dcaba3b2a5e3f9668d205d39fee2c715721 678e506a95b319c573ba7e84703b06d7275ab80e +a346767b52f3fed1eb8d33ffdbbec6b2dfd2d7d7 --- ## Optional hash table with comma-separated file list read -r -d '' SPACK_CHERRYPICKS_FILES <<- \ @@ -21,3 +22,6 @@ read -r -d '' SPACK_CHERRYPICKS_FILES <<- \ ## [hash]: [description] ## 292b0dcaba3b2a5e3f9668d205d39fee2c715721: fix: write created time field with OCI buildcache config ## 678e506a95b319c573ba7e84703b06d7275ab80e: fix: don't map prefix to view root for pkgs excluded from view +## a346767b52f3fed1eb8d33ffdbbec6b2dfd2d7d7: feat: debuggable installations (source hook, symbol +## splitting, gdbinit, OCI autopush) plus debuginfod, squashed and cherry-picked via open draft +## PR spack/spack#52949